"So....what are you doing Saturday night?"
"...how about Friday?"
Woody Allen, Play It Again, Sam
If the product test suite is highly automated, then test execution should be a single command which is part of the nightly build. This requires not only a mature test suite, but a reasonably mature product. There is a trade off between the frequency of test runs and analysis time required to digest the results. Analysis can be a heavy burden when large numbers of tests are failing. In the best case, planning for test execution is primarily a question of run time resources.
If the test suite is not highly automated, then humans must perform test procedures by hand. This is bad for several reasons:
In this discussion, planning for test execution is treated as a question of run time resources.
Developing test code is time consuming. If tests can be written such that a single piece of test code can run in multiple environments, then the coding investment can be substantially leveraged.
What constitutes and environment depends on the product. The implementation of environments in tests is often through environment variables passed to the test program. For example, when testing a compiler tool like Purify, the environment includes
The cross product of these environment variables makes one test program execute as dozens of tests. There are, of course, pros and cons to this:
Copyright 1998 Anne Powell