Test Execution

"So....what are you doing Saturday night?"
"Committing suicide."
"...how about Friday?"

Woody Allen, Play It Again, Sam

If the product test suite is highly automated, then test execution should be a single command which is part of the nightly build. This requires not only a mature test suite, but a reasonably mature product. There is a trade off between the frequency of test runs and analysis time required to digest the results. Analysis can be a heavy burden when large numbers of tests are failing. In the best case, planning for test execution is primarily a question of run time resources.

If the test suite is not highly automated, then humans must perform test procedures by hand. This is bad for several reasons:

• it is an insanely boring task
• errors become harder to find the longer you look
• engineer time is expensive

Planning Test Execution

In this discussion, planning for test execution is treated as a question of run time resources.

• # of test suite runs is directly proportional to:
• # of development checkpoints
• frequency of status reports
• # of test suite runs is inversely proportional to:
• product stability
• product stability is inversely proportional to:
• difference between the current and target code quality
• # of bugs found recently
• # of bugs fixed recently
• source code change rate
• time to run tests is directly proportional to:
• # of tests
• # of environments
• time to run tests is inversely proportional to:
• % of tests automated
• Scheduling test execution resources requires:
• Automated: ((frequency of runs) * (# of environments)) / (degree of parallellism)
• Manual: (frequency of runs) * (# of test cases) * (# of environements) * (time per test)

Development Checkpoints

• nightly builds
• major builds (alpha, beta, FCS)
• special builds (Purify'd or instrumented for coverage information)

Test Environments

Developing test code is time consuming. If tests can be written such that a single piece of test code can run in multiple environments, then the coding investment can be substantially leveraged.

What constitutes and environment depends on the product. The implementation of environments in tests is often through environment variables passed to the test program. For example, when testing a compiler tool like Purify, the environment includes

• the runtime operating system (many test.c files contain #ifdef logic for platform dependent behavior)
• what compiler to use
• what compiler flags to use

The cross product of these environment variables makes one test program execute as dozens of tests. There are, of course, pros and cons to this:

Pro

• creates many more variations than could be created manually.
• can be used to reduce the number of test driver scripts needed.

Con

• often difficult to weed out combinations that don't make sense
• can easily create more tests than can be run and/or analyzed in a reasonable time.

Copyright 1998 Anne Powell