"To know a thing well, know its limits. Only when pushed beyond its tolerance will the true nature be seen."
Test case development is a software design project. Each suite of tests should be described at a high level in a Test Design Specification. This document should describe the purpose of each test and the methods used in testing. Other test engineers should be able to gain a high level understanding of the suite by reading the document.
The major test design factors are
These relationships provide some insight in how test development is affected by its environment. For the mathematical purists, inverse relations here do have a non-zero lower bound.
Any control that the test can exert over the object is a hook. The user interface is a group of test hooks. Depending on the type of product and the product quality goals, the user visible interfaces may be insufficient. Areas where additional control may be required include
Test engineers should discuss test hooks with development engineers as soon as functional specifications are available. If some bypass is added for the tests, the tester must ensure that the normal user path is not forgotten.
It is common practice to use a file to store expected, or gold, results. There are two major gold file theories on result checking. Both are methods for coping with unpredictable results.
It is sometimes possible to bypass gold files and calculate expected results at run time. In this scenario, a test might use a brute force algorithm to check a more complex one. This style of testing trades run-time speed to avoid the problems of gold file management.
The simplest test harness needs to
Finding the tests is usually implement either by explicitly telling the test harness the name of each test to be run, or by naming the tests in some predictable manner so they can be found by searching.
The run time split of responsibilities between the harness and the tests is a complex problem. There seem to be an infinite number of ways to make the cut. Some test harnesses (like TET) provide an API to the tests. Some harnesses, like many of the GUI testers, may provide a complete test development and runtime environment. Some test harnesses may provide values for test variables at run time (TestExpert). Some test harnesses do nothing but start the test program and record its output and exit status.
To further blur the boundary between test harness and test, most test suites will have some collection of utility functions/programs that all tests in that suite share. Test engineers often consider these utilities to be part of the harness for that suite. These utilities typically include set-up and result checking code.
Results should be gathered into a central location for review and archiving. The review audience will of course include the engineer who ran the test, but might be of interest to a broader audience. It is often useful to have several versions of the same report available:
A formatted verbose report can be filtered to obtain the others. If you can run your tests every night, then it's useful to send the pass/fail count report along with the list of which tests failed as email to the team. That way developers can see quickly if their last change broke something.
Results should be archived to answer questions like "did this test ever run?" and "how could we have missed that bug?" At a minimum, results from regression runs should be saved several months.
Copyright 1998 Anne Powell