Fault Investigation

"I know what you're thinking about," said Tweedledum, "but it isn't so, nohow." "Contrariwise," continued Tweedledee, "if it was so, it might be; and if it were so, it would be; but as it isn't, it ain't. That's logic."

Lewis Carroll

For every test failure, some one must

• read the test log
• figure out what happened
• try to recreate the problem
• fix the product or fix the test

Bug writing rates per test engineer tend to run between three and seven bugs per week, as a rule of thumb. I have seen projects where the rate per test engineer was 6 a day. If you see this, you probably have a serious problem. This does not include test failures that are test problems. Those are usually fixed without a bug report being written.

Relationships for planning fault investigation

• fault analysis time is directly proportional to:
• # of test suite runs
• # of tests failing
• # of environments
• product complexity
• bug report quality (more detailed, more time)
• # of new versus known failures
• fault analysis time is inversely proportional to:
• quality of product documentation
• expertise of the test engineer with the product
• usefulness of the information in the test log
• Scheduling fault analysis requires:
• (# of suite runs) * (# auto tests failing + # manual tests) * (analysis per test)
  where
• # tests failing and analysis time per test should be decreasing over time per major revision
• or, you need to have some historical data for expected bug arrival rates

Copyright 1998 Anne Powell