Testing Mistakes continued...

From the “find important bugs” standpoint, the first testing effort was superior. It found 100 bugs before release, whereas the second found only 74. But I think you can make a strong case that the second effort is more useful in practical terms. Let me restate the twosituations in terms of what a test manager might say before release:
1. “We have tested subsystem 1 very thoroughly, and we believe we’ve found almost allof the priority 1 bugs. Unfortunately, we don’t know anything about the bugginess ofthe remaining five subsystems.”
2. “We’ve tested all subsystems moderately thoroughly. Subsystem 1 is still very buggy.The other subsystems are about 1/10th as buggy, though we’re sure bugs remain.

”This is, admittedly, an extreme example, but it demonstrates an important point. Theproject manager has a tough decision: would it be better to hold on to the product formore work, or should it be shipped now? Many factors - all rough estimates of possiblefutures - have to be weighed: Will a competitor beat us to release and tie up the market? Will dropping an unfinished feature to make it into a particular magazine’s special “JavaDevelopment Environments” issue cause us to suffer in the review? Will critical customerX be more annoyed by a schedule slip or by a shaky product? Will the product be buggyenough that profits will be eaten up by support costs or, worse, a recall?

The testing team will serve the project manager better if it concentrates first on providingestimates of product bugginess (reducing uncertainty), then on finding more of the bugsthat are estimated to be there. That affects test planning, the topic of the next theme.

It also affects status reporting. Test managers often err by reporting bug data without putting it into context. Without context, project management tends to focus on one graph:


The flattening in the curve of bugs found will be interpreted in the most optimistic possible
way unless you as test manager explain the limitations of the data:
· “Only half the planned testing tasks have been finished, so little is known about half
the areas in the project. There could soon be a big spike in the number of bugs
found.”
· “That’s especially likely because the last two weekly builds have been lightly tested.

“Furthermore, based on previous projects with similar amounts and kinds of testing effort, it’s reasonable to expect at least 45 priority-1 bugs remain undiscovered. Historically, that’s pretty high for a successful product.”