What’s an important bug? Important to whom? To a first approximation, the answer must be “to customers”. Almost everyone will nod their head upon hearing this definition, but do they mean it? Here’s a test of your organization’s maturity. Suppose your product is a system that accepts email requests for service. As soon as a request is received, it sends a reply that says “your request of 5/12/97 was accepted and its reference ID is NIC-051297-
3”.
A tester who sends in many requests per day finds she has difficulty keeping track of which request goes with which ID. She wishes that the original request were appended to the acknowledgement. Furthermore, she realizes that some customers will also generate many requests per day, so would also appreciate this feature. Would she:
- file a bug report documenting a usability problem, with the expectation that it will be assigned a reasonably high priority (because the fix is clearly useful to everyone, important to some users, and easy to do)?
- file a bug report with the expectation that it will be assigned “enhancement request” priority and disappear forever into the bug database?
- file a bug report that yields a “works as designed” resolution code, perhaps with an email “nastygram” from a programmer or the development manager?
- not bother with a bug report because it would end up in cases (2) or (3)?
If usability problems are not considered valid bugs, your project defines the
testing task too narrowly. Testers are restricted to checking whether the product does what was intended, not whether what was intended is useful. Customers do not care about the distinction, and testers shouldn’t either.
Testers are often the only people in the organization who use the system as heavily as an expert. They notice usability problems that experts will see. (Formal usability testing almost invariably concentrates on novice users.) Expert customers often don’t report usability problems, because they’ve been trained to know it’s not worth their time.
Instead, they wait (in vain, perhaps) for a more usable product and switch to it. Testers can prevent that lost revenue. While defining the purpose of testing as “finding bugs important to customers” is a step forward, it’s more restrictive than I like. It means that there is no focus on an estimate of quality (and on the quality of that estimate). Consider these two situations for a product with five subsystems.
- 100 bugs are found in subsystem 1 before release. (For simplicity, assume that all bugs are of the highest priority.) No bugs are found in the other subsystems. After release, no bugs are reported in subsystem 1, but 12 bugs are found in each of the other subsystems.
- Before release, 50 bugs are found in subsystem 1. 6 bugs are found in each of the other subsystems. After release, 50 bugs are found in subsystem 1 and 6 bugs in each of the other subsystems.
