Analyzing and Interpreting Test Results - Software Testing: Testing Across the Entire Software Development Life Cycle - page 196

Another interesting conclusion can be drawn from the project defect curve com-
parisons. Sometimes it is instructive to ask the question, “What is the risk/reward of
shortening completion of the next project by 25%?” For projects like our examples
that are 24 weeks long, a 25% schedule reduction would set the completion date back
to week 16. The development team will raise a hewn cry that the project due dates
are too short now for them to do their best job. The project defect curve comparison
can inject some objectivity into the discussion.
Starting with the prior project, there were 70 customer-discovered defects. For the
sake of discussion, consider that the economics and resources necessary to correct 70
customer-discovered defects are an unwelcome cost but not a fi nancial showstopper.
If your next defect tracking log looks like Project A, then you can draw a vertical
line at week 16 on the x -axis and predict how many more customer defects will occur
because testing will be stopped short of the usual 24-week schedule. You can report back
that by completing Project A on week 16 will result in about 70 more customer-discov-
ered defects than the two defects originally predicted. The bottom line is the fact that
your improved testing enables your company to consider completing development proj-
ects faster with no substantial increase in risk from the prior project testing perspective.
If your next defect tracking log looks like Project B, then you can draw a verti-
cal line at week 16 on the x -axis and predict how many more customer defects will
occur because testing will be stopped short of the usual 24-week schedule. You can
report back that completing Project B on week 16 will result in about 650 customer-
discovered defects, up from the 196 predicted for the 24-week completion date. Both
numbers are substantially higher than the 70 from the prior project. The bottom-line
is the fact that your less effective testing prohibits your company from considering
completing development projects faster.
12.7 THE RAYLEIGH CURVE—GUNSIGHTS FOR
DEFECT DISCOVERY PATTERNS
The fi rst analytical hurdle for a software development project team is to capture,
analyze, and compare enough defect discovery history that the in-progress project
comparisons become credible. The second analytical hurdle is to fi nd independent
measures that indicate how close the project team is getting to industry-wide defect
discovery rates. There is a dearth of published defect discovery rates because com-
panies are reluctant to publicly admit any defect numbers because of its potentially
damaging effect on the corporate image.
The only helpful source of industry-wide defect rates over the last 10 years has
been a mathematical formula called the Rayleigh curve. If you are immediately
skeptical that a mathematical formula could predict your company's defect patterns
taking into account all the nuances of how you develop and test software, your skep-
ticism is well founded and should not be suspended during this section. The reason
we examine the Rayleigh curve for defect analysis is because worthwhile results are
produced with intelligent use. “Intelligent use” is the key to success because any-
body can crank out a Rayleigh curve for a project using zero or less intelligence and
produce a meaningless curve.