A Quantitative Analysis into the Economics of Correcting Software Bugs - Computational Intelligence in Security for Information Systems - page 200

2010 during a series of Audits of both code security and system security code associ-
ated with [14]. The code projects came from a combination of financial services com-
panies, media companies and web development firms. The data will be released
online by the authors.
It is clear from the results that there is an optimized ideal for software testing. Fig.
1 demonstrates the costs of testing and notes how each subsequent bug costs more to
find than the previous one. The costs of finding bugs go up as the cost of software is
tested to remove more bugs.
It has been noted that “ there is a clear intuitive basis for believing that complex
programs have more faults in them than simple programs ” [9]. As the size and hence
complexity of the code samples increased, the amount of time and costs required to
write and debug the code increased (Fig. 2). What was unexpected was that the num-
ber of bugs/LOC did not significantly change as the size of the program increased
(Fig. 3). Rather, there was a slight, but statistically insignificant decline in the number
of bugs noted in more complex programs per line of code. So whilst the number of
bugs did increase, this occurred in a linear fashion to the cost increase which occurred
exponentially.
The calculated number of hours per line of code (Fig. 2) increased exponentially
with an exponent of around 1.56. In this study, the largest program sampled was ap-
proximately 300,000 SLOC (source lines of code). This relates directly to complexity
with longer programs costing more both in time and money.
A financial calculation of internal costs of mitigating the bugs was also conducted
based on the bugs found within a year of the code being released. This period was
selected as it comes to reason that if the bug has not been found in a 12 month period,
it would be expensive to find.
The results of the analysis of the data demonstrated that the costs of testing can be
analyzed. In Fig. 1, the cost (calculated as a function of analysis time) of finding each
additional bug is exponentially more expensive than the last to find. As a conse-
quence, this also increases the mean cost of the project. The more bugs are sought, the
higher the cost. This is offset against the costs of fixing bugs in the field later in the
paper.
Of particular interest is the distribution of bugs as a percentage of code. We can see
that there is no strong correlation between the levels of bugs in code and the length of
the code (R 2 = -14.48 ). There are more bugs in large code, but the number of bugs
per line does not increase greatly.
The distribution of bugs was fairly even for all sizes of code in the study and was
positively skewed. The numbers of bugs discovered was far too high. The reported
numbers of bugs in large commercial software releases average around 4% [9]. The
mean (Fig. 4) for the study was 5.529% (or 55 errors per 1,000 lines of source code).
Many of these are functional and did not pose security threats, but the economics of
repairing the remaining bugs remains.
We need to move from " Lines of Code per day " as a productivity measure to a
measure that takes debugging and documentation into account. This could be some-
thing such as " Lines of clean, simple, correct, well-documented code per day " [3].
This also has problems, but it does go a long way towards creating a measure that
incorporates the true costs of coding.