Information Technology Reference
In-Depth Information
Table 1 is the number of testable branches, obtained by subtracting dead branches from
the total number of branches in the corresponding class.
Test runs.
We tested each class in
hour
long. This supposedly made the test runs long enough so that branch coverage level
reaches a plateau. But we found out that even after
30
runs with different seeds with each run
6
hours, random testing is still
capable of exercising some new branches with a very low probability. We chose
16
hour
runs because the branch coverage level already increases very slowly after that, and
because
6
6
hours corresponds to an overnight testing session.
Computing infrastructure.
We conducted the experiment on 9 PCs with Pentium 4 at
3.2GHz, 1GB of RAM, running Linux Red Hat Enterprise 4. The version of AutoTest
used in the experiment is modified to include instrumentation for monitoring the branch
coverage. AutoTest was the only CPU intensive program running during testing.
3R su s
This section presents results that answer five main questions:
1. Is the level of the branch coverage achieved by random testing predictable?
2. Is the branch coverage exercised by random testing similar from one test run to
another?
3. Is the number of faults discovered by random testing predictable?
4. Are the faults uncovered by different test runs similar?
5. Is there a correlation between the level of coverage and the number of faults
uncovered?
3.1
Predictability of Coverage Level
Because AutoTest might not be able to test all branches of a class due to its random
nature, it is unlikely that testing sessions achieve total coverage, or even just constant
results over all tested classes. As an example, it might be extremely difficult to sat-
isfy a complex precondition guarding a routine with such a random approach. Another
example is that the visibility of a routine might not let AutoTest test it freely.
This intuition is confirmed by the results presented in Figure 1 which shows the
median of the branch coverage level for each class over time. The branch coverage
level ranges from
. As a first result, we can see that the branch coverage of some
classes reaches a plateau at less than
0
to
1
0
.
85
while most of them have a plateau at or above
0
.
9
. The thick curve in Figure 1 is the median of medians of the branch coverage level
of all the classes. Over all
14
classes, the branch coverage level achieved after
6
hours
of testing ranges from
0
.
82
to
0
.
98
. On average, the branch coverage level is
0
.
93
, with
a standard deviation of
of the median.
While the maximum coverage is variable from one class to another, the actual evolu-
tion of branch coverage compared to the maximum coverage achieved through random
testing is similar:
0
.
04
, corresponding to
4
.
67%
93%
of all exercised branches are exercised in the first
10
minutes,
96%
in
30
minutes, and
97%
in the first hour. Section 4 contains an analysis of branches
not exercised.
Search WWH ::
Custom Search