Is Branch Coverage a Good Measure of Testing Effectiveness? - Empirical Software Engineering and Verification - page 193

Is Branch Coverage a Good Measure of
Testing Effectiveness?

Yi Wei 1 , Bertrand Meyer 1 , and Manuel Oriol 2

1 Chair of Software Engineering, ETH Zurich, Switzerland
{ yi.wei,bertrand.meyer } @inf.ethz.ch
2 Department of Computer Science, University of York, United Kindom
[e-mail]

Abstract. Most approaches to testing use branch coverage to decide on the qual-
ity of a given test suite. The intuition is that covering branches relates directly to
uncovering faults. The empirical study reported here applied random testing to
14 Eiffel classes for a total of 2520 hours and recorded the number of uncovered
faults and the branch coverage over time. For the tested classes, (1) random test-
ing reaches 93% branch coverage (2) it exercises almost the same set of branches
every time, (3) it detects different faults from execution to execution, (4) during
the first 10 minutes of testing, while branch coverage increases rapidly, there is a
strong correlation between branch coverage and the number of uncovered faults,
(5) over 50% of the faults are detected at a time where branch coverage hardly
changes, and the correlation between branch coverage and the number of uncov-
ered faults is weak.
These results provide evidence that branch coverage is not a good stopping
criterion for random testing. They also show that branch coverage is not a good
indicator for the effectiveness of a test suite.

Keywords: random testing, branch coverage, experimental evaluation.

1 Introduction

Various studies[11,4] show that random testing is an effective way of detecting faults.
Random testing is also attractive because it is easy to implement and widely applicable.
For example, when insufficient information is available to perform systematic testing,
random testing is more practical than any alternative [10]. Many practitioners think
that, to evaluate the effectiveness of a strategy, branch coverage -the percentage of
branches of the program that the test suite exercises - is the criterion of choice. It is
a weaker indicator of test suite quality than other coverage criteria such as predicate
coverage or path coverage [15]. Branch coverage is widely used because of its ease of
implementation and its low overhead on the execution of the program [18] under test. As
an example the European Cooperation for Space Standardization (ECSS) gives 100%
branch coverage as one of the measures to assure the quality of a critical software [6]. 1
Many practitioners and researchers dismiss random testing because it only achieves
low branch coverage. We used AutoTest [4], an automatic, random-based testing tool

1

Section 6.2.3.2.