Replication of Software Engineering Experiments - Empirical Software Engineering and Verification - page 62

run has moved a debate in ESE. There are researchers that recommend reusing
some of the baseline experiment materials to run replications [2, 26] with the
aim of assuring that the replications are similar and results can be compared.
There are researches who advise the use of different protocols and materials to
those employed in the baseline experiment [10, 27] with the aim of preserving
the principle of independence and preventing error propagation in replications
that use the same materials. Others suggest using alternative ways of verifying
the experimental results [28] with the aim of understanding the problems that
replication have had to date in SE experiments. This debate can probably be
put down to the fact that replication has still not satisfactorily tailored to ESE.
In this chapter we study the concept of replication with the aim of getting
a better understanding of its use in ESE. This chapter is organized as follows.
Section 2 describes the statistical perspective of replication. Section 3 discusses
replication in science. Section 4 reviews different types of replication accepted
in different experimental disciplines. Section 5 discusses the differences between
the concepts of replication and reproduction. Section 6 describes adequate varia-
tions in replication. Section 7 discusses some types of replications in SE. Section
8 presents the purposes that a replication can serves. Section 9 presents the
conclusions. Finally, Annex A lists and describes replication typologies found in
other disciplines.

2 Statistical Perspective of Replication

Sample size is an essential element in a controlled experiment. An adequate sam-
ple size increases the possibilities of the effect observed in the sample occurring
in the real population. The accuracy level of the results grows in proportion to
the sample size.
One of the commonly used coe cients for representing effect size observed in
an experiment is Cohen's d [29]. This coe cient is used to measure the differences
between the treatments studied in the experiment. The effect size indicates how
much better one treatment is compared to another. This coecient is usually used
with one-digit accuracy. For example [29], d =0.2 represents a small effect, d =0.5
indicates a medium effect or d =0.8 is a large effect. The sample size required to
satisfy a one-digit accuracy level can be calculated from (1): the function in (1) is
derived from (2) and (3), where the differences in the confidence intervals (left and
right) are equal at the specified accuracy level, in this case 0.1.

2+ d 2
2(0 . 0255102) 2

N = (1)

2 1 . 96 deviation ( d )=0 . 1 (2)

× ×

deviation ( d )= n 1+ n 2
n 1 n 2

d 2
2( n 1+ n 1)

+ (3)