Replication of Software Engineering Experiments - Empirical Software Engineering and Verification - page 64

The results of a single execution of an experiment is threatened by type I
error 2 . Having more (internal) replications of the same experiment considerably
reduces this type of error. For example, if an experimenter establishes the sig-
nificance level α of an experiment at 0.05, which represents a 1:20 probability
of obtaining a chance result, the likelihood of again obtaining an accidental re-
sult drops to 1:400 (p = 0.05 × 0.05 = 0.0025) if the experiment is identically

internally repeated again.
The sample size of experiments run in SE is not large enough to accurately
estimate the effect size under study. Therefore, identical replications are required
to be able to estimate the effect size with any accuracy. However, identical repli-
cations are virtually impossible when they are carried out in other sites [25].

3 Replication in Science

In science, replication refers to the repetition of a previously run experiment.
Some definitions of replication in science are:

1. “Replication refers to a conscious and systematic repeat of an original study”
[33].
2. “Replication is traditionally defined as the duplication of a previously pub-
lished empirical study” [34].
3. “Replication is a methodological tool based on a repetition procedure that is
involved in establishing a fact, truth or piece of knowledge” [35].
4. “Replication - the performance of another study statistically confirming the
same hypothesis” [36].
5. “Replication is the repetition of the basic experiment. That is, replication
involves repeating an experiment under identical conditions, rather than
repeating measurements on the same experimental unit” [37].
6. “The deliberate repetition of research procedures in a second investigation
for the purpose of determining if earlier results can be repeated” [38].
7. “Is the process of going back, or re-searching an observation, investigation,
or experimentation to compare findings” [39].

The value of replication has been widely recognized in a number of scientific
disciplines. Popper [40] claimed that “We do not take even our own observations
quite seriously, or accept them as scientific observations, until we have repeated
and tested them. Only by such repetitions can we convince ourselves that we
are not dealing with a mere isolated 'coincidence', but with events which, on
account of their regularity and reproducibility, are in principle inter-subjectively
testable”. Hempel [41] realized the importance of reckoning with more than one

2 Type I error occurs when the null hypothesis is rejected while it is true, i.e. when
there is believed to be a significant difference between the treatments that the ex-
periment compares and there is, in actual fact, no such difference.