Replication of Software Engineering Experiments - Empirical Software Engineering and Verification - page 69

5 Replication vs. Reproduction
According to the typologies we found, most researchers use the term replication to
refer to the repetition of an experiment, although some use the term reproduction
or reproducibility to describe this repetition. So it seems that many researchers
consider the two terms to be synonyms. Likewise, Wikipedia uses these terms in-
distinctly and defines reproducibility as “one of the main principles of the scientific
method, and refers to the ability of a test or experiment to be accurately repro-
duced, or replicated, by someone else working independently” [76].
Some researchers, however, do make a distinction between the two terms.
Cartwright [77], for example, suggests that replicability “doing the same exper-
iment again” should be distinguished from reproducibility “doing a new experi-
ment”. For Cartwright [77] the replication of an experiment refers to repeating
a new experiment very closely following the experimental protocol used in the
previous experiment, whereas reproduction refers re-examining a previously ob-
served result using a different experimental protocol to what was employed in
the previous experiment.
According to Cartwright [77], replication does not guarantee that the observed
result represents the reality under observation. The result can be artifactual,
i.e. a product of the materials or the instruments used in the experiment. To
guarantee that the result is consistent with the reality under observation, we have
to undertake a reproduction using different experimental protocols to ensure that
the observed result is independent of the procedure, materials or instruments
used in the experiments that arrived at the result.
When the results are repeatable using the same experimental protocol, the
experimenters can be confident that they have observed some sort of phenomenon
that is stable enough to be observed more than once. But, as it was observed
using the same experimental protocol, there could be a very close relationship
between the protocol and the phenomenon. As Radder put it [78], “[this result]
does not imply any agreement about what the phenomenon is. Some interpreters
may even argue that the phenomenon is an artifact, because, though it is stable,
it is not to be attributed to the object under study but to certain features of
the apparatus”, where the term apparatus refers to the instruments, materials
or procedures used, i.e. the experimental protocol. Cartwright [77] claims that
“reproducibility, then, is a guard against errors in our instruments” in such a
situation. According to Cartwright [77], though, reproduction is not absolutely
necessary, as the better designed the instruments (apparatus) are, the less likely
it is to have to use reproducibility.
Reproduction can be seen as a sort of triangulation, where the experimenters
use different experimental protocols in an attempt to validate or corroborate
the findings of the previous experiment [79]. According to Park [80], “These
triangulation strategies can be used to support a conceptual finding, but they
are not replications of any degree”.