Replication of Software Engineering Experiments - Empirical Software Engineering and Verification - page 72

Each specific aim of a replication type denotes an aspect of the experiment
that needs to be verified. The more experimental aspects or elements are verified,
the greater the confidence that the observed effect is not artifactual. An effect
observed in an experiment may not be observed at sites other than where it was
replicated, by other researchers, using other materials or methods or under other
conditions. Different replication types should be run to check that the different
experiment elements do not bias the observed findings and that the experiment
results are real.
Consequently, there are several degrees of similarity between a replication
and the baseline experiment. The changes serve different replication purposes.
Although the general purpose of a replication is to check a previously observed
finding, each replication type has special goals depending on what specific ele-
ment of the experiment is to be checked.
7 Types of Replications in SE
We did not find any specific research aiming to build a typology or classification
of replications in the field of ESE. We did locate, however, three works in our
discipline that classified replications as part of the research conducted.
The first piece of research is a master's thesis [81] that set out to study the use
of replication of controlled experiments in ESE. Almqvist [81] surveys 44 articles
describing 51 controlled experiments and 31 replications. He runs a systematic
review as a method for identifying relevant articles. In Chapter 4 of the thesis,
Almqvist [81] defines several categories for organizing the identified experiments.
In one of the categories, he develops a classification for categorizing the identi-
fied replications. Almqvist takes the replication types described by Lindsay and
Ehrenberg [68] as a reference and adds internal and external replication .Onthis
basis, he defines the following four types of replications:
1. Similar-external replications.
2. Improved-internal replications.
3. Similar-internal replications.
4. Differentiated-external replications.
The second classification is found in an article by Basili et al. [2], presenting
a framework for organizing sets of related studies. This article describes the
different aspects of the framework being one of these aspects a classification of
replications composed of three major categories, where two of these categories
define several types of replications. Basili et al. [2] illustrate the classification
with examples of different replications that they have run. The classification is
composed of a total of six replication types:
1. Strict replications.
2. Replications that vary the manner in wich the experiment is run.
3. Replications that vary variables intrinsic to the object of study.
4. Replications that vary variables instrinsic to the focus of the evaluation.