Replication of Software Engineering Experiments - Empirical Software Engineering and Verification - page 73

5. Replications that vary context variables in the environment in wich the so-
lution is evaluated.
6. Replications that extend the theory.
The third classification is found in a research conducted by Krein and Knut-
son [82]. The paper presents a framework for organizing research methods in
SE. Krein and Knutson [82] define a replication taxonomy with four types of
replications:
1. Strict replication. Which is meant to replicate a prior study as precisely as
possible.
2. Differentiated replication. Which intentionally alters aspects of the prior
study in order to test the limits of that study's conclusions.
3. Dependent replication. Which is a study that is specifically designed with
reference to one or more previous studies, and is, therefore, intended to be
a replication study.
4. Independent replication. Which addresses the same questions and/or hy-
potheses of a previous study, but is conducted without knowledge of, or
deference to, that prior study either because the researchers are unaware of
the prior work, or because they want to avoid bias.
Other ESE works mention replication types but do not refer to any classification.
For example, Brooks et al. [83] and Mendonca et al. [84] mention differences
between internal and external replication . Shull et al. [26] discuss some types
of replications ( exact , independent , dependent and conceptual replications )to
describe the role that they play in ESE. Finally, Lung et al. [85] mention two
types of replication ( literal and theoretical replication ) to explain the type of
replication that they ran, and Mandic et al. [86] discuss two types of replications,
namely, exact or partial replications , and replications designed to improve the
goal of the original experiment.
8 Purposes of Replication in ESE
The elements of an experiment to be replicated vary depending on the purpose
of the replication. We have identified five elements that can vary in a replication:
1. Experimenters . The experimenters in a replication can be the same peo-
ple as participated in the baseline experiment, different experimenters or a
mixture of both, though some cooperation between the baseline experiment
researchers and the replicators.
2. Site . The replication can be run at the same site as the baseline experiment
or at another place.
3. Experimental Protocol . This term refers to the experimental design, instru-
ments, materials, experimental objects, forms and procedures used to run an
experiment. The experimental protocol is how these elements are set up for
use by the experimenter to observe the effects of the treatments. Different
elements of the protocol can be changed in a replication.