Replication of Software Engineering Experiments - Empirical Software Engineering and Verification - page 83

applied. Larger variations may lead to more interesting discoveries in addition
to the original study, but they will be followed by smaller variations if more
global replications fail to produce new “facts”. If even modest variations fail to
reproduce the results, a more or less exact replication is needed.

A.8 Kantowitz et al. [65]

Direct Replication. This is the attempt to repeat the experiment as closely as is
practical, with as few changes as possible in the original method.

Systematic Replication. The experimenter attempts to vary factors believed to
be irrelevant to the experimental outcome. If the phenomenon is not illusory,
it will survive these changes. If the effect disappears, then the researcher has
discovered important boundary conditions on the phenomenon being studied.

Conceptual Replication. One attempts to replicate a phenomenon, but in a way
radically different from the original experiment.

A.9 Kelly et al. [72]

Literal Replication. The earlier findings may be reexamined using the same ma-
nipulations (independent variables, experimental procedures, etc.) and measures
(dependent variables, methods of data analysis, etc.).

Operational Replication. If the experimenter wishes to vary criterion measures,
the experiment would be termed an operational replication. In this instance,
the dependent variable would represent a different operationalization of the con-
struct; the essential conceptual meaning would remain unchanged.
Instrumental Replication. This replication is carried out when the dependent
measures are replicated and the experimental manipulations are varied. Varia-
tions in the implementation of experimental procedures which do not go beyond
the originally established relationship would be included in this category.

Constructive Replication. A constructive replication attempt may be identified
when both manipulations and measures are varied. This replication involves the
attempt to achieve equivalent results using an entirely original methods recipe.

A.10 La Sorte [33]

Retest Replication. In its general form retest replication is a repeat of an original
study with few if any significant changes in the research design. The retest has
two major purposes: 1) it acts as a reliability check of the original study, and
2) inconsistencies and errors in procedure and analysis can be uncovered in the
repeat. Although the retest increases one's confidence that the findings are not
artifactual, it does not eliminate the possibility of error in process, especially
when the same investigator conducts both studies.

Internal Replication. The differences between the retest and internal replica-
tion are mainly procedural. Instead of seeking confirmation of an original study,