Replication of Software Engineering Experiments - Empirical Software Engineering and Verification - page 70

In this respect, the concept of replication given by Cartwright [77] would fit the
first form of verification described in the previous section, whereas the concept
of reproduction adheres to the second form of verification that we identified in
the replication typologies.

6 Variation among Replications

Based on the different replication types that we have found, replications appear
to fall into three groups:

1. Replications that vary little or not at all with respect to the baseline exper-
iment.
2. Replications that do vary but still follow the same experimental protocol as
the baseline experiment.
3. Replications that use different experimental protocol to check the baseline
experimental results i.e. reproductions.

Tables 6 and 7 list the replication types that fall into these first two groups. The
third group corresponds with the second type of verification presented in section
4 (Use a different experimental protocol to the baseline experiment).

Table 6. Replications with few or no variations that adhere to the baseline experiment

Replication Type Author(s)

Close Replication Lindsay and Ehrenberg [68]

Direct Replication Schmidt [35]; Kantowitz et al. [65]

Exact Replication Van IJzendoorn [63]; Tsang and Kwan [66]

Experimental Replication Leone and Schultz [73]

Literal Replication Lykken [69]; Kelly et al. [72]

Real Replications Evanschitzky and Armstrong [64]

Reproducibility of the material reali-
zation of an experiment

Radder [75]

Sequential Replication Monroe [74]

Statistical Replication Hunter [31]

Strict Replication Hendrick [70]; Monroe [74]

Type 0 Easley et al. [61]

Types A..D Bahr et al. [62]

Type I Easley et al. [61]

Based on the descriptions of the replications, it appears that a replication can
have different levels of similarity to the baseline experiment. In other words, the
elements of the experiment structure do not necessarily have to be the same in
the replication. Table 8 shows some experimental elements that, according to
the typologies we have found, do not necessarily have to be the same in each
replication. Note that the type (or aim) of the replication differs depending on
this change.