Replication of Software Engineering Experiments - Empirical Software Engineering and Verification - page 66

In most typologies, the authors give the replication types an original name.
They tend, therefore, to use their own terms to refer to a replication type. There
are some exceptions, like Kelly et al. [72], who use the same terms as are applied
in Lykken's [69] typology. In the identified typologies, we also find that there is
no intra- or inter-disciplinary standardization for naming replication types.
The identified typologies were found to have two purposes: 1) some authors
developed the typology to classify existing sets of replications; 2) other authors
generated the typology for no particular purpose. Within this purpose, some
authors illustrate the replication types using a number of existing replications,
whereas others develop the typology and use examples to describe the replication
types. Table 2 shows the possible usage of typologies.

Table 2. Typologies usage

Typologies generated for understanding replication types

Typologies generated
to classify existing
sets of replications

With examples of
real replications

With imaginary
examples

Bahr et al. [62] Lindsay and Ehrenberg [68] Hendrick [70]

Kelly et al. [72] Tsang and Kwan [66] Monroe [74]

Leone and Schultz [73] Kantowitz et al. [65] Radder [75]

Evanschitzky and
Armstrong [64]

Lykken [69] Easley et al. [61]

Hendrick [70] Hunter [31]

La Sorte [33] Schmidt [35]

Van IJzendoorn [63] Finifter [71]

Mittelstaedt and Zorn [67]

Examining the typologies, we found that experiment results were not always
verified by running the experiment over again. Neither did the replication always
repeat the experimental protocol of the baseline experiment. We have identified
three major groups of methods for verifying findings:

1. Follow the same experimental protocol used in the baseline experiment.
The degree of similarity between the replication and the baseline experiment
vary. For verification purpose some of the elements of the baseline experi-
ment can be changed or modified in the replication. For example, Tsang and
Kwan [66] use the term empirical generalization when the study is repeated
on different populations. Monroe [74] uses the term independent replication
when the study is repeated by different researchers.
This type of replication is used for different purposes. According to Lykken
[69], for example, the purpose of operational replication is to check that
the experimental recipe produces the same results with another researcher.
Tsang and Kwan's [66] empirical generalization purpose is to test the extent
to which the study results are generalizable to other populations.