Replication of Software Engineering Experiments - Empirical Software Engineering and Verification - page 68

Table 4. Using a different experimental protocol

Replication Type Author(s)

Conceptual Extension Tsang and Kwan [66]

Conceptual Replication Hendrick [70]; Schmidt [35]; Kantowitz et al. [65]

Constructive Replication Lykken [69]; Kelly et al. [72]

Corroboration Leone and Schultz [73]

Differentiated Replication Lindsay and Ehrenberg [68]

Generalization and Extension Tsang and Kwan [66]

Reproducibility of the result of
an experiment

Radder [75]

Systematic Replication Finifter [71]

Theoretical Replication La Sorte [33]

Type III Easley et al. [61]

Types I..P Bahr et al. [62]

3. Use existing data sets from a previous experiment to reanalyse the data
employing either the same analysis procedures or others. This modus
operandi is useful for verifying whether errors were made during the data
analysis stage or whether the outcomes are affected by any particular data
analysis technique. Some replication types reanalyse the statistical models
instead of the existing study data. Different names are used for this type of
verification. For example, La Sorte [33] calls it internal replication ; Finifter
[71] terms it pseudoreplication , and Tsang and Kwan [66] describe it as check-
ing of analysis and reanalysis of data . Table 5 shows the replication types
we identified that fall into this category.

Table 5. Reanalyzing existing data

Replication Type Author(s)

Checking of Analysis Tsang and Kwan [66]

Complete Secondary Analysis Van IJzendoorn [63]

Data Re-analyses Evanschitzky and Armstrong [64]

Internal Replication La Sorte [33]

Pseudoreplication Finifter [71]

Reanalysis of Data Tsang and Kwan [66]

Restricted Secondary Analysis Van IJzendoorn [63]

Types I, II Mittelstaedt and Zorn [67]

If we want one term to identify each of the three forms of verification, we would
surely refer to the third one as re-analysis , because the descriptions clearly allude
to this term. However, the naming of the other two forms causes some confusion.
Do both forms adhere to the concept of replication, or does each one introduce
a different concept? The authors of some of the articles that we consulted to
identify the typologies use the terms replication and reproduction indistinctly.
This led us to examine whether these two concepts are equivalent or different.