Replication of Software Engineering Experiments - Empirical Software Engineering and Verification - page 75

studied the concept of replication as it is used in other scientific disciplines with
the aim of getting a better understanding of this mechanism.
Although we identified several replication typologies, replication types are not
standardised at either the intra or interdisciplinary level. Some authors use the
same replication name, although they each define the replication differently. Also
authors use different replication names to refer to equivalent replications types.
Several of the different replication types that we have found describe changes
of the structure of the experiment to be replicated. That is, replication can
have different levels of similarity to the baseline experiment. The changes to the
experiment in a replication are linked with the verification purposes. Although
the aim of a replication is to verify the experimental outcomes, a replication has
specific purposes depending on which elements in the experiment are varied.
All different replication purposes havetobereachedandsatisfiedinorder
for an experiment result to be considered verified. A systematic approach where
different types of replications are planified can help experimenters to advance
step by step in the verification path.
Discovering new conditions influencing the results of the experiments (and
thus software development) is an important co-lateral effect of replications. With
a better understanding of these conditions, we will be able to assemble the small
segments learnt in systematically varied replications to put together a piece of
knowledge.
References
1. Tichy, W.: Should Computer Scientists Experiment more? Computer 31(5), 32-40
(1998)
2. Basili, V., Shull, F., Lanubile, F.: Building Knowledge through Families of Exper-
iments. IEEE Transactions on Software Engineering 25(4), 456-473 (1999)
3. DeMarco, T.: Software Engineering: An Idea Whose Time has Come and Gone?
IEEE Software 26(4), 95-96 (2009)
4. Meyer, B.: Credible Objective Answers to Fundamental Software Engineering
Questions. LASER Summer School on Software Engineering (2010)
5. Meyer, B.: Empirical Research: Questions from Software Engineering. In: 4th Inter-
national Symposium on Empirical Software Engineering and Measurement (ESEM
2010) (2010)
6. Brinberg, D., McGrath, J.E.: Validity and the Research Process, p. 176. Sage
Publications, Inc. (June 1985)
7. Hayes, W.: Research Synthesis in Software Engineering: A Case for Meta-Analysis.
In: METRICS 1999: Proceedings of the 6th International Symposium on Software
Metrics, p. 143. IEEE Computer Society (1999)
8. Miller, J.: Can Results from Software Engineering Experiments be Safely Com-
bined? In: METRICS 1999: Proceedings of the 6th International Symposium on
Software Metrics, p. 152. IEEE Computer Society (1999)
9. Miller, J.: Applying Meta-analytical Procedures to Software Engineering Experi-
ments. J. Syst. Softw. 54(1), 29-39 (2000)
10. Miller, J.: Replicating Software Engineering Experiments: A poisoned Chalice or
the Holy Grail. Information and Software Technology 47(4), 233-244 (2005)