Replication of Software Engineering Experiments - Empirical Software Engineering and Verification - page 60

Replication of Software Engineering Experiments

Natalia Juristo and Omar S. Gomez

Facultad de Informatica,
Universidad Politecnica de Madrid,
Boadilla del Monte 28660, Madrid, Espana
[e-mail], [e-mail]

Abstract. Experimentation has played a major role in scientific
advancement. Replication is one of the essentials of the experimental
methods. In replications, experiments are repeated aiming to check their
results. Successful replication increases the validity and reliability of the
outcomes observed in an experiment.
There is debate about the best way of running replications of Soft-
ware Engineering (SE) experiments. Some of the questions that have
cropped up in this debate are, “Should replicators reuse the baseline ex-
periment materials? Which is the adequate sort of communication among
experimenters and replicators if any? What elements of the experimental
structure can be changed and still be considered a replication instead of
a new experiment?”. A deeper understanding of the concept of replica-
tion should help to clarify these issues as well as increase and improve
replications in SE experimental practices.
In this chapter, we study the concept of replication in order to gain
insight. The chapter starts with an introduction to the importance of
replication and the state of replication in ESE. Then we discuss replica-
tion from both the statistical and scientific viewpoint. Based on a review
of the diverse types of replication used in other scientific disciplines, we
identify the different types of replication that are feasible to be run in
our discipline. Finally, we present the different purposes that replication
can serve in Experimental Software Engineering (ESE).

Keywords: Experimental Replicaction, Types of Replication, Experi-
mental Software Engineering, Empirical Software Engineering.

1 Introduction

Experimentation should be an indispensable part of SE research. As Tichy says
[1], “Experimentation can help build a reliable base of knowledge and thus re-
duce uncertainty about which theories, methods, and tools are adequate”. Basili
[2] claims that “Experimental SE is necessary, common wisdom, intuition, spec-
ulation, and proofs of concepts are not reliable sources of credible knowledge”.

This work has been performed under research grant TIN 2008-00555 of the Span-
ish Ministry of Science and Innovation, and research grant 206747 of the Mexico's
National Council of Science and Technology (CONACyT).