Replication of Software Engineering Experiments - Empirical Software Engineering and Verification - page 74

4. Construct Operationalizations . Operationalizations describe the act of trans-
lating a construct into its manifestation. In a controlled experiment we have
cause and effect operationalizations. The cause operationalizations repre-
sent the primary treatments to be evaluated in the experiment (independent
variables) whereas the effect operationalizations represent the response vari-
ables (dependent variables) used to measure the effects of the treatments.
Both types of operationalization contain elements that can be varied in a
replication.
5. Population Properties . In SE experiments there are at least two populations
that are worth generalizing: the subjects and the experimental objects with
which subjects work or interact during the experiment. The generalization
takes place when the replication changes the properties of the subject or the
experimental objects.
Based on the elements that may vary in a replication, we identify the following
purposes of a replication in ESE:
1. Control for Sampling Error . If the basic elements of the baseline experiment
structure are kept unchanged, the purpose of the replication is to verify that
the results output by that experiment are not chance outcomes. This function
is useful for verifying that the effect identified in the baseline experiment is
not due to a Type-I error.
2. Control for Experimenters. If different experimenters run the replication,
then it aims is to verify that the experimenters do not influence the results.
3. Control for Site. If the replication is run at another site, then it aims is to
verify that the results are independent of the site where the experiment is
run.
4. Control for Artifactual Results . If the experimental protocol is changed, the
purpose of the replication is to verify that the observed results are not arti-
factual, that is, they reflect reality and are not a product of the experimental
protocol setup.
5. Determine Limits for Operationalizations . If the operationalizations are
changed a replication aims to determine the range of variation of the primary
treatments (independent variables) and the measures (dependent variables)
used to gauge the effects of the treatments.
6. Determine Limits in the Population Properties . If the population properties
are changed, the purpose of the replication is to determine the types of
experimental subject or objects to which the results of the replication hold.
9 Conclusions
Replication plays an important role in scientific progress where facts are at least
as important as ideas [31]. Experiments have to be replicated to identify ev-
idences. If we want to build up a SE body of knowledge based on empirical
evidence, different types of replications have to be run. In this chapter we have