Information Technology Reference
In-Depth Information
For effect sizes
d
=0.2,
d
=0.5 and
d
=0.8, a sample size of
N
=1,567,
N
=1,729
and
N
=2,028 is required, respectively. Fig. 1 shows the graph of the resulting
function in (1).
N
4000
3000
2000
1000
d
2
1
0
1
2
Fig. 1.
Sample (
N
) necessary for a particular effect size (
d
) with one-digit accuracy
To be able to estimate effect sizes with one-digit accuracy, we need to repeat
the same experiment to increase the sample size and reach the required level
showed in Fig. 1. In the set of controlled SE experiments examined by Dyb a
et al. [30], the average sample size of the samples used in these experiments is
N
=55 (55 observations per experiment).
For an average sample size of 50 observations, the same study would have
to be repeated 31 times to satisfy the sample size required for an effect size of
d
=0.2; the same study would have to be repeated 34 and 40 times, respectively,
to get an effect size of
d
=0.5 and
d
=0.8. Consequently, experiment repetitions
have to be equal. For increasing sample size the replications have to measure
the independent and dependent variables in exactly the same manner, using
exactly the same experimental protocol, and they should all sample the same
populations [31].
Since experimental conditions are hard to control in ESE, one option worth
considering to satisfy the statistical requirement of identical repetitions is run-
ning internal replications (at the same site and by the same experimenters) of
SE experiments. Through internal repetitions, the sample comes closer to the
interval of observations [1,537; 2,305] required to be confident that the observed
effect (from 0, none; to 1
1
, very large) occurs not only in the sample used in the
experiment but also in the real population.
1
Note that effect size over 1 is possible. In fact Kampenes et al. [32] show that 32%
of the experiments published in SE have an effect size greater than 1. The bigger
the effect size the bigger the sample.
Search WWH ::
Custom Search