Information Technology Reference
In-Depth Information
Ta b l e 3 .
Root mean squared errors of the replication prediction methods for different fixed dis-
tributions (mean values of 100 runs)
μ
1
=
20
,
sd
1
=
2
,
μ
2
=
21
,
sd
2
=
2
Approach 5
10
15
20
25
30
35
40
45
50
NLS
179
.
7 157
.
1 192
.
1 77
.
7 153
.
5 55
.
3 102
.
6 62
.
0 145
.
2 149
.
1
Power
282
.
1 223
.
5 166
.
9 125
.
2 102
.
4 167
.
8 155
.
6 103
.
6 97
.
9 27
.
6
#invalid
25
13
9
9
7
6
5
6
6
5
Approach 55
60
65
70
75
80
85
90
95
NLS
153
.
1 101
.
9 129
.
3 100
.
0 80
.
7 70
.
9 65
.
0 60
.
6 57
.
1
Power
96
.
1 38
.
3 26
.
8 24
.
9 25
.
5 25
.
3 24
.
8 25
.
7 26
.
3
#invalid
3
3
2
2
1
1
1
1
1
μ
1
=
20
,
sd
1
=
2
,
μ
2
=
22
,
sd
2
=
2
Approach 5
10
15
20
25
30
35
40
45
50
NLS
160
.
1 91
.
1 74
.
8 134
.
5 38
.
5 22
.
0 18
.
0 15
.
9 14
.
7 13
.
8
Power
237
.
4 145
.
0 13
.
8 11
.
3 9
.
2
8
.
0
8
.
5
8
.
5
8
.
8
9
.
1
#invalid
13
8
3
2
2
2
1
1
1
0
Approach 55
60
65
70
75
80
85
90
95
NLS
13
.
3 12
.
8 12
.
5 12
.
2 12
.
0 11
.
8 11
.
7 11
.
5 11
.
4
Power
9
.
2
9
.
2
9
.
2
9
.
3
9
.
2
9
.
3
9
.
3
9
.
4
9
.
4
#invalid
0
0
0
0
0
0
0
0
0
μ
1
=
20
,
sd
1
=
2
,
μ
2
=
23
,
sd
2
=
2
Approach 5
10
15
20
25
30
35
40
45
50
NLS
115
.
4 103
.
3 8
.
1
6
.
3
5
.
6
5
.
2
4
.
9
4
.
8
4
.
6
4
.
6
Power
42
.
5 4
.
1
3
.
1
3
.
1
3
.
3
3
.
5
3
.
5
3
.
6
3
.
6
3
.
6
#invalid
6
5
5
5
5
4
4
4
4
4
Approach 55
60
65
70
75
80
85
90
95
NLS
4
.
5
4
.
4
4
.
4
4
.
4
4
.
3
4
.
3
4
.
3
4
.
2
4
.
2
Power
3
.
6
3
.
6
3
.
7
3
.
7
3
.
7
3
.
7
3
.
7
3
.
7
3
.
7
#invalid
4
4
4
4
4
4
4
4
4
sample sizes,
α
=0
.
05
, a fixed power value of
0
.
8
, and the one-sided test setting.
The result is an estimation how many samples are needed.
We apply the both prediction methods to the same fixed distributions as in Section 5.1
and capture the root mean squared error (RMSE). As both methods generate unrealistic
high replication estimations in some cases, we have introduced a maximal threshold.
Whenever this threshold (
1000
in our experiments) is exceeded, the corresponding value
is set to the threshold value. Additionally, we count how many times no interception
point could be computed for the NLS method (marked with “#invalid”). The results of
these experiments are shown in Table 3. One graph of the second setting (
μ
1
=20
,
sd
1
=2
vs.
μ
2
=22
,
sd
2
=2
) is shown in Figure 7.
The experimental results do not identify one of the methods as better. Depending on
the number of p values taken into account and depending on the different distributions,
one or the other method leads to a lower RMSE. A direct comparison is not really
possible, as the NLS method leads to invalid values in some cases. Especially, if only
few values are used, the regression does not lead to a valid interception point (25 out
100 for the first setting and 5 p values). For the first two distribution pairs (those with a
higher overlap) and low numbers of p values (5 and 10), the NLS method leads to better
mean error of the 100 performed runs. Early prediction results are of special interest as
it allows for an early intervention (of the system or user).
