Information Technology Reference
In-Depth Information
TABLE 1 The Results of 400 Experiments of Simulation 1.1
R
R
R
R
R
E(
)
SD(
)
RMSE
1
(
)
RMSE
2
()
apparent
0.0000
0.0000
0.1354
0.1006
cross
0.1039
0.1060
0.1381
0.1060
jack
0.0951
0.0864
0.1274
0.0865
boot
0.0786
0.0252
0.1078
0.0334
ideal
0.1006
0.0000
0.0906
0.0000
Note
: RMSE
1
is the root mean squared error about the true excess, and RMSE
2
is that
about the expected excess error. The expected excess error is
E
(
R
) for ideal.
Each of the six simulations consisted of 400 experiments. The results of
all 400 experiments of simulation 1.1 are summarized in Table 1. In each
experiment, we estimate the excess error
R
by evaluating the realized pre-
diction rule on a large number (5,000) of new observations. We estimate
the expected excess error by the sample average of the excess errors in the
400 experiments. To compare the three estimators, I first remark that in
the 400 experiments, the bootstrap estimate was closest to the true excess
error 210 times. From Table 1 we see that since
(
)
=
(
)
=
()
=
ˆ
ˆ
Er
0 1039
.
,
Er
0 0951
.
,
ER
0 1006
.
cross
jack
r
r
are all close,
cross
and
jack
are nearly unbiased estimates of the expected
excess error
E
(
R
), whereas
boot
with expectation
E
(
boot
) = 0.0786 is
biased downwards. [Actually, since we are using the sample averages of the
excess errors in 400 experiments as estimates of the expected excess errors,
we are more correct in saying that a 95% confidence interval for
E
(
cross
) is
(0.0935), 0.1143), which contains
E
(
R
), and a 95% confidence interval
for
E
(
jack
) is (0.0866, 1036), which also contains
E
(
R
). On the other
hand, a 95% confidence interval for
E
(
boot
) is (0.0761, 0.0811), which
does not contain
E
(
R
).] However,
corss
and
jack
have enormous standard
deviations, 0.1060 and 0.0864, respectively, compared to 0.0252, the
standard deviation of
boot
. From the column for RMSE
1
,
r
r
r
r
r
r
r
r
(
)
<
(
)
<
() () ()
ˆ
ˆ
ˆ
ˆ
ˆ
RMSE
r
RMSE
r
RMSE
r
~
RMSE
r
~
RMSE
r
,
1
ideal
1
boot
1
app
1
cross
1
jack
r
with RMSE
1
(
boot
) being about one-third of the distance between
RMSE
1
(
ideal
) and RMSE
1
(
app
). The same ordering holds for RMSE
2
.
Recall that simulations 1.1, 1.2, and 1.3 had the same underlying distri-
bution but differing sample sizes,
n
= 20, 40, and 60. As sample size
increased, the expected excess error decreased, as did the mean squared
error of the apparent error. We observed a similar pattern in simulations
2.1, 2.2, and 2.3, where the sample sizes were again
n
= 20, 40, and 60,
r
r
