Cross-Validation, the Jackknife, and the Bootstrap: Excess Error Estimation in Forward Logistic Regression - Common Errors in Statistics - page 183

A

C

1.1 J

B

1.2

1.3

2.1

2.2

2.3

0.00 0.04 0.08 0.12 0.16 0.20

FIGURE 2 95% (nonsimultaneous) Confidence Intervals for RMSE 1 . In each set
of simulations, there are five confidence intervals for, respectively, apparent (A),
cross-validation (C), jackknife (J), bootstrap (B), and ideal (1) estimates of the
excess error. Each confidence interval is indicated by — —. The middle vertical bar
in each confidence interval represents the value of the estimate.

and the dimension of t i was increased to p = 6 and S, b 0 , and b given in
(4.3). For larger sample sizes, bias corrections to the apparent error
became less important. It is still interesting, however, to compare mean
squared errors. For all six simulations, I plot RMSE 1 's in Figure 2 and
RMSE 2 's in Figure 3. It is interersting to note that the ordering noticed in
simulation 1.1 of the root mean squared error of the five estimates also
held in the other five simulations. That is,

() () ()

ˆ ˆ ˆ

RMSE r ~ RMSE r ~ RMSE r ,

1 app 1 cross 1 jack

r

and RMSE 1 ( boot ) is about one-third of the distance between
RMSE 1 ( ideal )and RMSE 1 ( app ). Similar remarks hold for RMSE 2 . Cross-
validation and the jackknife offer no improvement over the apparent error,
whereas the improvement given by the bootstrap is substantial.
The superiority of the bootstrap over cross-validation has been observed
in other problems. Efron (1983) discussed estimates of excess error and

r r