Information Technology Reference
In-Depth Information
0.40
0.40
1
0.30
0.30
1
1
1
2
1
1
jack
0.20
1
boot
0.20
2
1
7
1
1
1
2
1
2
1
5
1
4
1
5
2
3
0.10
0.10
1
1
1
1
5
*
5
9
*
*
1
0.00
0.00
*
0.00
0.10
0.20
0.30
0.40
0.00
0.10
0.20
0.30
0.40
cross
cross
FIGURE 4
Scatterplots to Compare
corss
,
jack
and
boot
. The scatterplots sum-
marize the relationships among the three estimates for the first 100 experiments
of simulation 1.1. The numerals indicate the number of observations; * indicates
greater than 9.
r
r
r
6. CONCLUSIONS
Because complicated prediction rules depend intricately on the data and
thus have grossly optimistic apparent errors, error rate estimation for com-
plicated prediction rules is an important problem. Cross-validation is a
time-honored tool for improving the apparent error. This article compares
cross-validation with two other methods, the jackknife and the bootstrap.
With the help of increasingly available computer power, all three methods
are easily applied to Gregory's complicated rule for predicting the
outcome of chronic hepatitis. Simulations suggest that whereas the jack-
knife and cross-validation do not offer significant improvement over the
apparent error, the bootstrap shows substantial gain.
REFERENCES
Efron B.
The Jackknife, the Bootstrap, and Other Resampling Plans
, Philadelphia:
Society for Industrial and Applied Mathematics 1982.
—— “Estimating the Error Rate of a Prediction Rule: Improvements on Cross-
Validation,”
Journal of the American Statistical Association
, 1983;
78
:316-331.
Friedman JR. “A Recursive Partitioning Decision Rule for Nonparametric Classifi-
cation,”
IEEE Transactions on Computers
, C-26, 1977; 404-408.
Geisser S. “The Predictive Sample Reuse Method With Applications,”
Journal of
the American Statistical Association
, 1975;
70
:320-328.
Gong G. “Cross-Validation, the Jackknife, and the Bootstrap: Excess Error Estima-
tion in Forward Logistic Regression,” unpublished Ph.D. thesis, Stanford Uni-
versity 1982.
Rao CR.
Linear Statistical Inference and Its Applications
, New York: John Wiley
1973.
Stone M. “Cross-Validatory Choice and Assessment of Statistical Predictions,”
Journal of the Royal Statistical Society
, 1974;
36
:111-147.
