Information Technology Reference
In-Depth Information
Before leaving the chronic hepatitis data, I mention that other predic-
tion rules might be used. Examples include more complicated forms of
variable selection such as best subset regression and alternative models
such as discriminant analysis. Friedman (1977) applied recursive partition-
ing to these data to obtain a binary-decision tree. I chose to focus atten-
tion on the rule based on forward logistic regression because it is the rule
actually proposed and used by Gregory, the experimenter. The question of
choosing an optimal prediction rule was not my goal.
4. THE PERFORMANCE OF CROSS-VALIDATION, THE
JACKKNIFE, AND THE BOOTSTRAP IN SIMULATIONS
In the previous section we saw the cross-validation, jackknife, and boot-
strap estimates of expected excess error for Gregory's rule. These estimates
give bias corrections to the apparent error. Do these corrections offer real
improvements? Introduce the “estimators”
app
∫ 0, the zero-correction
estimate corresponding to the apparent error, and
ideal
∫
E
(
R
), the best
constant estimate if we knew the expected excess error
E
(
R
). To compare
cross
,
jack
,
boot
against these worst and best cases
app
and
ideal
, we
perform some simulations.
To judge the performance of estimators in the simulations, we use two
criteria:
r
r
r
r
r
r
r
(
)
12
()
=
[
]
2
ˆ
ˆ
RMSE
1
RERR
-
,
the root mean squared error (RMSE) about the excess error, and
(
)
12
()
=
[
]
2
ˆ
ˆ
-
()
RMSE
2
RERER
,
the root mean squared error about the expected excess error. Notice that
since
[
]
=
2
[
(
)
-+
(
)
]
2
ˆ
ˆ
ˆ
ˆ
ˆ
ERREq
-
+
Rq
R
,
app
app
R
RMSE
1
( ) also measures the performance of the bias-corrected estimate
app
+ as an estimator of the true error
app
+
R
.
I pause to clarify the distinction between excess error and expected
excess error. In the chronic hepatitis problem, the training sample that
Gregory observed led to a particular realization (3.2). The excess error is
the difference between the true and apparent error of the realized rule h
based on this training sample. The expected excess error averages the
R
q
q
F
