Biology Reference
In-Depth Information
is not a reliable estimate of the standard error in that parameter. Rather, the permutation
test yields an estimate of the range of parameter values possible under the null model sim-
ulated by the test. In contrast, the standard deviation of the bootstrap estimates of the
same parameter yields a reliable estimate of its standard error because the bootstrap
resampling simulates a repetition of the process of selecting specimens from the popula-
tion (
Efron and Tibshirani, 1993
). When used for hypothesis testing, both methods tend to
give very similar results, so it is difficult (and perhaps unnecessary) to determine which
approach is preferable in most cases. To some extent, the choice between them appears to
be a matter of preference among writers of software. There are some reasons to think that
permutation tests may yield a more exact achieved significance level (ASL) than bootstrap
approaches (
Efron and Tibshirani, 1993; Good, 1994
), but this is at the cost of precluding
estimates of confidence intervals (or standard errors) on the statistics involved.
The Jackknife
Jackknife methods (
Quenouille, 1949; Tukey, 1958
) also preceded bootstrap methods
and, to some extent, have been supplanted by them. Jackknife estimates are obtained by
resampling such that one element is left out at a time. If there are N specimens in a sam-
ple, then it is possible to form N jackknife data sets, each with N
1 specimens. If we
2
again look at the set
C
:
C
5
f
C
1
;
C
2
;
C
3
;
C
4
;
C
5
g
(8A.27)
The five possible jackknife versions of
C
are:
5
f
C
2
;
;
;
C
5
g
C
J1
C
3
C
4
(8A.28)
C
J2
5
f
C
1
;
C
3
;
C
4
;
C
5
g
(8A.29)
C
J3
5
f
C
1
;
C
2
;
C
4
;
C
5
g
(8A.30)
C
J4
5
f
C
1
;
C
2
;
C
3
;
C
5
g
(8A.31)
C
J5
5
f
C
1
;
C
2
;
C
3
;
C
4
g
(8A.32)
Jackknife data sets will always be more similar to the original data set than bootstrap
sets are because bootstrapping offers a greater variety of ways of resampling the data. The
jackknife may be viewed as an approximation to the bootstrap (
Efron and Tibshirani,
1993
), and it is a good approximation when the changes in the statistic are smooth or lin-
ear with respect to changes in the data. The mean is a linear statistic, but the median is not
(because the median may change abruptly as observations are added or subtracted from
the sample). Therefore, jackknife and bootstrap estimates of the mean will not differ much
but estimates of the median may differ considerably.
There are some approaches to combining the bootstrap and the jackknife (see particu-
larly
Efron, 1992; Efron and Tibshirani, 1993
, Chapter 19, on assessing the error of boot-
strap estimates), but otherwise the jackknife appears to offer few advantages over the
bootstrap.
Cross-validation testing of models (
Manly, 1997
) is somewhat similar to jackknife test-
ing. Cross-validation is used to test the performance of predictive models, like regression,
