Biology Reference
In-Depth Information
slightly different confidence level, even changing by several percentage points, but it is
highly unlikely to yield p
0.05. Similarly, in such a bootstrap t-test, if the difference in
bootstrap means never exceeds the observed difference in means (in 100 bootstrap sets), a
single repetition of the bootstrap calculations at 100 bootstrap sets confirms that p
,
0.05
appears to be reasonable (although the authors would probably use more than 100 repeti-
tions in results intended for publication, just to be cautious). The difficulty arises when the
bootstrap estimate of the p-value is very close to the desired confidence level (p
,
0.05 in
5
this example). In such a case, a large number of bootstrap sets may be warranted.
It is worth remembering that for N
Bootstrap
sets, the smallest confidence level we could
possibly estimate is 1/N
Bootstrap
e.g. for 1000 bootstraps, the smallest confidence level we
could ever hope to estimate is 1/1000
0.001. The estimate of the confidence interval at
0.001, using 1000 bootstrap sets, is essentially based on the value obtained from a single
bootstrap set (the one producing the largest or smallest value out of the 1000 sets exam-
ined). This suggests that it would be more appropriate to use 10 000 to 20 000 sets to
obtain an estimate of the confidence interval at 0.001, so that the estimate is based on the
results of 10 to 20 bootstrap sets (the 10 or 20 most extreme values out of the 10 000 or
20 000 total sets). In most cases, it is not necessary to estimate confidence intervals at 0.1%
(0.001) because 5% confidence intervals are the standard, and are achievable with lower
numbers of bootstraps.
When in doubt about the number of bootstrap sets that should be used to establish a
particular confidence interval, the safest approach is to repeat the analysis after doubling
the number of bootstrap sets (to determine whether that doubling alters the confidence
level). This doubling should be repeated until the estimate stabilizes; the iterative
approach may be time-consuming, but it is preferable to a blind reliance on a rule of
thumb.
5
Summary
Randomization tests provide a useful alternative to the more familiar analytical statisti-
cal approaches, particularly when the observed distribution departs substantially from the
assumptions of analytic models, or when no analytic estimate is available for the confi-
dence interval of a specific statistic needed for the analysis. The performance of these
methods appears to be equal to that of analytic methods even though the greater flexibility
of these randomization approaches does come at the cost of increased computational time
and it will sometimes be necessary to produce specialized software for novel tests).
References
Anderson, M. J. (2001a). A new method for non-parametric multivariate analysis of variance. Austral Ecology, 26,
32
46.
Anderson, M. J. (2001b). Permutation tests for univariate or multivariate analysis of variance and regression.
Canadian Journal of Fisheries and Aquatic Sciences, 58, 626
639.
Anderson, M. J. (2006). Distance-based tests for homogeneity of multivariate dispersions. Biometrics, 62, 245
253.
Anderson, M. J., & Robinson, J. (2001). Permutation tests for linear models. Australian & New Zealand Journal of
Statistics, 43,75
88.
