Information Technology Reference
In-Depth Information
fails. Every methodology has its drawbacks and its advantages, its assump-
tions, and its sources of error. Let us seek the best from each statistical
procedure.
The balance of this chapter is devoted to exposing the frailties of four of
the “new” (and revived) techniques: bootstrap, Bayesian methods, meta-
analysis, and permutation tests.
BOOTSTRAP
Many of the procedures discussed in this chapter fall victim to the erro-
neous perception that one can get more out of a sample or series of
samples than one actually puts in. One bootstrap expert learned he was
being considered for a position because management felt, “your knowl-
edge of the bootstrap will help us to reduce the cost of sampling.”
Michael Chernick, author of
Bootstrap Methods: A Practitioner's Guide
,
Wiley, 1999, has documented six myths concerning the bootstrap:
1. Allows you to reduce your sample size requirements by replacing
real data with simulated data—Not.
2. Allows you to stop thinking about your problem, the statistical
design and probability model—Not.
3. No assumptions necessary—Not.
4. Can be applied to any problem—Not.
5. Only works asymptotically—Necessary sample size depends on the
context.
6. Yields exact significance levels—Never.
Of course, the bootstrap does have many practical applications, as wit-
nessed by its appearance in six of the chapters in this topic.
2
Limitations
As always, to use the bootstrap or any other statistical methodology
effectively, one has to be aware of its limitations. The bootstrap is of
value in any situation in which the sample can serve as a surrogate for the
population.
If the sample is not representative of the population because the sample
is small or biased, not selected at random, or its constituents are not inde-
pendent of one another, then the bootstrap will fail.
Canty et al. [2000] also list data outliers, inconsistency of the bootstrap
method, incorrect resampling model, wrong or inappropriate choice of
statistic, nonpivotal test statistics, nonlinearity of the test statistic, and dis-
creteness of the resample statistic as potential sources of error.
2
If you're counting, we meet the bootstrap again in Chapters 10 and 11.
