Testing Hypotheses: Choosing a Test Statistic - Common Errors in Statistics - page 60

Nonsystematic Dependence. If the observations are interdependent and
fall into none of the preceding categories, then the experiment is fatally
flawed. Your efforts would be best expended on the design of a cleaner
experiment. Or, as J. W. Tukey remarked on more than one occasion, “If
a thing is not worth doing, it is not worth doing well.”
COMPARING VARIANCES
Testing for the equality of the variances of two populations is a classic
problem with many not-quite-exact, not-quite-robust, not-quite-powerful-
enough solutions. Sukhatme [1958] lists four alternative approaches and
adds a fifth of his own; Miller [1968] lists 10 alternatives and compares
four of these with a new test of his own; Conover, Johnson, and Johnson
[1981] list and compare 56 tests; and Balakrishnan and Ma [1990] list
and compare nine tests with one of their own.
None of these tests proves satisfactory in all circumstances, because
each requires that two or more of the following four conditions be
satisfied:
1. The observations are normally distributed.
2. The location parameters of the two distributions are the same or
differ by a known quantity.
3. The two samples are equal in size.
4. The samples are large enough that asymptotic approximations to
the distribution of the test statistic are valid.
As an example, the first published solution to this classic testing
problem is the z test proposed by Welch [1937] based on the ratio of the
two sample variances. If the observations are normally distributed, this
ratio has the F distribution, and the test whose critical values are deter-
mined by the F distribution is uniformly most powerful among all unbi-
ased tests (Lehmann, 1986, Section 5.3). But with even small deviations
from normality, significance levels based on the F distribution are grossly
in error (Lehmann, 1986, Section 5.4).
Box and Anderson [1955] propose a correction to the F distribution for
“almost” normal data, based on an asymptotic approximation to the per-
mutation distribution of the F ratio. Not surprisingly, their approximation
is close to correct only for normally distributed data or for very large
samples. The Box-Anderson statistic results in an error rate of 21%, twice
the desired value of 10%, when two samples of size 15 are drawn from a
gamma distribution with four degrees of freedom.
A more recent permutation test (Bailor, 1989) based on complete enu-
meration of the permutation distribution of the sample F ratio is exact