Information Technology Reference
In-Depth Information
assigned at random to six subjects so that three subjects got one treatment
and three the other, there would have been a total of 20 possible assign-
ments of treatments to subjects.
3
To determine a
p
value, we compute for
the data in hand each of the 20 possible values the test statistic might have
taken. We then compare the actual value of the test statistic with these 20
values. If our test statistic corresponds to the most extreme value, we say
that
p
= 1/20 = 0.05 (or 1/10 = 0.10 if this is a two-tailed permutation
test).
Against specific normal alternatives, this two-sample permutation test
provides a most powerful unbiased test of the distribution-free hypothesis
that the centers of the two distributions are the same (Lehmann, 1986,
p. 239). For large samples, its power against normal alternatives is almost
the same as Student's
t
test (Albers, Bickel, and van Zwet, 1976). Against
other distributions, by appropriate choice of the test statistic, its power can
be superior (Lambert, 1985; and Maritz, 1996).
Testing Equivalence
When the logic of a situation calls for demonstration of similarity rather
than differences among responses to various treatments, then equivalence
tests are often more relevant than tests with traditional no-effect null
hypotheses (Anderson and Hauck, 1986; Dixon, 1998; pp. 257-301).
Two distributions
F
and
G
such that
G
[
x
] =
F
[
x
- d] are said to be
equivalent provided that |d|<D, where D is the smallest difference of clini-
cal significance. To test for equivalence, we obtain a confidence interval for
d, rejecting equivalence
only if
this interval contains valuse in excess of D.
The width of a confidence interval decreases as the sample size increases;
thus a very large sample may be required to demonstrate equivalence just
as a very large sample may be required to demonstrate a clinically signifi-
cant effect.
Unequal Variances
If the variances of the two populations are not the same, neither the
t
test
nor the permutation test will yield exact significance levels despite pro-
nouncements to the contrary of numerous experts regarding the permuta-
tion tests.
More important than comparing the means of populations can be determining
why the variances are different.
There are numerous possible solutions for the Behrens-Fisher problem of
unequal variances in the treatment groups. These include the following:
3
Interested readers may want to verify this for themselves by writing out all the possible
addignments of six items into two groups of three, 1 2 3 / 4 5 6, 1 2 4 / 3 5 6, and so forth.
