Multiple Comparison Procedures - Analysis of Variance Designs - page 126

Tests assuming homogeneity of variance outnumber those that do not,
and the range of choices is therefore more substantial. For a posteriori
tests such as these, the extreme ends of the continuum tend to be generally
avoided. Fisher's LSD procedure is easily the most liberal (powerful); since
it does not control familywise error at all, you run a greater risk of making
more false positive judgments than expected of an alpha level of .05.
Although using the Fisher LSD procedure based on an F ratio (tested at
an alpha of .05) has its supporters (e.g., Carmer & Swanson, 1973), we
suggest that if you use the LSD procedure at all, you should restrict it to
a very small number of planned comparisons. We therefore agree with
Keppel (1991) that it should not be used in an a posteriori manner to
make several comparisons. On the other side of the spectrum, Scheffe's
(1953) procedure is easily the most conservative of the available post hoc
procedures; many researchers believe that it may be too conservative unless
the research context warrants a low, Type I risk level and advise against its
general use; we concur.
Keppel and Wickens (2004) recommend against using the S-N-K and
Tukey-b tests partly because of their lack of control of familywise error.
We would agree and add to that list the Duncan test.
The Bonferroni a very good test but is quite conservative, and it appears
that many people pass on it for that reason. Sid ak's variation of the Bonfer-
roni test gives it a bit more power but we have not seen a huge endorsement
rate for it either.
The two tests that are actively recommended by respected authors
(e.g., Howell, 1997; Keppel & Wickens, 2004) are the Tukey HSD test
and the R-E-G-W-Q. Both hold the expected familywise error rate at no
worse than the alpha level. These tests are probably in the midrange of the
power spectrum, perhaps shaded somewhat toward the conservative end;
the R-E-G-W-Q is more powerful than the Tukey, but the Tukey is the
morewidelyusedofthetwo.Ifyouopttoperformaposterioripairwise
comparisons, then we would encourage you to consider one of them.
However, there are alternative strategies covered later in this chapter that
you should consider before automatically reaching for any post hoc test.
7.9 COMPUTING A TUKEY HSD TEST BY HAND
Conducting a multiple comparison test is accomplished in a two-step
process. The first step involves displaying all possible differences among
the means under scrutiny, which we will refer to as creating a matrix
of differences . A second step involves some simple arithmetic operations
using a special statistical formula for the multiple comparison test.
Table 7.2 depicts the five treatment means from our SAT preparation
example arranged in ascending order for both the columns and rows of
the matrix. Such an arrangement allows us to make (by subtracting one
mean from another) all possible pairwise comparisons. For example, we