Information Technology Reference
In-Depth Information
path toward alpha inflation when we have more than two groups. In our
SAT study time example with five groups, for example, we can make four
orthogonal comparisons and thus run the risk of at least slightly inflating
our alpha level.
7.6.2 EXPECTED CHANGES IN PROBABILITIES
Cohen (1996, pp. 491-492) gives us a way to compute the probability of
making at least one Type I error if we perform a certain number (
j
)of
independent comparisons evaluated at the .05 alpha level:
05)
j
probability of at least one Type I error
=
1
−
(1
−
.
.
(7.2)
Howell (1997) indicates that this is an expected
familywise error rate.
For
example, if we had five groups and made all four orthogonal comparisons
(
j
4), the probability of making at least one Type I error is about .18 (this
is about three times greater than our .05 alpha level); if we had seven groups
and made all of the twenty-one pairwise comparisons that were possible
(
j
=
21), the probability of making a Type I error is at least .66. Translating
this latter case into odds, the odds of incorrectly concluding that at least
one mean pair of the twenty-one pairs was significantly different are no
worse than 2 to 1. Such a probability of committing a Type I error is
unacceptably high.
=
7.6.3 DEALING WITH ALPHA INFLATION
There are three general situations where we should be very attentive to
the possibility of alpha inflation: (a) when we perform a posteriori com-
parisons, (b) when we make all of the possible orthogonal comparisons,
and (c) when we invoke one of the procedures that calls for a relatively
large number of the possible comparisons to be made (e.g., performing
all pairwise comparisons). Under these three types of circumstances, it is
very common for researchers to compensate in one manner or another
for alpha inflation; as we will see, most of the commonly used so-called
post hoc procedures have some compensation mechanism built in to the
calculations.
7.7 GENERAL CATEGORIES OF MULTIPLE COMPARISON
PROCEDURES
There are four general categories of multiple comparison procedures that
we talk about in this chapter. These categories represent more in the
way of arbitrary dividing lines than hard-and-fast different conceptions.
Nevertheless, our schema has been partially structured around the way in
which researchers generally work with these procedures.
7.7.1 POST HOC COMPARISONS
One approach to multiple comparisons is to use what are often referred
to as
post hoc tests
. Post hoc tests are comparisons of pairs of means
