Information Technology Reference
In-Depth Information
and to rely uncritically on computer output, often using the 'default
option' when something a little different (usually, but not always, a little
more complicated) is correct, or at least more appropriate.”
MULTIPLE TESTS
When we perform multiple tests in a study, there may not be journal room
(nor interest) to report all the results, but we do need to report the total
number of statistical tests performed so that readers can draw their own
conclusions as to the significance of the results that are reported.
We may also wish to correct the reported significance levels by using
one of the standard correction methods for independent tests (e.g.,
Bonferroni; for resampling methods, see Westfall and Young, 1993).
Several statistical packages—SAS is a particular offender—print out the
results of several dependent tests performed on the same set of data—for
example, the
t
test and the Wilcoxon. We are not free to pick and choose.
Before we view the printout, we must decide which test we will employ.
Let
W
a
denote the event that the Wilcoxon test rejects a hypothesis at
the a significance level. Let
P
a
denote the event that a permutation test
based on the original observations and applied to the same set of data
rejects a hypothesis at the a significance level. Let
T
a
denote the event
that a
t
test applied to the same set of data rejects a hypothesis at the a
significance level.
It is possible that
W
a
may be true when
P
a
and
T
a
are not, and so forth.
As Pr {
W
a
or
P
a
or
T
a
|
H
} £ Pr {
W
a
|
H
} = a, we will have inflated the
Type I error by picking and choosing after the fact which test to report.
Vice versa, if our intent was to conceal a side effect by reporting that the
results were not significant, we will inflate the Type II error and deflate
the power b of our test, by an after-the-fact choice as b = Pr {not (
W
a
and
P
a
and
T
a
)|
K
} £ Pr{
W
a
|
K
}.
To repeat, we are not free to pick and choose among tests; any such
conduct is unethical.
Both the comparison and the test statistic must
be specified in advance of examining the data.
BEFORE YOU DRAW CONCLUSIONS
Before you draw conclusions, be sure you have accounted for all missing
data, interviewed nonresponders, and determined whether the data were
missing at random or were specific to one or more subgroups.
During the Second World War, a group was studying planes returning
from bombing Germany. They drew a rough diagram showing where the
bullet holes were and recommended those areas be reinforced. A statisti-
