Information Technology Reference
In-Depth Information
The chief errors in practice lie in failing to report all of the following:
•
Whether we used a one-tailed or two-tailed test and why.
•
Whether the categories are ordered or unordered.
•
Which statistic was employed and why.
Chapter 9 contains a discussion of a final, not inconsiderable source of
error, the neglect of confounding variables that may be responsible for cre-
ating an illusory association or concealing an association that actually exists.
INFERIOR TESTS
Violation of assumptions can affect not only the significance level of a test
but the power of the test, as well; see Tukey and McLaughlin [1963] and
Box and Tiao [1964]. For example, while the significance level of the
t
test is robust to departures from normality, the power of the
t
test is not.
Thus, the two-sample permutation test may always be preferable.
If blocking including matched pairs was used in the original design, then
the same division into blocks should be employed in the analysis. Con-
founding factors such as sex, race, and diabetic condition can easily mask
the effect we hoped to measure through the comparison of two samples.
Similarly, an overall risk factor can be totally misleading (Gigerenzer,
2002). Blocking reduces the differences between subjects so that differ-
ences between treatment groups stand out—that is , if the appropriate
analysis is used. Thus, paired data should always be analyzed with the
paired
t
test or its permutation equivalent, not with the group
t
test.
To analyze a block design (for example, where we have sampled sepa-
rately from whites, blacks, and Hispanics), the permutation test statistic is
S
=S
b
=1
S
j
x
bj
, where
x
bj
is the
j
th observation in the control sample in the
b
th block, and the rearranging of labels between control and treated
samples takes place separately and independently within each of the B
blocks (Good, 2001, p. 124).
Blocking can also be used after the fact if you suspect the existence of
confounding variables and if you measured the values of these variables as
you were gathering data.
9
Always be sure your choice of statistic is optimal against the alternative
hypotheses of interest for the appropriate loss function.
To avoid using an inferior less sensitive and possibly inaccurate statistical
procedure, pay heed to another admonition from George Dyke [1997]:
“The availability of 'user-friendly' statistical software has caused authors to
become increasingly careless about the logic of interpreting their results,
9
This recommendation applies only to a test of efficacy for all groups (blocks) combined.
p
values for subgroup analyses performed after the fact are still suspect; see Chapter 1.
