Information Technology Reference
In-Depth Information
Chapter 6
Strengths and Limitations
of Some Miscellaneous
Statistical Procedures
T
HE GREATEST ERROR ASSOCIATED WITH THE USE OF
statistical procedures is
to make the assumption that one single statistical methodology can suffice
for all applications.
From time to time, a new statistical procedure will be introduced or an
old one revived along with the assertion that at last the definitive solution
has been found. As is so often the case with religions, at first the new
methodology is reviled, even persecuted, until it grows in the number of
its adherents, at which time it can begin to attack and persecute the
adherents of other, more established dogma in its turn.
During the preparation of this text, an editor of a statistics journal
rejected an article of one of the authors on the sole grounds that it made
use of permutation methods.
“I'm amazed that anybody is still doing permutation tests . . .” wrote
the anonymous reviewer, “There is probably nothing wrong technically
with the paper, but I personally would reject it on grounds of irrelevance
to current best statistical practice.” To which the editor sought fit to add,
“The reviewer is interested in estimation of interaction or main effects in
the more general semiparametric models currently studied in the literature.
It is well known that permutation tests preserve the significance level but
that is all they do is answer yes or no.”
1
But one methodology can never be better than another, nor can estima-
tion replace hypothesis testing or vice versa. Every methodology has a
proper domain of application and another set of applications for which it
A double untruth. First, permutation tests also yield interval estimates; see, for example,
Garthwaite [1996]. Second, semiparametric methods are not appropriate for use with small-
sample experimental designs, the topic of the submission.
1
