Information Technology Reference
In-Depth Information
Put another way, finding at least one cluster of events in time or in space
has a greater probability than finding no clusters at all (equally spaced
events).
How can we determine whether an observed association represents an
underlying cause and effect relationship or is merely the result of chance?
The answer lies in our research protocol. When we set out to test a spe-
cific hypothesis, the probability of a specific event is predetermined. But
when we uncover an apparent association, one that may well have arisen
purely by chance, we cannot be sure of the association's validity until we
conduct a second set of controlled trials.
In the International Study of Infarct Survival [1988], patients born
under the Gemini or Libra astrological birth signs did not survive as long
when their treatment included aspirin. By contrast, aspirin offered appar-
ent beneficial effects (longer survival time) to study participants from all
other astrological birth signs.
Except for those who guide their lives by the stars, there is no hidden
meaning or conspiracy in this result. When we describe a test as significant
at the 5% or 1-in-20 level, we mean that 1 in 20 times we'll get a signifi-
cant result even though the hypothesis is true. That is, when we test to
see if there are any differences in the baseline values of the control and
treatment groups, if we've made 20 different measurements, we can
expect to see at least one statistically significant difference; in fact, we will
see this result almost two-thirds of the time. This difference will not repre-
sent a flaw in our design but simply chance at work. To avoid this undesir-
able result—that is, to avoid attributing statistical significance to an
insignificant random event, a so-called Type I error—we must distinguish
between the hypotheses with which we began the study and those that
came to mind afterward. We must accept or reject these hypotheses at the
original significance level while demanding additional corroborating evi-
dence for those exceptional results (such as a dependence of an outcome
on astrological sign) that are uncovered for the first time during the
trials.
No reputable scientist would ever report results before successfully
reproducing the experimental findings twice, once in the original labora-
tory and once in that of a colleague.
2
The latter experiment can be partic-
ularly telling, because all too often some overlooked factor not controlled
in the experiment—such as the quality of the laboratory water—proves
responsible for the results observed initially. It is better to be found wrong
2
Remember “cold fusion?” In 1989, two University of Utah professors told the newspapers
that they could fuse deuterium molecules in the laboratory, solving the world's energy prob-
lems for years to come. Alas, neither those professors nor anyone else could replicate their
findings, though true believers abound, http://www.ncas.org/erab/intro.htm.
