Information Technology Reference
In-Depth Information
Chapter 15
Statistical Principles
Statistical thinking will one day be as necessary for efficient citi-
zenship as the ability to read and write.
Samuel S. Wilks
Statistics are no substitute for judgement.
Henry Clay
We use experiments and take observations to study the behaviour of a system, to
test hypotheses, to investigate the effect of manipulations and optimizations, and,
overall, to produce evidence for our arguments. The elementary material of evidence
is measurement: the reduction of complex phenonema to numerical scores that can
be recorded, compared, and analyzed.
Raw numbers, however, are dangerously deceptive in their apparent certainty. If
we find in an experiment that our system has a higher score than that of a competitor,
the measure itself may be inaccurate or misleading; it may be only an approximation
to the real-world quality that we are attempting to measure. Second, even if the mea-
sure is appropriate, the value it provides in a single test may be subject to variability
and randomness—in the choice of experimental inputs, in the conditions in which
the experiment is run, or in human assessment of the outcomes.
To gain trust in experiments, we need to repeat them, giving sets of results to
which we can apply statistical methods. Having multiple experiment results not only
provides aggregated, stable measurements, but lets us use the tools of statistical
inference to determine how confident we should be in our conclusions. Repeated
experiments also provide insights into system behaviour through allowing us to
observe variability, success, and failure in a systematic way. An introduction to
statistical methods for experimentation is beyond the scope of this topic, but all
researchers should be aware of relevant statistical principles, and be able to judge
when use of statistics is necessary for their work.
