Information Technology Reference
In-Depth Information
Chapter 14
Experimentation
The senses deceive from time to time, and it is prudent never to
trust wholly those who have deceived us even once.
Rene Descartes
A hypothesis is … a mere trial idea, a tentative suggestion con-
cerning the nature of things. Until it has been tested, it should
not be confused with a law … Plausibility is not a substitute for
evidence, however great may be the emotional wish to believe.
E. Bright Wilson, Jr.
An Introduction to Scientific Research
Even the clearest and most perfect circumstantial evidence is
likely to be at fault, after all, and therefore ought to be received
with great caution.
Mark Twain
Pudd'nhead Wilson's Calendar
The use of experiments to verify hypotheses is one of the central elements of science.
In computing, experiments—most commonly an implementation tried against test
data—are used for purposes such as confirming hypotheses about algorithms and sys-
tems. An experiment can verify, for example, that a system can complete a specified
task, and can do so with reasonable use of resources. A tested hypothesis becomes
part of scientific knowledge if it is sufficiently well described and constructed, and
if it is convincingly demonstrated.
Some people disagree with the view that rigorous experiments are essential in
computer science; or, if they do not explicitly disagree, may hold a low opinion of
papers that have no new theory and are “merely” experimental. Yet such views are
in stark contrast to the role of experiments in other disciplines. Experiments are an
essential part of sound science.
Experiments in computing take diverse forms, from tests of algorithm perfor-
mance to human factors analysis. However, the principles underlying good experi-
mentation are much the same regardless of what is being investigated. Tests should
