Information Technology Reference
In-Depth Information
that it is unrealistic or simplistic, with properties that mean that the observed results
would not occur in practice. Thus simulations are powerful tools, but, ultimately,
need to be verified against reality.
Experiment
. An experiment is a full test of the hypothesis, based on an
implementation of the proposal and on real—or highly realistic—data. In an experi-
ment there is a sense of
really doing it
, while in a simulation there is a sense of
only
pretending
. For example, artificial data provides a mechanism for exploring behav-
iour, but corresponding behaviour needs to be observed on real data if the outcomes
are to be persuasive.
In some cases, though, the distinction between simulation and experiment can be
blurry, and, in principle, an experiment only demonstrates that the hypothesis holds
for the particular data that was used; modelling and simulation can generalize the
conclusion (however imperfectly) to other contexts.
Ideally an experiment should be conducted in the light of predictions made by a
model, so that it confirms some expected behaviour. An experiment should be severe;
seek out tests that seem likely to fail if the hypothesis is false, and explore extremes.
The traditional sciences, and physics in particular, proceed in this way. Theoreticians
develop models of phenomena that fit known observations; experimentalists seek
confirmation through fresh experiments.
Use of Evidence
Different forms of evidence can be used to confirm one another, with say a simulation
used to provide further evidence that a proof is correct. But the different forms should
not be confused with one another. For example, suppose that for some algorithm there
is a mathematical model of expected performance. Encoding this model in a program
and computing predicted performance for certain values of the model parameters is
not an experimental test of the algorithm and should never be called an experiment;
it does not even confirm that the model is a description of the algorithm. At best it
confirms claimed properties of the model.
When choosing whether to use a proof, model, simulation, or experiment as evi-
dence, consider how convincing each is likely to be to the reader. If your evidence is
questionable—say a simplistic and assumption-laden model, an involved algebraic
analysis and application of advanced statistics, or an experiment on limited data—the
reader may well be skeptical of the result. Select a form of evidence, not so as to
keep your own effort to a minimum, but to be as persuasive as possible.
Having identified the elements a research plan should cover, end-to-start reasoning
suggests how these elements should be prioritized. The write-up is themost important
thing; so perhaps it should be started first. Completing the report is certainly more
important than hastily running some last-minute experiments, or quickly browsing
the literature to make it appear as if past work has been fully evaluated.
Some novice researchers feel that the standards expected of evidence are too high,
but readers—including referees and examiners—tend to trust work that is already
published in preference to a new, unrefereed paper, and have no reason to trust work
