Information Technology Reference
In-Depth Information
significance, select typical results and explain why they are typical, theorize about
anomalies, show why the results confirm or disprove the hypothesis, and make the
results interesting. That is, motivate the work.
Experiments are only valuable if they are carefully described. The description
should reflect the care taken—it should be clear to the reader that possible prob-
lems were considered and addressed, and that the experiments do indeed provide
confirmation (or otherwise) of the hypothesis. A key principle is that the experiment
should be verifiable and reproducible. Results are valueless if they are some kind
of singleton event: repetition of the experiment should yield the same outcomes.
And results are equally valueless if they cannot be repeated by other researchers.
The description, of both hypothesis and experiment, should be in sufficient detail to
allow some form of replication by others. The alternative is a result that cannot be
trusted.
Researchers must decide which results to report. As discussed earlier, researchers
should have logs of experiments recording their history, including design decisions
and false trails as well as the results, but such logs usually contain much material of
no interest to others. And some results are anomalous—the product of experimental
error or freak event—and thus not relevant. But reported results should be a fair
reflection of the experiment's outcomes.
If a test fails on some data sets and succeeds on others, it is unethical to conceal
the failures, and the existence of failure should be stated as prominently as that of
success. Likewise, reporting just one success might lead the reader towonder whether
it was no more than a fluke.
Not all experiments are directly relevant to the hypothesis. An experiment might
be used, for example, to make a preliminary choice between possible approaches
to a problem; other experiments might be inconclusive or lead to a dead end. It
may nonetheless be interesting to the reader to know that these experiments were
undertaken—to know why a certain approach was chosen, for example. For such
experiments, if the detail is unlikely to be interesting it is usually sufficient to briefly
sketch the experiment and the outcome.
The experimental outcomes reported in a paper may represent only a fraction of
the work that was undertaken in a research program. There will have been exploratory
stages and different kinds of failures, and the reported runs may well be carefully
chosen to represent a broad range of experiments. Thus the published record of the
work is highly selective.
In other disciplines of science, researchers are expected to keep detailed notebooks
recording ideas, methods, experiments, data, participants, outcomes, and so on. These
notebooks fill a variety of roles, in particular providing a complete history of the
research, allowing the experiments to be reproduced, and proving that the published
work took place as described—in labs in the biological sciences, for example, it may
be required that a senior scientist sign and date each page.
Notebooks have not acquired these roles in computer science. However, as dis-
locations of software, parameters used in a particular experiment, data used as input
(or the filenames of the data), logs of output (or the filenames of the logs), interpre-
