Information Technology Reference
In-Depth Information
where the evidence is thin. Moreover, experienced researchers are well aware that
skepticism is justified. It has been said, with considerable truth, that most published
research findings are false; and unpublished findings are worse.
This means that a paper must be persuasive. Your written work is the one chance
to persuade readers to accept the ideas, and they will only do so if the evidence and
arguments are complete and convincing.
Approaches to Measurement
A perspective on the history of science is that it is also a history of development
of tools of measurement. Our understanding of the laws of physics followed from
development of telescopes, voltmeters, thermometers, and so on. Each improvement
in the measurement technology has refined our understanding of the underlying
properties of the universe.
From this perspective, the purpose of experimentation is to take measurements
that can be used as evidence. A good choice of measure is essential to to practical
system improvement and to persuasive and insightful writing. The measurements are
intended to be a consequence of some underlying phenomenon that is described by
a theory or hypothesis. In this approach to research, phenomena—the eternal truths
studied by science—cannot change, but the measurements can, because they depend
on the context of the specific experiment. Measurements can be quantitative, such
as number or duration or volume—the speed of a system, say, or an algorithm's
efficiency relative to a baseline. They can also be qualitative, such as an occurrence
or difference—whether an outcome was achieved, or whether particular features
were observed.
As you develop your research questions, then, you should ask
what is to be mea-
sured?
and
what measures will be used?
For example, when examining an algorithm,
will it be measured by execution time? And if so, what mechanism will be used to
measure it? This question can be tricky to answer for a single-threaded process run-
ning on a single machine. For a distributed process using diverse resources across a
network, there probably is no perfect answer, only a range of choices with a variety
of flaws and shortcomings, each of which needs to be understood by you and by
your readers.
There is then a critical, but more subtle, question: you need to be satisfied that the
properties being measured are logically connected to the aims of the research. Typ-
ically, research aims are
qualitative
. We seek to improve an interface, accelerate an
algorithm, extract information from an image, generate better timetables for lectures,
and so on. Measurement is
quantitative
; we find a property that can be represented
as a quantity or value. For example, the effectiveness of machine translation systems
is sometimes assessed by counting the textual overlap (words or substrings) of a
computer translation with that made by a human. However, such a measure is obvi-
ously imperfect: not only are there many possible human translations, but a highly
overlapping text can still be incoherent, that is, not a good translation.
