The Structure of Objectivist Studies - Evaluation Methods in Biomedical Informatics - page 95

Measurement and Demonstration Studies
The importance of measurement can be underscored by establishing a
formal distinction between studies undertaken to develop and refine
methods for making measurements, which are called measurement studies ,
and the subsequent use of these methods to address questions of direct
importance in informatics, which are called demonstration studies . Estab-
lishing a distinction between these types of studies, and lending them
approximately equal status in this textbook, are steps intended to ensure
that measurement issues are not overlooked.
Measurement studies, then, seek to determine with how much error an
attribute of interest can be measured in a population of objects, often also
indicating how this error can be reduced. In an ideal objectivist measure-
ment, all observers agree on the result of the measurement. Therefore, any
disagreement is due to measurement error, which should be minimized. The
more agreement among observations, the “better” is the measurement. It is
also important that the observers are observing the intended attribute and
not something different. Measurement procedures developed and vetted
through measurement studies provide researchers with what they need to
conduct demonstration studies. Once it is known with how much error an
attribute can be measured using a particular procedure, the measured
values of this attribute can be employed as a variable in a demonstration
study to draw inferences about the performance, perceptions, or effects of
an information resource. For example, once a measurement study has estab-
lished the error inherent in measuring the speed of an information resource,
a related demonstration study would explore whether a particular resource
has sufficient speed—with speed measured using methods developed in the
measurement study—to meet the needs of researchers and clinicians.
As this discussion unfolds, numerous relations between measurement
and demonstration study design will be seen, but there are also many impor-
tant distinctions. There are differences in terminology that can become
somewhat confusing. For example, measurement studies are concerned with
attributes and objects, whereas demonstration studies are concerned with
variables and subjects. With measurement, the concern is with differences
between individual objects and how accurate the measurement is for each
one. With demonstration studies, the primary interest is usually at the level
of the group and how accurately the mean (or some other indicator of
central tendency) of a variable for that group can be estimated. Of course,
the two issues are intertwined. It is impossible to conduct a satisfactory
demonstration study using poorly performing measurement methods. As is
seen in subsequent chapters, errors in measurement can make differences
between groups more difficult to detect or can produce apparent differ-
ences when none are truly present.