Developing and Improving Measurement Methods - Evaluation Methods in Biomedical Informatics - page 145

Developing and Improving
Measurement Methods
This chapter moves the theoretical emphasis of Chapter 5 to actual mea-
surement practice. In Chapter 5 we introduced theories of measurement
that were, in effect, theories of error. In this chapter we address specific pro-
cedures for estimating error and designing measurement methods to reduce
error. We discuss the structure of measurement studies, the mechanics of
conducting them, and how one uses the results of these studies to improve
measurement techniques. We consider how to develop measurement
methods that yield acceptably reliable and valid results. Recalling that the
key to objectivist measurement is the use of equivalent independent obser-
vations, we organize much of this chapter around three different categories
of independent observations that arise frequently in measurement prob-
lems in informatics: first, when the repeated observations in a measurement
process are tasks completed by either persons or information resources;
second, when the repeated observations are the opinions of judges about
clinical cases or scientific problems; and third, when the repeated observa-
tions are items or questions on forms. Although the same general mea-
surement concepts apply to all three categories, there are issues of
implementation and technique specific to each.
Structure of Measurement Studies:
Objects, Observations, and Scales
Recall from Chapter 5 that in measurement studies we typically make
multiple independent observations on each of a set of objects. The data
collected during a measurement study take the form of an objects-by-
observation matrix (Figure 6.1). In the objectivist worldview, all indepen-
dent observations of the same phenomenon should yield the same result.
The closer the observations approach agreement for each object, the more
reliable, and therefore objective and trustworthy, the measurement process
can be considered to be. Disagreement reflects subjectivity on the part of
145