Developing and Improving Measurement Methods - Evaluation Methods in Biomedical Informatics - page 160

Self-Test 6.4
For the measurement problem and study described in Self-Test 6.3, to what
category does each facet belong?
Pragmatics of Measurement Using Tasks,
Judges, and Items
In this section, we decompose the multifacet general measurement problem
and focus separately on each important facet of measurement in infor-
matics. We address these pragmatics of measurement one facet at a time,
for several reasons. First, each facet raises its own unique set of pragmatic
issues for measurement. Second, as was noted earlier, many measurement
problems in informatics are dominated by a single facet as a source of mea-
surement error, and in these common cases, a one-facet measurement study
based on the classical theory is sufficient. For each facet, we explore the fol-
lowing: (1) in studies, why the results for a given object vary from obser-
vation to observation and how much variation to expect; (2) in practice,
how many levels of the facet are needed for reliable measurement; and (3)
what can be done to improve this aspect of measurement. We discuss each
facet using applicable object classes (professionals, clients, information
resources) as examples. We focus first on tasks as the facet of interest, then
on judges, and then on items. The three decomposed measurement prob-
lems are illustrated in Figure 6.4.
Task Facet
Many evaluation studies in informatics are performed using real-world
tasks or laboratory simulations of these tasks. The relevant object classes—
the entities undertaking these tasks—may be persons (professionals, stu-
dents, or patients) as actual or potential users of information resources, the
information resources themselves, or groups of persons. In field studies (see
Chapter 3), the tasks are naturally occurring within the work environment;
in laboratory studies the tasks may be invented, simulated, or abstracted
for purposes of control. How these persons or resources perform these tasks
depends on the goals of the study. They may be asked to diagnose, inter-
pret, analyze, predict, retrieve pertinent literature or molecular structures,
propose management, or critique the performance of others. For many
reasons, selection and design of these tasks, for measurement as well as
demonstration studies, is the most challenging aspect of objectivist study
design.
Sources of Variation Among Tasks
The performance of persons and information resources is highly dependent
on the content of the material with which they are challenged. For informa-