Developing and Improving Measurement Methods - Evaluation Methods in Biomedical Informatics - page 164

For studies of information resource performance (where one or more
resources themselves are the objects of measurement), it is difficult to give
an analogous figure for the required number of tasks, because few mea-
surement studies have been performed. Swets, 12 for example, pointed to the
need for “large and representative samples” but did not quantify “large.”
In domains other than medicine, for example weather forecasting, it is
common to test prognostic systems with thousands of cases. Jain's 13 other-
wise thorough discussion of work loads for computer system performance
testing did not directly address the size of the work load necessary for reli-
able measurement. Lacking guidelines for the number of tasks to use for
studying information resource performance in biomedical domains, the
most sensible approach is to conduct measurement studies in advance, and
resolve the problem empirically. If the results of an appropriately designed
measurement study yield unacceptable reliability, the investigator should
change the measurement process as indicated by the results of the study.
Improving Measurement with the Task Facet
When persons are the objects of measurement, it is important to challenge
these persons with a set of tasks that is large enough for adequate mea-
surement, but no larger than necessary. The longer the task set, the greater
the risk of fatigue, noncompliance, or half-hearted effort; data loss through
failure to complete the task set; or expense if the individuals are compen-
sated for their work. The inherent task-to-task variability in performance
cannot be circumvented, but many other steps can be taken to ensure that
every task in a set is adding useful information to a study. The approaches
to improve measurement in this domain are multiple: (1) careful abstract-
ing of case data, (2) sampling a large number of tasks from a known domain,
(3) attention to how performance is scored, and (4) systematic assignment
of tasks to objects.
Abstracting
Much of the published research in informatics is based on case or problem
abstractions of various types to provide a representation of a case that is
completely consistent wherever and whenever it is employed. Typically, a
subset of findings from a clinical case is extracted from the patient's chart
and summarized in a concise written document, creating the ubiquitous
“paper cases” that embody the tasks for a study. 14 Yet, care providers in the
real world work with live patients who provide verbal and nonverbal cues
about their condition. These cues are revealed over time in ways that a
paper abstract cannot capture. The systematic effects of these abstractions
on biomedical informatics studies represent a validity issue from a mea-
surement perspective, and are largely unexplored. 15 However, it is clear that
inconsistent abstracting diminishes the intercorrelations between cases
comprising a set, and thus increases measurement error. To address this