The Structure of Objectivist Studies - Evaluation Methods in Biomedical Informatics - page 99

Self-Test 4.3
Clarke and colleagues developed the TraumAID system 5 to advise on initial
treatment of patients with penetrating injuries to the chest and abdomen.
Measurement studies of the utility of TraumAID's advice required panels
of judges to rate the adequacy of management of a set of “test cases”—as
recommended by TraumAID and as carried out by care providers. To
perform this study, case data were fed into TraumAID to generate a treat-
ment plan for each case. The wording of TraumAID's plans was edited care-
fully so that judges performing subsequent ratings would have no way of
knowing whether the described care was performed by a human or rec-
ommended by computer. Two groups of judges were employed in the mea-
surement studies: one from the medical center where the resource was
developed, the other a group of senior physicians from across the country.
For this measurement situation, name the attribute of interest and the
object class; describe the instrumentation. List some issues that might be
clarified by this measurement study.
[Answers are found at the end of the chapter.]
Gold Standards and Informatics
A final issue before leaving the topic of measurement is the often-used
term “gold standard” and its relation to measurement in informatics.
In biomedical informatics, the lack of so-called gold standards is often
bemoaned. 6 As we embark on an exploration of objectivist methods, it is
timely to ask: What exactly is a gold standard? By traditional definition, a
gold standard is a composite of two notions. In the first sense, a gold stan-
dard is an expression of practice carried out perfectly: the optimal therapy
for a given biomedical problem or the best differential diagnosis to be
entertaining at a particular point in the evolution of a case. In the second
sense, a gold standard implies complete acceptance or consensus. For a
given situation, everyone qualified to render a judgment would agree to
what the gold standard is. These two aspects of a gold standard are tightly
interrelated. If there exists only one standard of care, it must be a standard
everyone would endorse completely.
In most real-world situations of sufficient interest to merit study, perfect
(gold) standards of practice do not exist. In treating patients, in conducting
a high-throughput array analysis, or in the process of learning to program
a computer, there is usually no unequivocal “best thing to do” at a particular
point in time. Given two or more scenarios of professional practice, inde-
pendent observers who are measuring the quality of this practice will not
be in perfect agreement as to which scenario represents better practice.
Health care, research, or education conducted with support from infor-
mation technology cannot be definitively compared to practice without