The Structure of Objectivist Studies - Evaluation Methods in Biomedical Informatics - page 100

technology support because of the measurement problem that stems from
the absence of absolute standards.
The lack of gold standards in biomedicine has led some to adopt a view
that it is not useful to conduct formal empirical studies of biomedical infor-
mation resources because these studies would always be tainted by the
fuzziness of whatever standard is employed as the basis of comparison. Such
individuals might argue further that instinctive, marketplace, or political
interpretations of the value of these resources should be relied upon
instead. Other, less nihilistic researchers might still conduct empirical
studies, but their studies are designed to bypass the gold-standard issue. For
example, instead of comparing the performance of an information resource
against an imperfect standard of practice, about which there is necessarily
disagreement among human experts, such studies might seek to show that
the resource agrees with the experts to the same extent that experts agree
with each other. 7
In the chapters to follow, the position is taken that gold standards, even
if unattainable, are worth approximating. That is, “tarnished” or “fuzzy”
standards are better than no standards at all. As a theory of measurement
is developed, a method for addressing the fuzziness of gold standards is
developed, joining others who have engaged in similar efforts. 8,9 Perfect
gold standards do not exist in biomedical informatics or in any other
domain of empirical research, but the extent to which these standards are
less than perfect can be estimated and expressed as forms of measurement
error. Knowledge of the magnitude and origin of this error enables the
researcher, in many cases, to consider the error in statistical analyses,
thereby drawing stronger conclusions than otherwise would be possible.
Although zero measurement error is always the best situation, a good esti-
mate of the magnitude of the error is sufficient to allow rigorous studies to
be conducted. Studies comparing the performance of information resources
against imperfect standards, so long as the degree of imperfection has been
estimated, represent a stronger approach than studies that bypass the issue
of a standard altogether.
With this position as a backdrop (and with apologies for a colorfully
mixed metaphor), the gold standard becomes a sort of red herring. More
pragmatically, any standard employed in a study can be viewed as a mea-
sured value of a chosen attribute, which can expediently be accepted as a
standard, knowing that it approximates but is not necessarily equal to the
true “gold standard.” For example, a patient's discharge diagnosis might be
the imperfect standard in a study of an information resource supporting
medical diagnosis. 10 Although it is known that a discharge diagnosis some-
times proves later to have been incorrect, it is the best measure available,
and so it is accepted. The alternative to accepting a less-than-perfect stan-
dard would be not to do the study. An error-free appraisal of the patient's
diagnosis may not be available until the patient's death—and may never be
available if the patient fully recovers from his or her illness. Consistent with