Subjectivist Approaches to Evaluation - Evaluation Methods in Biomedical Informatics - page 248

Subjectivist Approaches
to Evaluation
As usual, the most significant results of the project are not measurable with a t-test.
(M. Musen, summarizing his experience with a 5-year project, personal communi-
cation, 1996)
With this chapter we turn a corner. The previous five chapters have dealt
almost exclusively with objectivist approaches to evaluation. These
approaches are useful for answering some, but by no means all, of the inter-
esting and important questions that challenge investigators in biomedical
informatics. The subjectivist approaches, introduced here and in Chapter 10,
address the problem of evaluation from a different set of premises, as ini-
tially discussed in Chapter 2. These premises derive from philosophical
views that may be unfamiliar and perhaps even discomforting to some
readers. They challenge some fundamental beliefs about scientific method
and the validity of our understanding of the world that develops from objec-
tivist investigation. They argue that, particularly within the realm of evalu-
ation of information resources, the kind of “knowing” that develops from
subjectivist studies may be as useful as that which derives from objectivist
studies. While reading what follows in this chapter, it may be tempting to
dismiss subjectivist methods as informal, imprecise, or “subjective.” When
carried out well, however, these studies are none of the above. They are
equally objective, but in a different way. Professionals in informatics, even
those who choose not to conduct subjectivist studies, can come to appreci-
ate the rigor, validity, and value of this work.
Chapter 2 introduced four subjectivist approaches to evaluation: con-
noisseurship, quasi-legal, professional review, and illuminative/responsive.
Chapters 9 and 10 focus on what we have called the illuminative/
responsive approach to evaluation.* This approach is rooted in the inves-
tigative traditions of ethnography and social anthropology, traditions that
emphasize observation of naturally occurring behavior in defined cultural
* Many proponents of these methods refer to them generically as “qualitative”
methods.
248