Subjectivist Approaches to Evaluation - Evaluation Methods in Biomedical Informatics - page 250

These more specific, more explanatory, and more probing questions,
shown in italics, are often what those who commission evaluation studies—
and others with interest in an information resource—want to know. Some
of these deeper questions are difficult to answer using objectivist
approaches to evaluation. It may be that these questions are never dis-
cussed, or are deferred as interesting but “subjective” issues during discus-
sions of what should be the foci of an evaluation study. These questions may
never be asked in a formal or official sense because of a perception that the
methods do not exist to answer them in a credible way. This chapter and
Chapter 10 beg readers to suspend their own tendencies to this belief.
Many of these deeper questions derive their importance from life in a
pluralistic world. As discussed in Chapters 1 and 2, information resources
are typically introduced into complex organizations where there exist com-
peting value systems: different beliefs about what is “good” and what is
“right,” which translate into different beliefs about whether specific changes
induced by information resources are beneficial or detrimental. These
beliefs are real to the people who hold them and difficult to change. Indeed,
there are many actors playing many roles in any real-world setting where
an information resource is introduced. Each actor, as an individual and a
member of multiple groups, brings a unique viewpoint to questions about
inextricably fuzzy constructs such as need, quality, and benefit. If these con-
structs are explored in an evaluation study, perhaps the actors should not
be expected to agree about what these constructs mean and how to measure
them. Perhaps need, quality, and benefit do not inhere in an information
resource. Perhaps they are dependent on the observer as well as the
observed. Perhaps evaluation studies should be conducted in ways that doc-
ument how these various individuals and groups “see” the resource, and not
in ways that assume there is a consensus when there is no reason to believe
one exists. Perhaps there are many “truths” about an information resource,
not just one.
Definition of the Responsive/Illuminative Approach
The responsive/illuminative approach to evaluation is designed to address
the deeper questions: the detailed “whys” and “according to whoms” in
addition to the aggregate “whethers” and “whats.” As defined in Chapter 2,
the responsive/illuminative approach seeks to represent the viewpoints of
those who are users of the resource or otherwise significant participants in
the environment where the resource operates. The goal is “illumination”
rather than judgment. The investigators seek to build an argument that pro-
motes deeper understanding of the information resource or environment
of which it is a part. The methods used derive largely from ethnography. As
such, the investigators immerse themselves physically in the environment
where the information resource is or will be operational and collect data