Subjectivist Approaches to Evaluation - Evaluation Methods in Biomedical Informatics - page 251

primarily through observations, interviews, and reviews of documents. The
designs—the data collection plans—of these studies are not rigidly prede-
termined and do not unfold in a fixed sequence. They develop dynamically
and nonlinearly as the investigators' experience accumulates. The study
team begins with a minimal set of orienting questions; the deeper questions
that receive more thorough study evolve from initial investigation. Investi-
gators keep records of all data collected and the methods used to collect
and analyze them. Reports of responsive/illuminative studies tend to be
written narratives. Such studies can be conducted before, during, or after
the introduction of an information resource.

Support for Subjectivist Approaches

It is not surprising that endorsements for subjectivist approaches come from
those who routinely undertake such studies. A more compelling endorse-
ment may come from designers of information resources themselves who
believe that subjectivist methods can provide a deeper understanding of
their own work and thus more useful information to guide their future
efforts. As suggested by the quotation that began this chapter, the results
of a study, when reduced to tables and tests of statistical significance, may
no longer capture what the developers see as most important. When a study
is “for” the developers, this can be a serious shortcoming.
Although subjectivist approaches may run counter to many readers'
notions of how one conducts empirical investigations, these methods and
their conceptual underpinnings are not at all foreign to the worlds of
information and computer science. The pluralistic, nonlinear thinking that
underlies subjectivist investigation shares many features with modern con-
ceptualizations of the information resource design process. Consider the
following statements from two highly regarded works addressing issues
central to resource design. Winograd and Flores 1 argued as follows:

In designing computer-based devices, we are not in the position of creating a formal
“system” that covers the functioning of the organization and the people within it.
When this is attempted, the resulting system (and the space of potential action for
people within it) is inflexible and unable to cope with new breakdowns or poten-
tials. Instead we design additions and changes to the network of equipment (some
of it computer based) within which people work. The computer is like a tool, in that
it is brought up for use by people engaged in some domain of action. The use of the
tool shapes the potential for what those actions are and how they are conducted.
. . . Its power does not lie in having a single purpose...but in its connection to the
larger network of communication (electronic, telephone, paper-based) in which
organizations operate [p. 170].

Norman 2 added:

Tools affect more than the ease with which we do things; they can dramatically affect
our view of ourselves, society, and the world [p. 209].