Subjectivist Approaches to Evaluation - Evaluation Methods in Biomedical Informatics - page 254

accepted. It is therefore possible to distinguish a good study from a bad one.
Finally, a neophyte can learn either or both types of approaches, initially by
reading textbooks and other methodological literature and ultimately by
conducting studies under the guidance of experienced mentors.
There are, at the same time, many fundamental differences between
objectivist and subjectivist approaches. First and foremost, subjectivist
studies are “emergent” in design. Objectivist studies typically begin with a
set of hypotheses or specific questions and a plan for addressing each
member of this set. There is also an assumption by the investigator that,
barring major unforeseen developments, the plan will be followed exactly.
(When objectivist investigators deviate from their plan, they do so apolo-
getically and view their having done it as a limitation of their study.) Not
following the plan is seen as a source of bias, because the investigator who
sees negative results emerging from the exploration of a particular ques-
tion or use of a particular measurement instrument might change strategies
in the hope of obtaining more positive findings. By contrast, subjectivist
studies typically begin with some general orienting issues that stimulate the
early stages of investigation. Through these initial investigations, the impor-
tant questions for further study begin to emerge. The subjectivist investi-
gator is willing, at virtually any point, to adjust future aspects of the study
in light of the most recent information obtained. Subjectivist investigators
are incrementalists; they live from day to day and have a high tolerance for
ambiguity and uncertainty. (In this respect, they are again much like good
software developers.) Also like software developers, skilled subjectivist
investigators must develop the ability to recognize when a project is
finished—when further benefit can be obtained only at great cost in time
and effort.
A second feature of subjectivist studies is a “naturalistic” orientation—a
reluctance to manipulate the setting of the study, which in most cases is the
work environment into which the information resource is introduced.
Because subjectivist studies avoid altering the environment in order to
study it, these studies have an appealing “ecological validity.” There is no
question that the results apply to the exact setting, work process, and culture
within which the information resource under study is deployed. The extent
to which the results can be safely generalized from that specific setting to
other similar settings depends very much on local circumstances. In sub-
jectivist investigation, however, the aim is rarely to generalize to other set-
tings, and more usually to gain better insight and understanding into the
specific setting under scrutiny. Control groups, placebos, purposefully alter-
ing information resources to create contrasting interventions, and other
techniques central to the construction of objectivist studies are typically not
used in subjectivist work. Subjectivist studies do employ quantitative data
for descriptive purposes and may additionally offer quantitative compar-
isons when the study setting offers up a natural experiment where such
comparisons can be made without altering how work is organized or per-