Evaluation as a Field - Evaluation Methods in Biomedical Informatics - page 23

paring real history with the fabrication created by the evaluation machine,
accurate conclusions can potentially be drawn about the effects of the
resource. Even if there was an evaluation machine, however, it could not
solve all our problems. It could not tell us why these effects occurred or
how to make the resource better. To obtain this information, we would have
to communicate directly with many of the actors in our real history in order
to understand how they used the resource and their views of the experi-
ence. There is usually more to evaluation than demonstrations of effects.
In part because there is no evaluation machine, but also because ways
are needed to answer additional important questions for which the
machine would be of little help, there can be no single solution to the
problem of evaluation. There is, instead, an interdisciplinary field of
evaluation with an extensive methodological literature. 2-4 This literature
details many diverse approaches to evaluation, all of which are currently
in use. These approaches will be introduced later in the chapter. The
approaches differ in the kinds of questions that are seen as primary,
how specific questions get onto the agenda, and the data-collection methods
ultimately used to answer these questions. In informatics, it is important
that such a range of methods is available because the questions of interest
can vary dramatically—from the focused and outcome-oriented (Does
implementation of this resource affect morbidity and/or mortality?) to the
practical, and market-oriented questions, such as those frequently stated by
Barnett: †
1. Is the system used by real people with real patients?
2. Is the system being paid for with real money?
3. Has someone else taken the system, modified it, and claimed they devel-
oped it?
Evaluation is challenging in large part because there are so many options
and there is almost never an obvious best way to proceed. The following
points bear repeating:
1. In any evaluation setting, there are many potential questions to address.
What questions are asked shapes (but does not totally determine) what
answers are generated.
2. There may be little consensus on what constitutes the best set of
questions.
3. There are many ways to address these questions, each with advantages
and disadvantages.
4. There is no such thing as a perfect evaluation.
Individuals conducting evaluations are in a continuous process of com-
promise and accommodation. At its root, the challenge of evaluation is to
† These questions were given to the authors in a personal communication on
December 8, 1995. A slightly different version of these questions is found on page
286 of Blum. 5