Evaluation as a Field - Evaluation Methods in Biomedical Informatics - page 33

groups. This approach thus seeks to simulate the “evaluation machine”
using randomization, controls, and statistical inference to argue that the
information resource was the cause of any differences observed. Examples
of comparison-based studies include the work of McDonald and colleagues
on physician reminders, 11 the studies from Stanford on rule-based diagnos-
tic systems, 12 and the work of Evans and colleagues on decision support in
antibiotic prescribing. 13 The controlled trials of medical decision support
systems, first reviewed systematically by Johnson and colleagues, 14 and later
by Hunt and colleagues, 15

fall under the comparison-based approach.

The Turing test 16 can be seen as a specific model for a comparison-based

evaluation.

Objectives-Based Approach

The objectives-based approach seeks to determine if a resource meets its
designers' objectives. Ideally, such objectives are stated in detail. This min-
imizes the ambiguity faced by evaluators when developing procedures to
measure the degree of attainment of these objectives. These studies are
comparative only in the sense that the observed performance of the
resource is viewed in relation to stated objectives. The concern is whether
the resource is performing up to expectations, not if the resource is out-
performing what it replaced. The objectives that are the benchmarks for
these studies are typically stated at an early stage of resource development.
Although clearly suited to laboratory testing of a new resource, this
approach can also be applied to testing a deployed resource as well. Con-
sider the example of a resource to provide advice to emergency room physi-
cians. 17 The designers might set as an objective that the system's advice
be available within 10 minutes of the time the patient is first seen. An
objectives-based evaluation study would measure the time for this advice
to be delivered and compare it to the pre-stated objective.

Decision-Facilitation Approach

With the decision-facilitation approach, evaluation seeks to resolve issues
important to developers and administrators, so these individuals can make
decisions about the future of the resource. The questions posed are those
that the decision-makers state, although those conducting the evaluation
may help the decision-makers frame these questions so they are more
amenable to empirical study. The data-collection methods follow from the
questions posed. These studies tend to be “formative” in focus. The results
of studies conducted at the early stages of resource development are used
to chart the course of further development, which in turn generates new
questions for further study. A systematic study of alternative formats for
computer-generated advisories, conducted while the resource to generate
the advisories is still under development, provides a good example of this
approach. 18