The Design of Demonstration Studies - Evaluation Methods in Biomedical Informatics - page 198

accuracy from a handful of initial clinical findings, for example the survival
of patients in intensive care. 17 In these unusual circumstances where we
have a close approximation to an “evaluation machine” (see Chapter 2) that
can tell us what would have happened to patients if we had not intervened,
we can compare what actually happens with what was predicted to draw
tentative conclusions about the impact of the information resource. Such
accurate predictive models, however, are unusual across all of biomedi-
cine, 16 so it is generally impossible to determine what students, health
workers or investigators would have done, or what the outcome of their
work would have been, had no information resource been available.
Instead, we must use various types of controls. A control, most generally, is
a group of observations (such as a distinct group of participants, tasks or
measures on those tasks) that are unlikely to be influenced by the inter-
vention of interest or that are influenced by a different intervention.
In the following sections we review a series of specific control strategies,
using as an anchor point the least controlled approach, a purely descriptive
study, and moving to increasingly more sophisticated approaches. We
employ, as a running example, an information resource that prompts doctors
to order prophylactic antibiotics for orthopedic patients to prevent post-
operative infections. In this example, the intervention is the installation and
commissioning of the reminder system, the participants are the physicians,
and the tasks are the patients cared for by the physicians. The dependent
variables include physicians' rate of ordering antibiotics (an effect measure,
in the parlance of Chapter 3) and the rate of postoperative infection aver-
aged across the patients cared for by each physician (an outcome measure).
The independent variables in each example below are an inherent feature of
the study design and derive from the specific control strategies employed.
Although they are not explicitly discussed in what follows to keep the pre-
sentation as focused as possible, measurement issues of the types addressed
in Chapters 5 and 6 (e.g., determining whether each patient's infection can
be accurately judged a postoperative infection) abound in this situation.
Descriptive (Uncontrolled) Studies
In the simplest possible design, a descriptive or uncontrolled study, we
install the reminder system, allow a suitable period for training, then make
our measurements. There is no independent variable. Suppose we discover
that the overall postoperative infection rate is 5% and that physicians order
prophylactic antibiotics in 60% of orthopedic cases. Although we have two
measured dependent variables, it is difficult to interpret these figures
without any comparison; it is possible that there has been no change attrib-
utable to the resource. This point is of course the weakness of the descrip-
tive study.
One way to understand the significance of these figures is to compare
them with the same measurements made using the same methods in a com-