The Design of Demonstration Studies - Evaluation Methods in Biomedical Informatics - page 199

parison group, which transforms the study from a descriptive to a compar-
ative one. Two types of comparison groups are possible: (1) historical con-
trols, comprising the same patient care environment (doctors and their
patients) before the system was installed; or (2) simultaneous controls, com-
prising a similar patient care environment not provided with the reminder
system.

Historically Controlled Studies
As a first improvement to the uncontrolled or descriptive study, let us con-
sider a historically controlled experiment, sometimes called a before-after
study. The investigator makes baseline measurements of antibiotic order-
ing and postoperative infection rates before the information resource is
installed and then makes the same measurements some time after the infor-
mation resource is in routine use. The independent variable is “time” and
has two levels: before and after resource installation. Let us say that at base-
line the postoperative infection rate was 10% and doctors ordered pro-
phylactic antibiotics in only 40% of cases; the postintervention figures are
the same as before (Table 7.4).
After reviewing these data, the evaluators may claim that the informa-
tion resource is responsible for halving the infection rate, especially because
this was accompanied by a 20% increase in doctors' prophylactic antibiotic
prescribing. However, many other factors might have changed in the
interim to cause these results, especially if there was a long interval between
the baseline and postintervention measurements. New staff could have
taken over the care of the patients, the case mix of patients on the ward
could have altered, new prophylactic antibiotics might have been intro-
duced, or clinical audit meetings might have highlighted the infection
problem causing greater medical awareness of it. Simply assuming that the
reminder system alone caused the reduction in infection rates is naive.
Other factors, known or unknown, could have changed meanwhile, making
untenable the assumption that the intervention is responsible for all of the
observed effects.
The weakness of crediting all benefit to the information resource in a his-
torically controlled study is highlighted by considering the likely response
of the resource developers to a situation where performance of clinicians
worsens after installing the information resource. Most developers, partic-

TABLE 7.4. Hypothetical results of a historically controlled study of an antibiotic
reminder system.
Time

Antibiotic prescribing rate (%) Postoperative infection rate

Baseline (before 40 10

installation)
After installation 60 5