The Design of Demonstration Studies - Evaluation Methods in Biomedical Informatics - page 202

TABLE 7.6. Results of a simultaneous randomized con-
trolled study of an antibiotic reminder system.
Postoperative infection rate (%)

Time Reminder physicians Control physicians

Baseline 11 10

After 6 8

intervention

difference between the two groups of patients is receipt of reminders by
their doctors. Provided that the sample size is large enough for these results
to be statistically significant, we can conclude with some confidence that
giving doctors reminders caused the reduction in infection rates. One lin-
gering question is why there was also a small reduction, from baseline to
postinstallation, in infection rates in control cases. Four explanations are
possible: chance, the checklist effect, the Hawthorne effect, and contami-
nation. These possibilities are discussed in detail in later sections.
When analyzing studies in which doctors or teams are randomized but
the measurements are made at the level of patients (randomization by
group), data analysis methods must be adjusted accordingly. In general,
when randomizing doctors or hospitals, it is a mistake to analyze the results
as if patients had been randomized—known as the “unit of analysis
error.” 11,19 This problem and potential methods for addressing it are dis-
cussed in the section on hierarchical or nested designs (later in this chapter).

Externally and Internally Controlled
Before-After Studies
An alternative approach to randomized simultaneous controls is to add
internal controls to an externally controlled before-after study. Using inter-
nal controls, we add to the study some new observations or dependent vari-
ables that would not be expected to be affected by the intervention. We
already discussed the benefits of external controls—comparing the rates of
infection and antibiotic prescribing with those in another ward where the
resource has not been implemented. However, the situation is strengthened
further by measuring one or more appropriate variables in the same ward
to check that nothing else in the clinical environment is changing during
the period of a before-after study. As long as this works out, and there are
no unexpected changes in the external site, one really can then begin to
argue that any change in the measurement of interest must be due to the
information resource. 20 However, the risk one takes in this kind of study
design is that the results often turn out to be hard or impossible to inter-
pret because of unforeseen changes in the dependent variable in the exter-
nal or internal controls.