The Design of Demonstration Studies - Evaluation Methods in Biomedical Informatics - page 204

TABLE 7.8. Hypothetical results of a randomized
crossover study of an antibiotic reminder system.
Postoperative infection rate (%)

Period Group A Group B

First 5 (reminders) 10 (control)

Second 7 (control) 5 (reminders)

Randomized Crossover Studies
In crossover studies, measurements are made on the same participants with
and without access to the information resource. In our example, this would
mean dividing up the study period into two equal halves—perhaps each 2
months long. Half of the doctors working on both wards (group A) would
then be randomized to use the reminders for the first 2 months only, fol-
lowed by 2 months without the reminders. The doctors who had no access
to the reminders during the first 2-month period (group B) would then get
access during the second period (Table 7.8). However, for this crossover
design to be valid, evaluators must assume that the participant has not
changed except for gaining (or losing) access to the information resource—
that there is no carryover. Thus in a crossover study of our antibiotic
reminder system, evaluators must assume that the user's performance is not
subject to learning, an assumption that usually needs to be tested. In this
example, learning from the decision support system is suggested by the
lower postoperative infection rates in group A during their control period
(which followed use of the reminder system) compared to group B during
their control period, which came before their exposure to any reminders.
So long as there is no carryover, the crossover design overcomes the
imperfections of historically controlled studies by arranging that there are
simultaneous randomized controls during all phases of the experiment.
Making each participant act alternately as intervention and control also
gives greater statistical power than a simple parallel group study, and avoids
the difficulty of matching control and information resource participants or
institutions. However, the crossover study can be used only with interven-
tions that achieve a temporary improvement in the attribute being mea-
sured, so this approach cannot be employed to evaluate information
resources that have significant carryover or educational effects. Proving that
the intervention causes no carryover can actually require more participants
and tasks than conducting a more convincing randomized parallel group
study. Because withdrawing the information resource from participants who
have previously had free access to it may antagonize them if they believe
it is beneficial, the crossover may have to be synchronized with staff
changeover. To be valid, it requires the assumption that the next group of
staff closely resembles the previous group. On the other hand, for studies
where it may be unacceptable for participants to be denied access to the