The Design of Demonstration Studies - Evaluation Methods in Biomedical Informatics - page 221

any improvements with the resource itself—other unknown changes may
also have been associated with the move.
2. Potential biases or threats to validity: The trial may fail because the
clinicians may refuse to fill out the three screens of data to request a test. It
will be hard to compare the appropriateness of paper-based and detailed
electronic requests relying on the supplied data alone, as there will be much
more data once the electronic requesting resource is in place. There may be
a carryover effect if patients are randomized—it would be better to ran-
domize doctors. The study results may be specific to the tests, electronic
requesting resource, and clinicians studied, so the generalizability of the
study results to others attending the bacteriology conference may be limited.
Improvements to implementation/plan: Carry out a pilot study to ensure
that the new electronic test request resource is usable and is likely to be
used before the trial. Determine whether a request was appropriate or not
by reference to case notes, not the data supplied. Randomize doctors, not
patients, to eliminate the carryover effect; analyze at the level of doctors.
Generalize from the study results to other settings with caution. Ideally,
recruit other labs and conduct a multicenter study.
3. Potential biases or threats to validity: The generalizability of the find-
ings from the study seem low, as this is a tertiary referral center handling
particularly challenging cases, and attracting high-flying staff. The benefits
therefore may not be replicated when the resource is rolled out to settings
where most patients have simpler problems and the staff are less able to
respond to the requests and interpret the resulting tests. The attention
drawn to the study by an announcement at a departmental meeting could
lead to a marked Hawthorne effect, thus reducing the apparent benefit from
the reminders. The judgment of appropriate test ordering is carried out by
a single cardiologist, whose views may not be shared by the community. The
judge of appropriate ordering is the same person as the source of the rules,
so the evaluation is circular: the testing is judged appropriate if it was done
as she said it should be done.
Improvements to implementation/plan: Recruit a variety of hospitals to
the study with a more typical case mix and staffing to enhance generaliz-
ability. Ignore the first 2 to 3 weeks of data during the trial to reduce the
impact of Hawthorne effects. Carry out some kind of consensus process to
develop broadly acceptable criteria of appropriate test ordering in cases
such as these. Rather than the circular process above, measure a patient
outcome, to see if more appropriate ordering actually helps the patients.
References
1. Donabedian A. Evaluating the quality of medical care. Millbank Mem Q
1966;44:166-206.
2. Wasson JH, Sox HC, Neff RK, Goldman L. Clinical prediction rules: applica-
tions and methodological standards. N Engl J Med 1985;313:793-799.