The Design of Demonstration Studies - Evaluation Methods in Biomedical Informatics - page 213

are performed, that generates performance improvement. As shown earlier
(Table 7.1), the impact of forms on decision making can equal that of com-
puter-generated advice, 13 so it must either be controlled for, quantified, or
ignored as described below. To control for the checklist effect, the same data
can be collected in the same way in control and information resource con-
ditions, even though the information resource's output is only available for
the latter group. 26 To quantify the magnitude of this effect, a randomly
selected “data collection only” group of patients can be recruited. 13 Some-
times the checklist effect is ignored by defining the intervention to include
both the revised data collection methods and the computation performed
on the data after it is collected. While this approach may be scientifically
unsatisfying, for purposes of evaluation it may be entirely satisfactory if the
stakeholders have no interest in separating the issues.
Data Completeness Effect
In some studies, the information resource itself may collect the data used
to assess a dependent variable. Thus more data are available in interven-
tion cases than in controls. The data completeness effect may cut both ways
in influencing study results. For example, consider a field study of an inten-
sive care unit (ICU) information resource where the aim is to compare
recovery rates from adverse events, such as transient hypotension between
patients monitored by the information resource with those allocated to tra-
ditional methods. Because the information resource logs adverse episodes
that may not be recorded by the manual system, the recovery rate may
apparently fall in this group of cases, because more adverse events are being
detected. To detect this bias, the completeness and accuracy of data col-
lected in the control and information resource groups can be compared
against some third method of data collection, perhaps in a short pilot study.
Alternatively, clinical events for patients in both groups should be logged
by computer even though the information resource's output is available
only for care of patients in the invention group. Subsequently, all data from
control patients would be reviewed for evidence of hypotensive episodes.
Feedback Effect
As mentioned in the earlier discussion, one interesting result of the classic
1986 study of the Leeds Abdominal Pain System 13 was that the diagnostic
accuracy of the control house officers spending 6 months in a training level
failed to improve over the period, whereas the performance of the doctors
given both data collection forms and monthly feedback did improve, start-
ing at 13% above control levels at month 1 and rising to 27% above control
levels at month 6 (Table 7.1). Providing these doctors with the opportunity
to capture their diagnoses on a form and encouraging them to audit
their performance monthly improved their performance, even though they
did not receive any decision support per se. Many information resources