The Design of Demonstration Studies - Evaluation Methods in Biomedical Informatics - page 207

3. At each of two metropolitan hospitals, 18 physicians are randomized
to receive computer-generated advice on drug therapy. At each hospital, the
first group receives advice automatically for all clinic patients, the second
receives this advice only when the physicians request it, and the third
receives no advice at all. Total charges related to drug therapy are measured
by averaging across all relevant patients for each physician during the study
period, where relevance is defined as patients whose conditions pertained
to the domains covered by the resource's knowledge base.
4. A new reminder system is installed in a hospital that has 12 internal
medicine services. During a 1-month period, care providers on six of the
services, selected randomly, receive reminders from the system. On the
other six services the reminders are generated by the system but not issued
to the care providers. An audit of clinical care on all services is conducted
to determine the extent to which the actions recommended by the
reminders were in fact taken.
5. A new computer-based educational tool is introduced in a medical
school course. The tool covers pathophysiology of the cardiovascular (CV)
and gastrointestinal (GI) systems. The class is divided randomly into two
groups. The first group learns CV pathophysiology using the computer and
GI pathophysiology by the usual lecture approach. The second group learns
GI pathophysiology by the computer and CV pathophysiology by the
lecture approach. Both groups are given a validated knowledge test,
covering both body systems, after the course. (Example drawn from Lyon
et al. 23 )
Validity and Inference
Internal vs. External Validity
We all want our demonstration studies to be valid and therefore credible.*
There are two aspects to study validity: internal and external. If a study is
internally valid, we can be confident in the conclusions drawn from the spe-
cific circumstances of the study: the population of participants actually
studied, the measurements made, and the interventions provided. We are
justified in concluding that the differences observed are due to the attrib-
uted causes. However, there are many potential threats to internal validity,
such as confounders, misclassification bias, and selection bias, which
we discuss later. Even if all these threats to internal validity are overcome
to our satisfaction, we also want our study to have external validity.
This means that the conclusions can be generalized from the specific setting,
* Note again the difference in the terminology of measurement and demonstration
studies. Validity of a demonstration study design, discussed here, is different from
validity of a measurement method, discussed in Chapter 5.