The Design of Demonstration Studies - Evaluation Methods in Biomedical Informatics - page 205

information resource altogether—as often happens in educational set-
tings—the crossover may be the only feasible randomized design, and may
overcome refusal to participate because it allows all participants access,
albeit for only half of the study period. Note that the statistical analysis
needs to take account of the crossover design by using paired tests.
Matched Controls as an Alternative to Randomization,
and the Fallacy of Case-Control Studies
The principle of controls is that they should sensitively reflect all the non-
specific influences and biases present in the study population, while being
isolated in some way from the effects of the information resource. As
argued earlier, it is only by random assignment that equivalence of the
groups can be achieved. Allocation of participants to control and interven-
tion groups may be done by other methods, such as matching, when ran-
domization is not feasible. When this is done, participants and tasks in the
control and intervention groups should be matched on all the features likely
to be relevant to the dependent variable. Usually, a pilot correlational study
is needed to identify which participant factors are most important. Let us
assume that participant age and prior use of information resources turn out
to be important predictors of participant use of an information resource. In
that case, the participants for a study could be divided up into two groups,
taking care that each older person with or without experience in the group
who are to be given access to the resource is matched by a similar person
in the control group.
However, matching controls prior to allocation in the way just described
is definitely not the same as carrying out a case-control study. In a case-
control study, investigators try to infer whether a dependent variable is
associated with one or more independent variables by analyzing a set of
data that has already been collected, so it is a retrospective study design.
For example, investigators could measure attitudes to computers in partic-
ipants who happened in the past to use the information resource (“cases”)
and compare them to attitudes of participants who, in the past, did not
(“controls”). This is an invalid comparison, as the fact that certain partici-
pants chose to use the resource is a clear marker of different levels of skill,
attitude, experience, uncertainty, etc., compared to those who ignored it.
Thus, any differences in outcome between participants in the two groups
are much more likely to follow from fundamental differences between the
participants involved than from use of the information resource. As a result,
case-control studies suffer from the most serious kinds of confounding and
bias, as discussed later in more detail.
One published example is a study that tried to attribute reduced length
of stay in hospital inpatients to use by their physician of the medical
library. 21 In the study, patient lengths of stay were compared in two groups
of patients: those for whose physicians a literature search had been con-