The Design of Demonstration Studies - Evaluation Methods in Biomedical Informatics - page 196

participants in the study, or they can be made into explicit independent
study variables if their effects are to be explored.
Volunteer Effect
A common bias in the selection of participants is the use of volunteers. It
has been established in many areas that people who volunteer as partici-
pants, whether to complete questionnaires, 14 participate in psychology
experiments, or test-drive new cars or other technologies, are atypical of the
population at large, being more intelligent, open to innovation, and extro-
verted. Although volunteers make willing participants for measurement
studies or pilot demonstration studies, they should be avoided in definitive
demonstration studies, as they considerably reduce the generality of find-
ings. One strategy is to include all participants meeting the selection crite-
ria in the study. However, if this would result in too many participants,
rather than asking for volunteers, it is better to randomly select a sample
of all eligible clinicians, following up invitation letters with telephone calls
to achieve as near 100% recruitment of the selected sample as possible.
Note again that this random selection is not the same as random allocation
of participants to groups, as discussed later in this chapter.
Number of Participants Needed
The financial resources required for an evaluation study depend critically
on the number of participants needed. The required number in turn
depends on the precision of the answer required from the study and the
risk investigators are willing to take of failing to detect a significant effect
(discussed later). Statisticians can advise on this point and carry out sample
size calculations to estimate the number of participants required. Some-
times, in order to recruit the required number of participants, some volun-
teer effect must be tolerated; often there is a trade-off between obtaining
a sufficiently large sample and ensuring that the sample is representative.
Selection of Tasks
In the same way that participants must be carefully selected to resemble
the people likely to use the information resource, any tasks or test cases the
participants complete must also resemble those that will generally be
encountered. Thus when evaluating a clinical order entry system intended
for general use, it would be unwise to use only complex patient cases from,
for example, a pediatric endocrinology practice. Although the order entry
system might well be of considerable benefit in endocrine cases, it is inap-
propriate to generalize results from such a limited sample to the full range
of cases seen in ambulatory care. An instructive example is provided by the
study of Van Way et al., 15 who developed a scoring system for diagnosing
appendicitis and studied the resource's accuracy using, exclusively, patients
who had undergone surgery for suspected appendicitis. Studying this group