Analyzing the Results of Demonstration Studies - Evaluation Methods in Biomedical Informatics - page 236

an actual study as an example. Our example is based on preliminary and
somewhat simplified results of a biomedical information retrieval study
conducted at the University of North Carolina. 13 The study explores
whether a Boolean search tool or Hypertext access to a text database results
in more effective retrieval of information to solve biomedical problems. The
biomedical information available to participants was a “fact and text” data-
base of bacteriology information and was identical across the two access
modes. With the Boolean search tool, participants framed their queries as
combinations of key words joined by logical and or or statements. With the
Hypertext mode, participants could branch from one element of informa-
tion to another via a large number of preconstructed links.
The results to be discussed here are based on data collected from a study
in which medical students were randomized to the Boolean or Hypertext
access mode. Participants were also randomized to one of two sets
of clinical case problems, each set comprising eight clinical infectious
disease scenarios. Students were given two passes through their eight
assigned problems. On the first pass they were asked to generate diagnos-
tic hypotheses using only their personal knowledge. Immediately thereafter,
on the second pass, they were asked to generate another set of diagnostic
hypotheses for the same set of problems but this time with aid from the text
database.
First we examine the basic structure of this study and note that it has two
independent variables, each measured at the nominal level:

• The first independent variable is access mode: Boolean or Hypertext.
• The second independent variable is the particular set of eight case prob-
lems to which students were assigned, arbitrarily labeled set A and set B.

Because each of the two independent variables has two levels and is fully
randomized, the study design is that of a complete factorial experiment (as
discussed earlier in the chapter) with four groups as shown in Table 8.7. The
table also shows the number of participants in each group. The dependent
variable is the improvement in the diagnostic hypotheses from the first pass
to the second—the differences between the aided and unaided scores—
averaged over the eight assigned cases. This variable was chosen because it

TABLE 8.7. Structure of the example information
retrieval study.

Structure, by assigned
problem set

Access mode A B

Boolean G1 G2

( n = 11) ( n = 11)

Hypertext G3 G4

( n = 10) ( n = 10)