Analyzing the Results of Demonstration Studies - Evaluation Methods in Biomedical Informatics - page 237

TABLE 8.8. Results of the example information retrieval
study.

Results, by assigned problem set
(mean ± SD)

Access mode A B

Boolean 11.2 ± 5.7 17.5 ± 7.3

( n = 11) ( n = 11)

Hypertext 15.7 ± 7.2 21.8 ± 5.2

( n = 10) ( n = 10)

estimates the effect attributable to the information retrieved from the text
database, controlling for each participant's prior knowledge of bacteriology
and infectious disease.
The logic of analyzing data from such an experiment is to compare the
mean values of the dependent variable across each of the groups. Table 8.8
displays the mean and standard deviations of the improvement scores for
each of the groups. For all participants, the mean improvement score is 16.4
with a standard deviation of 7.3.
Take a minute to examine Table 8.8. It should be fairly clear that there
are differences of potential interest between the groups. Across problem
sets, the improvement scores are higher for the Hypertext access mode than
the Boolean mode. Across access modes, the improvement scores are
greater for problem set B than for problem set A. The effect sizes for this
study are directly related to the differences between the means in each of
the cells of Table 8.8.*

Using ANOVA to Test Statistical Significance
The methods of ANOVA allow us to determine the probability that the
effect sizes reflected in differences between group means, whatever the
magnitude of these differences, arose due to chance alone. Group dif-
ferences attributable to each of the independent variables are called main
effects ; differences attributable to the independent variables acting in com-
bination are called interactions . The number of possible main effects is
equal to the number of independent variables; the number of possible
interactions increases geometrically with the number of independent vari-
ables. With two independent variables there is one interaction; with three

* A useful way of expressing effect sizes for this kind of study is Cohen's d , which,
for any pair of cells of the design, is the difference between the mean values
divided by the standard deviation of the observations. Use of Cohen's d allows
standardized expression of effect sizes in “standard deviation units,” which are com-
parable across studies. Traditionally, effect sizes of .8 standard deviations (or larger)
are interpreted as “large” effects, .5 standard deviations as “medium” effects, and .2
standard deviations (or smaller) as “small” effects. 14