Analyzing the Results of Demonstration Studies - Evaluation Methods in Biomedical Informatics - page 238

TABLE 8.9. Analysis of variance results for the information retrieval example.
Source Sum of squares df Mean square F ratio p

Main effects
Problem set 400.935 1 400.935 9.716 .003

Access mode 210.005 1 210.005 5.089 .030

Interaction
Problem set by 0.078 1 0.078 0.002 .966

access mode
Error 1568.155 38 41.267

independent variables there are four; with four independent variables there
are 11.*
In our example with two independent variables, we need to test for two
main effects and one interaction. Table 8.9 shows the results of ANOVA for
these data. Again, a full understanding of this table requires reference to a
basic statistical text. For purposes of this discussion, note the following:

1. The sum-of-squares is an estimate of the amount of variability in the
dependent variable attributable to each main effect or interaction. All other
things being equal, the greater the sum-of-squares, the more likely is the
effect to be statistically significant.
2. A number of statistical degrees of freedom ( df ) is associated with each
source of statistical variance. For each main effect, df is one less than the
number of levels of the relevant independent variable. Because each inde-
pendent variable in our example has two levels, df 1 for both. For each

interaction, the df is the product of the df s for the interacting variables. In
this example, df for the interaction is 1, as each interacting variable has a
df of 1. Total df in a study is one less than the total number of participants.
3. The mean square is the sum of squares divided by the df .
4. The inferential statistic of interest is the F ratio, which is the ratio of
the mean square of each main effect or interaction to the mean square for
error. The mean square for error is the amount of variability that is unac-
counted for statistically by the independent variables and the interactions
among them. The df for error is the total df minus the df for all main effects
and interactions.
5. Finally, with reference to standard statistical tables, a p value may be
associated with each value of the F ratio and the values of df in the ANOVA
table. A p value of less than .05 is typically used as a criterion for statistical
significance. In Table 8.9, the p value of the effect for problem set depends

=

* With three independent variables (A, B, C), there are three two-way interactions
(AB, AC, BC) and one three-way interaction (ABC). With four independent vari-
ables (A, B, C, D), there are six two-way interactions (AB, AC, AD, BC, BD, CD),
four three-way interactions (ABC, ABD, ACD, BCD), and one four-way interaction
(ABCD).