The Statistical Significance of F and Effect Strength - Analysis of Variance Designs - page 37

F applicable to our data set would have a mean of 1.20, that is, 12 ÷

(12 20. With increasingly greater degrees of freedom the

expected value - the mean of the sampling distribution - approaches one.

− 2) = 1 .

4.3 THE AREA UNDER THE SAMPLING DISTRIBUTION

Although this may seem rather obvious, we can state for the record that
100 percent of the area is subsumed by, or contained within, the F dis-
tribution. And thanks to the work of many statisticians, it is possible to
figure out the percentage of the area contained in a designated portion
of the curve. For our purposes, we want to know the F value at which 5
percent of the area lies beyond.
The F distribution is based on an infinite number of variance ratios,
but in a single research study, we actually obtain just a single F ratio, and
we must make a determination about the viability of group differences
based on that piece of information alone. Because any mean difference
between two groups can occur by chance, how can we possibly evaluate
“true” differences between the groups in a study by evaluating the single
F ratiothatwehaveobtained?
Theansweristhatwecanmakeaninferenceaboutthedifferences
being “probably true” or “probably reliably different” provided that we
are willing to accept some chance of being wrong. That is, we can interpret a
group difference that is “large enough” with respect to the error variance
as “probably true, reliable, or valid” even though we know that large
differences do occur by chance and that we may be looking at one now.
So researchers have learned to live with a certain amount of uncertainty.
How much uncertainty? The answer to that question relates precisely to
dividing the area of the F distribution into two parts: one containing
95 percent of the area and the other containing 5 percent of the area. And
because we are interested only in F ratios greater than one (where the
between-groups variance is larger than the within-groups variance), we
make our dividing point toward the right side of the distribution.

4.4 STATISTICAL SIGNIFICANCE
4.4.1 THE TRADITIONAL 5 PERCENT BENCHMARK
The scientific community has agreed that F ratios falling into the 95 percent
region are unremarkable in the sense that they occur fairly frequently. On
the other hand, the scientific community has also agreed that, all else being
equal, F ratios falling in the 5 percent area will be treated as “remarkable”
or noteworthy even though it is recognized that these do occur by chance
when the null hypothesis is true. The probability level at the boundary
separating remarkable from unremarkable regions is called an alpha level
or statistical significance level . Those F ratios falling into this 5 percent
region of the distribution - those that are said to be remarkable - are said
to be statistically significant. When we obtain such a result in our research,