Introduction to a variety of useful statistical ideas and techniques - Improving the User Experience through Practical Data Analytics - page 40

FIGURE 1.25
Revised output.
Keep in mind that this change occurred only because it was a very close case before
we changed the one data value, and the sample size was relatively small, and thus, the
sample mean changed from 4.50 to 4.60 (25% further above the H0 value of 4.10). In
most cases, a small change in one data value is not going to change your conclusion.
SIDEBAR: p -VALUE AND STATISTICAL VERSUS PRACTICAL SIGNIFICANCE
Most chapters in the rest of this text will have the p -value as an important, indeed, critical ingredient in
the discussion and decision process. There will be a chapter here and there that does not stress the p -value,
but only because the practical application involved will be more oriented to such things as the percent
of cases correctly classiied, rather than the actual statistical impact of a speciic variable's role in the
analysis. It tells you in a brief moment (by comparing it to 0.05 or whatever value of α is chosen) whether
you should accept or reject the null hypothesis, H0. In that sense, the software providing the p -value does,
indeed, make reaching the ultimate conclusion of accepting or rejecting H0 dramatically easier.
The p -value basically “says it all” about whether to accept or reject H0. However, there is
sometimes a difference between statistical signiicance (i.e., the rejection of H0, easily determined
by looking at the p -value), and what we might call practical signiicance . A population mean may
be indicated by a hypothesis test, beyond a reasonable doubt, to be higher than 4.10; however, if it is
actually 4.12, the difference is probably not meaningful in a practical sense. We would say that the
result is statistically signiicant, but not practically signiicant. The p -value does not say anything at
all about the practical signiicance, while saying it all about the statistical signiicance.
Yet, a inal word needs to be added here: when the sample size is relatively small—and that is
often the case in the world of UX research—a result that is statistically signiicant will be almost
always also practically signiicant . It is when you have a relatively large sample size that this issue
(the possible discrepancy between the statistical and practical signiicance) may occur.
Still, we should acknowledge that there is a vagueness of sorts in this discussion. Given a spe-
ciic signiicance level, say 0.05, there is no doubt when a result is statistically signiicant ; however,
practical signiicance is very context-dependent, and reasonable people can disagree about whether
a result is practically signiicant.