Introduction to a variety of useful statistical ideas and techniques - Improving the User Experience through Practical Data Analytics - page 33

SIDEBAR: BEYOND A REASONABLE DOUBT, YOUR HONOR!
The philosophy of hypothesis testing gives H0 the “beneit of the doubt” and imposes the “burden of
proof” on H1. You will not reject H0 unless the evidence against it is quite strong! This is a conserva-
tive approach, and has some similarity of thought process to a criminal court scenario in which you
give the defendant the beneit of the doubt and insist that you ind the defendant guilty (i.e., reject H0,
the status quo) only when the evidence against the defendant is “beyond a reasonable doubt.”
In the UX world, this ensures that the chance is low that we mistakenly adopt a new design,
only to ind out later that it has the same, or even has lower mean satisfaction than what you knew
was the case with the current design.
Keep in mind that the term “accept H0” is really the same as “Insuficient reason to reject H0.”
After all, if a jury comes back with a “not guilty” inding, it is NOT an afirmation of the defen-
dant's innocence, but, rather, a statement that there is insuficient evidence to convict the defendant.
May be Perry Mason should have really been a UX researcher!
Now, common sense would tell you that if X-bar is less than 4.10, there is no
evidence that the new design has a true mean satisfaction rating that is higher than the
current one, and thus, you (obviously) accept H0. And for a sample mean of 4.15, we
noted earlier that this is likely not suficient evidence to reject H0. But, what about
4.2 or 4.3? It is not clear!
On the other hand, if the mean satisfaction rating of a sample of, say, 25 people
is X-bar = 4.8, and nearly everyone gave it a “5”—with a few “4's,” then you would
intuitively reject H0, and conclude that the new design does have a higher true mean
satisfaction rating than 4.1.
But, your intuition will only get you so far in the world of statistics. The good
news is that software like Excel and SPSS will perform all the calculations that will
allow you to conidently accept or reject H0. But before we turn you loose, we need
to explain the all-important concept of the “ p -value.”
1.4.1 p -VALUE
The fact that the software (Excel, SPSS, other statistical software) provides a quan-
tity called “ p -value” makes the entire process of hypothesis testing dramatically sim-
pler. Let us tell you what the p -value represents and why it so greatly simpliies what
you need to do to perform a hypothesis test of virtually any kind. In fact, it will be a
rare chapter in which the p -value does not play a role.
The p -value, in simple terms, is the probability that you would obtain a speciic
data result as far away or farther away from what H0 says, assuming that H0 is true.
The logic is kind of like a “proof by contradiction” that you may have learned in high
school or freshman year in college—although we do not expect you to remember it.
We alluded to it earlier in this section when we described how the H0 gets the beneit
of the doubt and the H1 has the burden of proof. But, now, we describe the logic
more directly:
If the data results you obtain have a very small chance of occurring under the
assumption that H0 is true, then, indeed, we conclude that H0 is NOT true!!