Introduction to a variety of useful statistical ideas and techniques - Improving the User Experience through Practical Data Analytics - page 13

This sa ys that the probability (“ P ” stands for “probability”) that the interval
(X − e) to (X + e) contains the true value, μ, equals (1 − α). By tradition, industry
generally uses 0.95 (i.e., 95%) for the value of (1 − α). “Alpha” is the amount
of probability outside the conidence interval. So, if the conidence level (1 − α),
is 0.95, then α = 0.05. We will see this “0.05” used more directly in the next
section.
Here's an example. Suppose that after running a usability study with 4 par-
ticipants, we have these 4 data points on a 5-point (1-5) Likert-scale question for
the satisfaction of a new design: 2, 3, 4, 3. The sample mean, X-bar, is, of course,
(2 + 3 + 4 + 3)/4 = 3.0. As can be determined (we'll see how, very soon), we get a
95% conidence interval for the true mean of 1.70-4.30. The interval has 3.0 as its
center; it is pretty wide (in practice, we always prefer a narrower conidence interval,
since it means we have homed in more closely to the true mean), but that is primarily
because we have sampled only four satisfaction values, and there is a fair amount of
variability in the four results.
In more practical terms, we can say:
We have 95% conidence that the true mean satisfaction of all people who have
experienced the design is between 1.70 and 4.30.
SIDEBAR: CONFIDENT ABOUT THE INTERVAL, NOT THE TRUE MEAN!
Keep in mind that the true/population mean does not vary. It is what it is, albeit we do not know
what it is, and may never know what it is. What is subject to variation is the interval—the lower and
upper conidence limits themselves. Therefore, a theoretician may argue that the deinition of a con-
idence interval should always read, “…95% conidence that the interval contains the true mean…,”
and never, “…95% conidence that the true mean is in the interval.” In other words, the subject of
the sentence, that which is subject to uncertainty, should be the interval, not the true mean.
This is equivalent to saying that, “If we constructed a large number of 95% conidence intervals from
a large number of replicates of a given experiment, 95% of these intervals would contain the true mean.”
Now, this statement is accurate—but it probably won't get you any bonus points
with your design team, since the conidence interval is so wide. As we noted, the
small sample size, combined with the amount of variability in the four results, has
resulted in a wide conidence interval, which doesn't seem useful at irst glance. (We
acknowledge that you may feel uncomfortable stating that the true average is “almost
for sure in the interval, 1.7-4.3.”) In fact, it is useful: the conidence interval man-
dates great caution in what you claim for the result. (And trust us; you'll gain cred-
ibility by including the conidence intervals and lose credibility if you ignore them.)
Now, if the data had been 3, 4, 4, 4 (still, only four data points, but less variable
from one to another!!), the 95% conidence interval would be much narrower (which
we like better!), and would be,
2 . 95 ‐‐‐‐‐‐‐‐‐‐4 . 55,
centered around the X-bar value of 3.75.