Comparing two designs (or anything else!) using independent sample T-tests - Improving the User Experience through Practical Data Analytics - page 66

“18 for Design 1 and 20 for Design 2,” you reply.
“How can you make any conclusions with such a small sample size?” she retorts
with a huff.
“The p -value of .023 is saying that a difference of .82 (or more) in the sample
means of the two designs has only a 2.3% probability of occurring if the true means
are, indeed, the same.”
She pauses, but has one more salvo: What do you mean by “true means?”
“The true means are the ones we could obtain if we were magically able to poll
every customer we have. At last count, our database has roughly 33,000 customers,
so it's not likely we'll able to get to them all. Thus, we have to use a representative
sample and statistical techniques to make predictions about the true means.”
“Hmm…,” McCarthey says, staring down at her ubiquitous iPhone. You sense
retreat. She punches in a quick text to an unknown recipient, then gets up and bolts
for the door, saying: “OK, let's go with Design 1.”
As the team streams out of the meeting, Autumn Taylor seems to be taking the
news in stride despite the fact that her design lost. She approaches you and whis-
pers: “Hey, I'm really glad we did that survey, for a number of reasons. First, at
least I know we're choosing the design based on data and statistics, and not political
maneuvering.”
“No problem,” you reply. “What's the other reason?”
“The fact that you were able to get McCarthey to back down. That's a irst!”
2.8 ADDENDUM: CONFIDENCE INTERVALS
We learned in Chapter 1 how to construct a conidence interval for a true mean. We
said then (and still believe!) that conidence intervals are very useful. In fact, since a
sample mean does not come out equal to the true mean (except in an extremely rare
case, where the numbers have been rounded and miracle of miracles, we ind out later
that the sample mean turns out equal to the true population mean), a sample mean
is always more valuable to a decision maker if some measure of how far it might be
off from the actual population mean is provided. Indeed, the degree of uncertainty is
usually provided by a conidence interval.
A good way to provide a description of the potential range of values that the
true mean might be is to insert a conidence interval on a bar chart of means, as we
illustrated in Chapter 1. As we've noted, you can safely say that means are differ-
ent if their conidence intervals do not overlap. However, it they do overlap, you
cannot assume the means are not statistically different. Sadly, we've seen too many
erroneous conclusions drawn regarding the difference of means because someone
has simply eyeballed overlapping conidence intervals and proclaimed they were the
same. We now consider and, indeed, illustrate, how this incorrect conclusion can
come about.
Following this incorrect procedure may get you the correct answer, but that sim-
ply means you were lucky that the conidence intervals did not overlap. It is when the