Comparing two designs (or anything else!) using paired sample T-tests - Improving the User Experience through Practical Data Analytics - page 87

FIGURE 3.8
Output for illustrative example; Excel.
that the true mean could be zero and have every value of d between 5 and 7, and more
6's than any other value. 1
However, if we take the same data and treat them as independent samples, we will
get a very different answer! It will still be the case that the sample means differ by
5.833 pounds, but now, looking at the data as two different groups of people (i.e., 12
people, instead of 6 people, in the study), even with the right-hand column of people
picking up the weights and putting them in their pocket—the difference of near 6
(i.e., 5.833) will not appear that impressive, given that the weights of individuals
within each column vary a lot. So, the sample mean of column 1 is 166, but a 95%
conidence interval for that value is 166 ± 27.3 or 138.7-193.3, due to the large vari-
ability in weights of the 6 people in that column . For the 6 people in column 2, the
sample mean is 171.833 (5.833 higher than the 166), with a 95% conidence interval
of 171.833 ± 28.0 or 143.833-199.833: again, due to the large variability of weights
of the 6 people in that column . Given these relatively wide conidence intervals for
the true means (and, indeed, they overlap to a larger extent than they don't!), it is
clear that the true means could easily be the same. In fact, if we perform the two-
sample t-test for independent samples, we get the output in Figure 3.9 , with a p -value
0.7095 (see arrow in Figure 3.9 ), nowhere near to being signiicant.
1 By the way, if you wanted to make the argument that here, it makes more sense to consider the one-
sided p -value (it's really a case of “Is there an increase or not?”), then we cut the two-sided p -value in
half, getting even a tinier value: 3.74 E −06.