Comparing two designs (or anything else!) using independent sample T-tests - Improving the User Experience through Practical Data Analytics - page 67

conidence intervals do overlap that you cannot be certain if H0 would be rejected
by an independent samples t-test or not; conversely, you cannot conclude that H0 is
accepted and that the means are equal.
The reason that this method does not work is somewhat subtle. It rests upon the
fact that the correct conidence interval for the difference of two (independent sam-
ple) means is not the same as the difference in the separate conidence intervals—i.e.,
the conidence intervals for each mean separately.
Let us review what a conidence interval for the mean is. We will then show you
an example in which the aforementioned decision process using the two separate
conidence intervals will yield the incorrect answer.
A conidence interval for the mean is a statement about the value of the true mean.
We take the sample average/mean and add and subtract an amount, “e,” to form the
conidence interval, where the software uses the sample mean, the variability in the
data, and the sample size, along with the conidence level (which we talked about in
Chapter 1 and noted that traditionally, the conidence level is 95%). So, if the sample
mean is 4 (representing, say, the mean satisfaction with a design for some sample
size), and the 95% conidence interval is 2.8-5.2 (note: the center of the interval is 4),
this says that we are 95% conident that the interval 2.8-5.2 contains the true mean
(i.e., the mean for the entire population of people who might use the design and ill
out the satisfaction questionnaire). In looser terms, we are 95% conident that the true
mean is between 2.8 and 5.2. 4
Now consider the following two small data sets, one for each of two designs, with
different people examining each design (indeed, making it an independent sample
situation):

Design 1
Satisfaction Values

Design 2
Satisfaction Values

1 4

3 6

4 5

2 7

Let us refer to the sample means as (X-bar1) and (X-bar2) for Designs 1 and
2, respectively, and suppose a 10-point scale, 1-10. We have X-bar1 = 2.5 and
X-bar2 = 5.5.
The 95% conidence intervals for the true means, μ1 and μ2 (we introduced the
“μ notation” earlier), are:

4 We note here that the highest value possible is 5. Therefore, you might view the conidence interval
as 2.8-5. When the routine formulas for conidence intervals are used, no consideration of the practical
maximum or minimum is applied.