Background - Measuring the User Experience: Collecting, Analyzing, and Presenting Usability Metrics - page 27

EXCEL TIP
As illustrated in Figure 2.3, you can use the TTEST function in Excel to run a t test:
=TTEST(Array 1, Array 2, Tails, Type)
Array 1 and Array 2 refer to the sets of values that you want to compare. In Figure 2.3,
Array 1 is the set of ratings for Design 1 and Array 2 is the set of ratings for Design
2. Tails refer to whether your test is one-tailed or two-tailed. This relates to the tails
(extremes) of the normal distribution and whether you're considering one end or both
ends. From a practical standpoint, this is asking whether it is theoretically possible for
the difference between these two means to be in either direction (i.e., Design 1 either
higher or lower than Design 2). In almost all cases that we deal with, the difference
could be in either direction, so the correct choice is “2” for two tailed. Finally, Type
indicates the type of t test. For these independent samples (not paired), the Type is 2.
This t test returns a value of 0.047. So how do you interpret that? It's telling you the
probability that the difference between these two means is simply due to chance. So
there's a 4.7% chance that this difference is not significant. Since we were dealing with a
95% confidence interval, or a 5% alpha level, and this result is less than 5%, we can say
that the difference is statistically significant at that level.
2.4.2 Paired Samples
A paired samples t test is used when comparing means within the same set of
users. For example, you may be interested in knowing whether there is a differ-
ence between two prototype designs. If you have the same set of users perform
tasksusingprototypeAandthenprototypeB,andyouaremeasuringvariables
such as self-reported ease of use and time, you will use a paired samples t test.
With paired samples like these, the key is that you're comparing each per-
son to themselves. Technically, you're looking at the difference in each person's
data for the two conditions you're comparing. Let's consider the data shown in
Figure 2.4 , which shows “Ease of Use” ratings for an application after their ini-
tial use and then again at the end of the session. So there were 10 participants
who gave two ratings each. The means and 90% confidence intervals are shown
and have been graphed. Note that the confidence intervals overlap pretty widely.
If these were independent samples, you could conclude that the ratings are not
significantly different from each other. However, because these are paired sam-
ples, we've done a t test on paired samples (with the “Type” as “1”). That result,
0.0002, shows that the difference is highly significant.
Let's look at the data in Figure 2.4 in a slightly different way, as shown in
Figure 2.5 . This time we've simply added a third column to the data in which
the initial rating was subtracted from the final rating for each participant. Note
that for 8 of the 10 participants, the rating increased by one point, whereas for
2 participants it stayed the same. The bar graph shows the mean of those dif-
ferences (0.8) as well as the confidence interval for that mean difference. In a