Special Topics - Measuring the User Experience: Collecting, Analyzing, and Presenting Usability Metrics - page 214

click rates for two different links. The click-through rate for Link #1 is 1.4%
[145/(145 + 10,289)]. The click-through rate for Link #2 is 1.7% [198/(198 +
11,170)]. But are these two significantly different from each other? Link #2 got
more clicks, but it was also presented more times. To do a χ 2 test, you must first
construct a table of expected frequencies as if there were no difference in the
click-through rates of Link #1 and Link #2. This is done using the sums of the
rows and columns of the original table, as shown in Table 9.3 .

Table 9.3 Same data as Table 9.2 but with sums of rows and columns added a .
Observed Click No Click Sum

Link #1 145 10289 10434

Link #2 198 11170 11368

Sum 343 21459 21802

aThese are used to calculate expected frequencies if there were no differences in the click-through rates.
Table 9.3 Same data as Table 9.2 but with sums of rows and columns added a .
a These are used to calculate expected frequencies if there were no differences in the click-through rates.

By taking the product of each pair of row and column sums and divid-
ing that by the grand total you get the expected values as shown in Table 9.4 .
For example, the expected frequency for a “Click” on “Link #1” (164.2) is the
product of the respective row and column sums divided by the grand total:
(343×10,434)/21,802. The “CHITEST” function in Excel can then be used to
compare the actual frequencies in Table 9.2 to the expected frequencies in Table
9.4 . The resulting value is p = 0.04, indicating that a significant difference exists
between the click-through rates for Link #1 and Link #2.

Table 9.4 Expected frequencies if there were no differences in
click-through rates for Link #1 and Link #2, derived from sums
shown in Table 9.3.
Expected

Click No Click

Link #1 164.2 10269.8

Link #2 178.8 11189.2

Table 9.4 Expected frequencies if there were no differences in click-through rates for Link #1 and Link
#2, derived from sums shown in Table 9.3.

You should keep two important points about the χ 2 test in mind. First, the
χ 2 test must be done using raw frequencies or counts, not percentages. You com-
monly think of click-through rates in terms of percentages, but that's not how
you test for significant differences between them. Also, the categories used must
be mutually exclusive and exhaustive , which is why the preceding example used
“Click” and “No Click” as the two categories of observations for each link. Those
two categories are mutually exclusive and account for all possible actions that
could be taken on the link—either the user clicked on it or didn't.