Special Topics - Measuring the User Experience: Collecting, Analyzing, and Presenting Usability Metrics - page 226

cards. For the data in Table 9.9 , that would be 73%. But what if you want to compare
results from closed card sorts that had the same cards but different sets of groups? That
average maximum percentage will work well for comparisons as long as each set con-
tained the same number of groups. But if one set had only three groups and another
had nine groups, as in the Tullis (2007) study, it's not a fair metric for comparison. If
participants were simply acting randomly in doing the sorting with only three groups,
by chance they would get a maximum percentage of 33%. But if they were acting ran-
domly in doing a sort with nine groups, they would get a maximum percentage of
only 11%. So using this metric, a framework with more groups is at a disadvantage in
comparison to one with fewer groups.

We experimented with a variety of methods to correct for the number of
groups in a closed card sort. The one that seems to work best is illustrated in
Table 9.10 . These are the same data as shown earlier in Table 9.9 but with two
additional columns. The “2nd place” column gives the percentage associated
with the group that had the next-highest percentage. The “Difference” column
is simply the difference between the maximum percentage and the 2nd-place
percentage. A card that was pulled strongly to one group, such as Card #10, gets
a relatively small penalty in this scheme. But a card that was split more evenly,
such as Card #7, takes quite a hit.

The average of these differences can then be used to make comparisons
between frameworks that have different numbers of groups. For example, Figure
9.6 shows the data from Tullis (2007) plotted using this method. We call this a
measure of the percent agreement among the participants about which group
each card belongs to. Obviously, higher values are better.

Table 9.10 Same data as shown in Table 9.9 but with an additional two columns a .
Card Category A Category B Category C Max 2nd Place Difference
Card #1 17% 78% 5% 78% 17% 61%
Card #2 15% 77% 8% 77% 15% 62%
Card #3 20% 79% 1% 79% 20% 60%
Card #4 48% 40% 12% 48% 40% 8%
Card #5 11% 8% 81% 81% 11% 70%
Card #6 1% 3% 96% 96% 3% 93%
Card #7 46% 16% 37% 46% 37% 8%
Card #8 57% 38% 5% 57% 38% 18%
Card #9 20% 75% 5% 75% 20% 55%
Card #10 4% 5% 92% 92% 5% 87%
Average: 73% 52%
a “2nd place” refers to the next-highest percentage after the maximum percentage, and “Difference” indicates the difference between the
maximum percentage and the 2nd-place percentage.

Table 9.10 Same data as shown in Table 9.9 but with an additional two columns a .
a “2nd place” refers to the next-highest percentage after the maximum percentage, and “Difference” indicates the difference between the
maximum percentage and the 2nd-place percentage.