Special Topics - Measuring the User Experience: Collecting, Analyzing, and Presenting Usability Metrics - page 225

We used closed card sorting to compare different ways of organizing the func-
tions for a website (Tullis, 2007). We first conducted an open sort with 54 func-
tions. We then used those results to generate six different ways of organizing the
functions that we then tested in six simultaneous closed card-sorting exercises.
Each closed card sort used the same 54 functions but presented different groups
to sort the functions into. The number of groups in each “framework” (set of
group names) ranged from three to nine. Each participant only saw and used
one of the six frameworks.
In looking at the data from a closed card sort, the main thing you're inter-
ested in is how well the groups “pulled” the cards to them that you intend to
belong to those groups. For example, consider the data in Table 9.9 which shows
the percentage of participants in a closed card-sorting exercise who put each card
into each of the groups.
The other percentage, shown on the right in Table 9.9 , is the highest percent-
age for each card. This is an indicator of how well the “winning” group pulled
the appropriate cards to it. What you hope to see are cases like Card #10 in this
table, which was pulled very strongly to Group C, with 92% of the participants
putting it in that group. Ones that are more troubling are cases such as Card #7,
where 46% of the participants put it in Group A, but 37% put it in Group C—
participants were very “split” in terms of deciding where that card belonged in
this set of groups.
One metric you could use for characterizing how well a particular set of group
names fared in a closed card sort is the average of these maximum values for all the
Table 9.9 Percentage of participants in a closed card sort who put each of 10 cards into each of the
three groups provided.
Card Group A Group B Group C Max
Card #1 17% 78% 5% 78%
Card #2 15% 77% 8% 77%
Card #3 20% 79% 1% 79%
Card #4 48% 40% 12% 48%
Card #5 11% 8% 81% 81%
Card #6 1% 3% 96% 96%
Card #7 46% 16% 37% 46%
Card #8 57% 38% 5% 57%
Card #9 20% 75% 5% 75%
Card #10 4% 5% 92% 92%
Average: 73%
Table 9.9 Percentage of participants in a closed card sort who put each of 10 cards into each of the three
groups provided.