Special Topics - Measuring the User Experience: Collecting, Analyzing, and Presenting Usability Metrics - page 224

The “elbow” of that curve appears to be somewhere between 10 and 20, with a sample
size of 15 yielding a correlation of 0.90 with the full data set. Although it's hard to know
how well these results would generalize to other card-sorting studies with different
subject matter or different numbers of cards, they at least suggest that about 15 may be a
good target number of participants.
even just one more dimension yields particularly useful insights into card-sort-
ing data. Another point to keep in mind is that the orientation of the axes in an
MDS plot is arbitrary. You could rotate or flip the map any way you want and
the results would still be the same. The only thing that's actually important is the
relative distances between all pairs of the items.
The most common metric that's used to represent how well an MDS plot
reflects the original data is a measure of “stress” that's sometimes referred to as
Phi . Most of the commercial packages that do MDS analysis can also report the
stress value associated with a solution. Basically, it's calculated by looking at all
pairs of items, finding the difference between each pair's distance in the MDS
map and its distance in the original matrix, squaring that difference, and sum-
ming those squares. That measure of stress for the MDS map shown in Figure
9.5 is 0.04. The smaller the value, the better. But how small does it really need to
be? A good rule of thumb is that stress values under 0.10 are excellent, whereas
stress values above 0.20 are poor.
We find that it's useful to do both a hierarchical cluster analysis and an MDS
analysis. Sometimes you see interesting things in one that aren't apparent in the
other. Because they are different statistical analysis techniques, you shouldn't
expect them to give exactly the same answers. For example, one thing that's
sometimes easier to see in an MDS map is which cards are “outliers”—those
that don't obviously belong with a single group. There are at least two reasons
why a card could be an outlier: (1) It could truly be an outlier—a function that
really is different from all the others, or (2) it could have been “pulled” toward
two or more groups. When designing a website, you would probably want to
make these functions available from each of those clusters.
9.2.2 Analyses of Closed Card-Sort Data
Closed card sorts, where you not only give participants the cards but also the
names of the groups in which to sort them, are probably done less often than
open card sorts. Typically, you would start with an open sort to get an idea of
the kinds of groups that users would naturally create and the names they might
use for them. Sometimes it's helpful to follow up an open sort with one or more
closed sorts, mainly as a way of testing your ideas about organizing the func-
tions and naming the groupings. With a closed card sort you have an idea about
how you want to organize the functions, and you want to see how close users
come to matching the organization you have in mind.