Special Topics - Measuring the User Experience: Collecting, Analyzing, and Presenting Usability Metrics - page 218

the right. They measured the click rates on the articles (to read the full version of
the article). Version B, with article photos always to the right, increased clicks by
20% and total site pages viewed by 11%.
WHICH TESTWON.COM
Anne Holland runs a website called “Which Test Won” that's a treasure trove of
examples of A/B tests. As of the writing of the second edition of this topic, she
has about 300 examples of different A/B tests on her site, ranging from tests that
manipulated entire page designs to tests where the only difference was the color of a
single button. She posts a new test every week, encouraging readers to guess whether
the A or the B version of the test won. She also has a free e-mail newsletter alerting
readers to new tests on the site.
Carefully designed A/B tests can give you significant insight into what works
and what doesn't work on your website. Many companies, including Amazon,
eBay, Google, Microsoft, Facebook, and others, are constantly doing A/B tests
on their live sites, although most users don't notice it (Kohavi, Crook, &
Longbotham, 2009; Kohavi, Deng, Frasca, Longbotham, Walker, & Xu, 2012;
Tang, Agarwal, O'Brien, & Meyer, 2010). In fact, as Kohavi and Round (2004)
explained, A/B testing is constant at Amazon, and experimentation through A/B
testing is the main way they make changes to their site.
9.2 CARD-SORTING DATA
Card sorting as a technique for organizing the elements of an information
system in a way that makes sense to the users has been around at least since
the early 1980s. For example, Tullis (1985) used the technique to organize
the menus of a mainframe operating system. More recently, the technique
has become popular as a way of informing decisions about the information
architecture of a website (e.g., Maurer & Warfel, 2004; Spencer, 2009). Over
the years the technique has evolved from a true card-sorting exercise using
index cards to an online exercise using virtual cards. Although many UX pro-
fessionals seem to be familiar with the basic card-sorting techniques, fewer
seem to be aware that various metrics can be used in the analyses of card-sort-
ing data.
The two major types of card-sorting exercises are (1) open card sorts, where
you give users the cards that are to be sorted but let them define their own
groups that the cards will be sorted into, and (2) closed card sorts, where you
give users the cards to be sorted as well as the names of the groups to sort them
into. Although some metrics apply to both, others are unique to each.