Issue-Based Metrics - Measuring the User Experience: Collecting, Analyzing, and Presenting Usability Metrics - page 102

5.2 HOW TO IDENTIFY AN ISSUE
The most common way to identify usability issues is during a study in which you
are interacting with a participant directly. This might be in person or over the
phone using remote testing technology. A less common way to identify usability
issues is through some automated techniques such as an online study or by observ-
ing a video from a participant, similar to what is generated from a site like usert-
esting.com. This is where you don't have an opportunity to observe participants
directly but only have access to their behavioral and self-reported data. Identifying
issues through this type of data is more challenging but still quite possible.
Possibleusabilityissuesmightbepredictedbeforehandandtrackedduring
testsessions.Butbecarefulthatyou'rereally observing the issues and not just
finding them because you expected to. Your job is certainly easier when you
know what to look for, but you might also miss other issues that you never
considered. In our testing, we typically have an idea of what to look for, but
wealsotrytokeepanopenmindtospotthesurpriseissues.There'sno“right”
approach; it all depends on the goals of the evaluation. When evaluating prod-
ucts that are in an early conceptual stage, it's more likely that you won't have
preset ideas about what the usability issues are. As the product is further refined,
you may have a clearer idea of what specific issues you're looking for.
THE ISSUES YOU EXPECT MAY NOT BE THE ONES YOU FIND
One of the earliest sets of guidelines for designing software interfaces was published by
Apple (1982). It was called the Apple IIe Design Guidelines , and it contained a fascinating
story of an early series of usability tests Apple conducted. They were working on the
design of a program called Apple Presents Apple , which was a demonstration program for
customers to use in computer stores. One part of the interface to which the designers
paid little attention was asking users whether their monitor was monochrome or color.
Theinitialdesignofthequestionwas“Areyouusingablack-and-whitemonitor? ” (They
had predicted that users might have trouble with the word monochrome .) In the first
usability test, they found that a majority of the participants who used a monochrome
monitor answered this question incorrectly because their monitor actually displayed text
in green, not white!
Whatfollowedwasaseriesofhilariousiterationsinvolvingquestionssuchas“Does
yourmonitordisplaymultiplecolors?”or“Doyouseemorethanonecoloronthe
screen?”—allofwhichkeptfailingforsomeparticipants.Indesperation,theywere
considering including a developer with every computer just to answer this question, but
then they finally hit on a question that worked: ''Do the words above appear in several
different colors?'' In short, the issues you expect may not be the issues you find.
5.2.1 In-Person Studies
The best way to facilitate identifying usability issues during an in-person study
is using a think-aloud protocol. This involves having participants verbalize their