Issue-Based Metrics - Measuring the User Experience: Collecting, Analyzing, and Presenting Usability Metrics - page 103

thoughts as they are working through the tasks. Typically, the participants are
reporting what they are doing, what they are trying to accomplish, how confi-
dent they are about their decisions, their expectations, and why they performed
certain actions. Essentially, it's a stream of consciousness focusing on their inter-
action with the product. During a think-aloud protocol, you might observe the
following:

• Verbalexpressionsofconfusion,frustration,dissatisfaction,pleasure,or

surprise

• Verbalexpressionsofconfidenceorindecisionaboutaparticularaction

that may be right or wrong

• Participants not saying or doing something that they should have done or

said

• Nonverbalbehaviorssuchasfacialexpressionsand/oreyemovements

In addition to listening to participants, it is very important to observe their
behavior. Watching what they are doing, where they struggle, and how they suc-
ceed provides a great way to identify usability issues.

5.2.2 Automated Studies
Identifying usability issues through automated studies requires careful data col-
lection. The key is to allow participants to enter verbatim comments at a page or
task level. In most automated studies, several data points are collected for each
task:success,time,ease-of-userating,andverbatimcomments.Verbatimcom-
ments are the best way to understand any possible issues.

One way to collect verbatim comments is to require the participant to pro-
vide a comment at the conclusion of each task. This might yield some interest-
ing results, but it doesn't always yield the best results. An alternative that seems
to work better is to make the verbatim comment conditional. If the participant
provides a low ease-of-use score (e.g., not one of the two highest ratings), then
she is asked to provide feedback about why she rated the task that way. Having
a more pointed question usually yields more specific, actionable comments.
For example, participants might say that they were confused about a particu-
lar term or that they couldn't find the link they wanted on a certain page. This
type of task-level feedback is usually more valuable than one question after they
complete all the tasks (post-study). The only downside of this approach is if
the participant adjusts his ratings, after several questions, in order to avoid the
open-ended question.

5.3 SEVERITY RATINGS
Notallusabilityissuesarethesame:Somearemoreseriousthanothers.Some
usability issues mildly annoy or frustrate users, whereas others cause them to
make the wrong decisions or lose data. Obviously, these two different types of
usability issues have a very different impact on the user experience, and severity
ratings are a useful way to deal with them.