Performance Metrics - Measuring the User Experience: Collecting, Analyzing, and Presenting Usability Metrics - page 82

RETROSPECTIVE THINK ALOUD (RTA)
A technique that's gaining in popularity among many usability professionals is
retrospective think aloud (e.g., Birns, Joffre, Leclerc, & Paulsen, 2002; Guan, Lee,
Cuddihy,&Ramey,2006;Petrie&Precious,2010).Withthistechnique,participants
typically remain silent while they are interacting with the product being tested. Then,
after all the tasks, they are shown some kind of “reminder” of what they did during the
session and are asked to describe what they were thinking or doing at various points
in the interaction. The reminder can take several different forms, including a video
replay of screen activity, perhaps with a camera view of the user, or an eye-tracking
replay showing what the user was looking at. This technique probably yields the most
accurate task time data. There's also some evidence that the additional cognitive load
of concurrent think aloud causes participants to be less successful with their tasks. For
example, van den Haak, de Jong, and Schellens (2004) found that participants in a
usability study of a library website were successful with only 37% of their tasks when
usingconcurrentthinkaloud,buttheyweresuccessfulwith47%whenusingRTA.But
keepinmindthatitwilltakealmosttwiceaslongtorunsessionsusingRTA.
SHOULD YOU TELL PARTICIPANTS ABOUT THE TIME
MEASUREMENT?
An important question to consider is whether to tell participants you are record-
ing their time. It's possible that if you don't, participants won't behave in an effi-
cient manner. It's not uncommon for participants to explore different parts of a
website when they are in the middle of a task. On the flip side, if you tell them
they are being timed, they may become nervous and feel they are the ones being
tested and not the product. A good compromise is asking the participants to per-
form the tasks as quickly and accurately as possible, without volunteering that
they are being explicitly timed. If the participant happens to ask (which they rarely
do), then simply state that you are noting the start and finish time for each task.
4.3 ERRORS
Some user experience professionals believe errors and usability issues are essen-
tially the same thing. Although they are certainly related, they are actually quite
different. A usability issue is the underlying cause of a problem, whereas one or
more errors are a possible outcome of an issue. For example, if users are experi-
encing a problem in completing a purchase on an e-commerce website, the issue
(or cause) may be confusing labeling of the products. The error, or the result of
the issue, may be the act of choosing the wrong options for the product they
want to buy. Essentially, errors are incorrect actions that may lead to task failure.
4.3.1 When to Measure Errors
In some situations it's helpful to identify and classify errors rather than just docu-
ment usability issues. Measuring errors is useful when you want to understand the