Planning - Measuring the User Experience: Collecting, Analyzing, and Presenting Usability Metrics - page 50

using the live product and their own accounts (if applicable) and performing
tasks that are relevant only to them. It might also include evaluating the product
in the participants' environments, such as homes or workplaces.
Because they may be performing different tasks and their contexts of use may
be different, comparing across participants may be a challenge. Issue-based met-
rics may be the most appropriate for problem discovery. Assuming you capture
all the usability issues, it's fairly easy to convert those data into frequency and
type. For example, you might discover that 40% of the usability issues pertain
to high-level navigation and 20% of the issues to confusing terminology. Even
though the exact problems encountered by each participant might be differ-
ent, you can still generalize to a higher level category of issue. Examining the
frequency and severity of specific issues will reveal how many repeat issues are
being observed. Is it a one-time occurrence or part of a recurring problem? By
cataloging all the issues and assigning severity ratings, you may come away with
a quick-hit list of design improvements.
3.3.7 Maximizing Usability for a Critical Product
Although some products strive to be easy to use and efficient, such as a mobile
phone or washing machine, critical products have to be easy to use and efficient,
such as a defibrillator, voting machine, or emergency exit instructions on an air-
plane. What differentiates a critical product from a noncritical product is that
the entire reason for the critical product's existence is for the user to complete
a very important task. Not completing that task will have a significant negative
outcome.
Measuring usability for any critical product is essential. Just running a few
participants through the lab is rarely good enough. It's important that user per-
formance be measured against a target goal. Any critical product that doesn't
meet its target usability goal should undergo a redesign. Because of the degree of
certainty you want from your data, you may have to run relatively large numbers
of participants in the study. One very important metric is user errors. This might
include the number of errors or mistakes made while performing a specific task.
Errors are not always easy to tabulate, so special attention must be given to how
you define an error. It's always best to be very explicit about what constitutes an
error and what doesn't.
Task success is also important. We recommend using a binary approach to
success in this situation. For example, the true test of a portable defibrillator
machine is that someone can use it successfully by himself. In some cases, you
may wish to tie task success to more than one metric, such as completing the
task successfully within a specific amount of time and with no errors. Other effi-
ciency metrics are also useful. In the example of the defibrillator machine, sim-
ply using it correctly is one thing, but doing so in a timely manner is altogether
different. Self-reported metrics are relatively less important with critical prod-
ucts. What users think about their use of the product is much less important
than their actual success.