Issue-Based Metrics - Measuring the User Experience: Collecting, Analyzing, and Presenting Usability Metrics - page 115

careful participant recruitment, investing in wide task coverage is more fruitful
thanincreasingthenumberofusers.”
One technique that works well to increase task coverage in a usability test is
to define a set of tasks that all participants must complete and another set that
is derived for each participant. These additional tasks might be selected based
on characteristics of the participant (e.g., an existing customer or a prospect) or
mightbeselectedatrandom.Caremustbeexercisedwhenmakingcomparisons
across participants, as not all participants had the same tasks. In this situation,
you may want to limit certain analyses to the core tasks.
THE SPECIAL CASE OF MODERATOR BIAS IN AN
EYE-TRACKING STUDY
One of the more difficult aspects of moderating a usability study is controlling where
you look during the session. Moderators usually are looking either at the participant or
their interaction on a screen, or some other interface. This works well, except in the case
of an eye-tracking study. Most eye-tracking studies measure where participants look,
and whether participants are noting key elements on the interface. As a moderator, it
can be difficult not to look at the target when the participant is scanning the interface.
Participantscanpickuponthiseasily,begintonoticewhere you are looking, and use
that information as a guide to target. It happens very quickly and subtly. While this
behavior has not been reported in the user experience literature, we have observed it
during our own eye-tracking studies. The best thing to do is to be aware of it, and if you
find your eyes starting to wander to the target, simply refocus on the participant, what
they are doing, or, if you have to, some other element on the page. Or don't sit in the
room with the participant, if that's an option. When you're sitting with the participant in
an eye-tracking study, there's also a greater chance that the participant will naturally look
at you and away from the screen.
5.7 NUMBER OF PARTICIPANTS
There has been much debate about how many participants are needed in a
usability test to reliably identify usability issues. [For a summary of the debate,
seeBarnumet al.(2003).]NearlyeveryUXprofessionalseemstohaveanopin-
ion.Notonlyaremanydifferentopinionsfloatingaroundoutthere,butquite
a few compelling studies have been conducted on this very topic. From this
research, two different camps have emerged: those who believe that five partici-
pants are enough to identify most of the usability issues and those who believe
that five is nowhere near enough.
5.7.1 Five Participants is Enough
One camp believes that a majority, or about 80%, of usability issues will be
observedwiththefirstfiveparticipants(Lewis,1994;Nielsen&Landauer,1993;