Information Technology Reference
In-Depth Information
80%predictedbyNielsen(2000).However,inthisstudythescopeoftheweb-
sites being evaluated was very large, even though the task of buying something
wasverywelldefined.WoolrychandCockton(2001)discounttheassertionthat
five participants are enough, primarily because it does not take into account
individual differences.
TheanalysesbyLindgaardandChattratichart(2007)ofthenineusability
testsfromCUE-4alsoraisedoubtsaboutthemagicnumber5.Theycompared
the results of two teams, A and H, that both did very well, uncovering 42 and
43%,respectively,ofthefullsetofusabilityproblems.TeamAusedonly6par-
ticipants, whereas Team H used 12. At first glance, this might be seen as evidence
for the magic number 5, as a team that tested only 6 participants uncovered as
manyproblemsasateamthattested12.Butamoredetailedanalysisreveals
a different conclusion. In looking specifically at the overlap of usability issues
between just these two reports, they found only 28% in common. More than
70% of the problems were uncovered by only one of the two teams, ruling out
the possibility of the five-participant rule applying in this case.
THE EVALUATOR EFFECT
The Evaluator Effect (Hornbaek & Frokjaer, 2008; Jacobson, Hertzum, & John,
1998;Vermeern,vanKesteren,&Bekker,2003)inusabilitytestingsuggeststhatUX
professionals identify a different set of usability issues. In other words, there is little
agreement or overlap in the usability issues identified by different UX professionals. The
evaluatoreffecthasbeenobservedconsistentlyintheCUEstudiesledbyRolfMolich
(
http://www.dialogdesign.dk/CUE.html
).Mostrecently,CUE-9(Molich,2011)focused
ontheEvaluatorEffect.Mostofthe34testteamleadersinCUE-9wereconfidentthat
they found the most significant usability issues. However, there was little overlap in the
issues. Furthermore, the test teams felt that running more participants would not help
them identify more usability issues.
How do we reconcile this finding in the context of the recommended number of
participants? It is easy for a UX professional to say that he found most of the usability
issues after testing with 5-10 participants. In fact, they are usually very confident.
ButhowdotheyreallyknowunlesstheycomparetheirfindingstoanotherUX
professional? The fact is, they don't. It is quite possible that additional usability issues,
often significant, may be uncovered with an independent assessment by another UX
professional.
5.7.3
Our
Recommendation
We recommend maintaining flexibility regarding sample sizes in usability tests.
We feel it may be acceptable to test with 5-10 participants, using one UX team
when the following conditions are met:
Search WWH ::
Custom Search