Issue-Based Metrics - Measuring the User Experience: Collecting, Analyzing, and Presenting Usability Metrics - page 116

Virzi,1992).Thisisknownasthe“magicnumber5.”Oneofthemostimpor-
tant ways to figure out how many participants are needed in a usability test is to
measure p , or the probability of a usability issue being detected by a single test
participant. It's important to note that this p is different from the p value used
in tests of significance. The probabilities vary from study to study, but they tend
toaveragearound0.3,or30%.[Forareviewofdifferentstudies,seeTurner,
Nielsen,andLewis(2002).]Inaseminalpaper,NielsenandLandauer(1993)
foundanaverageprobabilityof31%basedon11differentstudies.Thisbasi-
callymeansthatwitheachparticipant,about31%oftheusabilityproblemsare
being observed.

Figure 5.7 shows how many issues
are observed as a function of the number
of participants when the probability of
detectionis30%.(Notethatthisassumes
all issues have an equal probability of
detection, which may be a big assump-
tion.) As you can see, after the first partic-
ipant,30%oftheproblemsaredetected;
after the third participant, about 66% of
the problems are observed; and after the
fifthparticipant,about83%oftheprob-
lems have been identified. This claim is
backed up not only by this mathemati-
cal formula, but by a\necdotal evidence
as well. Many UX professionals only test
with five or six participants during an iterative design process. In this situation,
it is relatively uncommon to test with more than a dozen, with a few exceptions.
If the scope of the product is particularly large or if there are distinctly differ-
ent audiences, then a strong case can be made for testing with more than five
participants.

100%

80%

60%

40%

20%

0%

1 2345678910

Number of Participants
Figure 5.7 Example showing how many users are required to observe the
total number of issues in a usability study, given a probability of detection.

CALCULATING P , OR PROBABILITY OF DETECTION

Calculatingtheprobabilityofdetectionisfairlystraightforward.Simplylineupallthe
usability issues discovered during the test. Then, for each participant, mark how many of
the issues were observed with that participant. Add the total number of issues identified
with each participant and then divide by the total number of issues. Each test participant
will have encountered anywhere from 0 to 100% of the issues. Then, take the average
forallthetestparticipants.Thisistheoverallprobabilityrateforthetest.Considerthe
example shown in this table.