Planning - Measuring the User Experience: Collecting, Analyzing, and Presenting Usability Metrics - page 53

3.4.1 Traditional (Moderated) Usability Tests
The most common usability method is a lab test that utilizes a relatively small
number of participants (typically 5 to 10). The lab test involves a one-on-one
session between a moderator (usability specialist) and a test participant. The
moderator asks questions of the participants and gives them a set of tasks to
perform on the product in question. The test participant is likely to be thinking
aloud as she performs the various tasks. The moderator records the participant's
behavior and responses to questions. Lab tests are used most often in forma-
tive studies where the goal is to make iterative design improvements. The most
important metrics to collect are about issues, including issue frequency, type,
and severity. Also, collecting performance data such as task success, errors, and
efficiency may also be helpful.
Self-reported metrics can also be collected by having participants answer
questions regarding each task or at the conclusion of the study. However, we
recommend that you approach performance data and self-reported data very
carefully because it's easy to overgeneralize the results to a larger population
without an adequate sample size. In fact, we typically only report the frequency
of successful tasks or errors. We hesitate even to state the data as a percentage
for fear that someone (who is less familiar with usability data or methods) will
overgeneralize the data.
Usability tests are not always run with a small number of participants. In
some situations, such as comparison tests, you might want to spend some extra
time and money by running a larger group of participants (perhaps 10-50
users). The main advantage of running a test with more participants is that as
your sample size increases, so does your confidence in your data. Also, this will
afford you the ability to collect a wider range of data. In fact, all performance,
self-reported, and physiological metrics are fair game. But there are a few met-
rics that you should be cautious about. For example, inferring website traffic pat-
terns from usability-lab data is probably not very reliable, nor is looking at how
subtle design changes might impact the user experience. In these cases, it is bet-
ter to test with hundreds or even thousands of participants in an online study.
FOCUS GROUPS VERSUS USABILITY TESTS
When some people first hear about usability testing, they believe it is the same as a
focus group. But in our experience, the similarity between the two methods begins and
ends with the fact that they both involve representative participants. In a focus group,
participants commonly watch someone demonstrate or describe a potential product
and then react to it. In a usability test, participants actually try to use some version of
the product themselves. We've seen many cases where a prototype got rave reviews from
focus groups and then failed miserably in a usability test.