Planning - Measuring the User Experience: Collecting, Analyzing, and Presenting Usability Metrics - page 59

with various financial transactions. As part of identifying criteria, you may seg-
ment participant types. For example, you may recruit a certain number of new par-
ticipants as well as ones who have experience with the existing product.
After deciding on the types of participants you want, you need to figure out
how many you need. As you saw in Section 2.1.2, the number of participants
needed for a usability test is one of the most hotly debated issues in the field.
Many factors enter into the decision, including the diversity of the user popula-
tion, the complexity of the product, and the specific goals of the study. As a gen-
eral rule of thumb, however, testing with about six to eight participants for each
iteration in a formative study works well. The most significant usability findings
will be observed with the first six or so participants. If there are distinct groups
of users, it's helpful to have at least four from each group.
For summative usability studies, we recommend having data from 50 to 100
representative users for each distinct user group. If you're in a crunch, you can go
as low as 30 participants, but the variance in the data will be quite high, making
it difficult to generalize the findings to a broader population. In the case of stud-
ies where you are testing the impact of potentially subtle design changes, having
at least 100 participants for each distinct user group is advisable.
After determining the sample size, you will need to plan the recruiting strat-
egy. This is essentially how you are actually going to get people to participate in
the study. You might generate a list of possible participants from customer data
and then write a screener that a recruiter uses when contacting potential partici-
pants. You might send out requests to participate via e-mail distribution lists.
You can screen or segment participants through a series of background questions
or you might decide to use a third party to handle all of the recruiting. Some
of these companies have quite extensive user panels to draw on. Other options
exist, such as posting an announcement on the web or e-mailing a specific group
ofpotentialparticipants.Differentstrategiesworkfordifferentorganizations.
DOES GEOGRAPHY MATTER?
One of the most common questions we get from our clients is whether we need to
recruit participants from different cities, regions, and countries. The answer is usually
no—geography doesn't matter when collecting usability data. It's very unlikely that
participants in New York are going to have a different set of issues than participants in
Chicago, London, or even Walla Walla, Washington. But there are some exceptions. If
the product you are evaluating has a large corporate presence in one location, it may bias
responses. For example, if you want to test Walmart.com in their hometown of Benton,
Arkansas, you might find it hard to get a neutral, unbiased set of results. Also, location
can have an impact on user goals for some products. For example, if you are evaluating
an e-commerce clothing website, you might collect different data from participants
in urban or rural settings, or participants in different countries, where the needs and
preferences can vary quite a bit. Even when it doesn't really make sense to test in
different locations, some clients still choose to test products in different regions, simply
to prevent senior management from questioning the validity of the results.