Planning - Measuring the User Experience: Collecting, Analyzing, and Presenting Usability Metrics - page 58

it is a formative study, you should make every attempt to get the findings back
to the stakeholders as quickly as possible to influence the next design iteration
and not slow down the project.
In the case of running a lab test with a larger number of participants (usually
more than a dozen), including metrics may have more of an impact on the bud-
get and timeline. The most significant cost impact may be any additional costs
for recruiting and compensating the participants. These costs depend on who
they are (e.g., internal to your company versus external), how participants are
recruited, and whether the test will be in a local lab or conducted with remote
sessions. The most significant impact on the timeline is likely to be the addi-
tionaltimerequiredtorunthelargernumberofparticipants.Dependingon
your billing or cost-recovery model, there may also be additional costs because
of the increased time for the usability specialists. Keep in mind that you will also
need extra time to clean up and analyze the data.
Running an online (unmoderated) study is quite different in terms of costs and
time. Typically, about half of the time is usually spent setting up the study, from
identifying and validating tasks, creating questions and scales, evaluating the pro-
totypes or designs, identifying and/or recruiting participants, and developing the
online script. Unlike traditional lab tests where a lot of time is spent collecting the
data, running an online study requires little, if any, time on the part of the usability
specialist for data collection. With most online usability testing technologies you
simply flip the switch and then monitor the data as they pour in.
The other half of the time is spent cleaning up and analyzing the data. It's
verycommontounderestimatethetimerequiredforthis.Dataareoftennot
in a format that readily allows analysis. For example, you will need to filter out
extreme values (particularly when collecting time data), check for data inconsis-
tencies, and code new variables based on the raw data (such as creating top-2-
box variables for self-reported data). We have found that we can run an online
study in about 100 to 200 person-hours. This includes everything from the plan-
ning phase through data collection, analysis, and presentation. The estimate can
vary by up to 50% in either direction based on the scope of the study. Many of
these details are covered in the topic “Beyond the Usability Lab: Conducting
Large-scale Online User Experience Studies” (Albert, Tullis, & Tedesco, 2010).
3.5.2 Participants
The specific participants in any usability study have a major impact on its find-
ings. It's critical that you carefully plan how to include the most representative
participants as possible in your study. The steps you will go through in recruiting
participants are essentially the same whether you're collecting metrics or not.
The first step is to identify recruiting criteria that will be used to determine
whether a specific person is eligible to participate in the study. Criteria should be
as specific as possible to reduce the possibility of recruiting someone who does
not fit the profile(s). We often recruit participants based on many characteristics,
including their experience with the web, years away from retirement, or experience