Performance Metrics - Measuring the User Experience: Collecting, Analyzing, and Presenting Usability Metrics - page 80

within an acceptable amount of time. In many
ways, the average is unimportant. The main
goal is to minimize the number of users who
need an excessive amount of time to complete
a task. The main issue is determining what the
threshold should be for any given task. One
way is to perform the task yourself, keeping
track of the time, and then double or triple that
number. Alternatively, you could work with the
product team to come up with a threshold for
each task based on competitive data or even a
best guess. Once you have set your threshold,
simply calculate the percentage of users above
or below the threshold and plot as illustrated
in Figure 4.5 .

Percent of Participants Who Completed Task in Less Than
One Minute

60%

50%

40%

30%

20%

10%

0%

Task 1 Task 2 Task 3 Task 4 Task 5

Figure 4.5 An example showing the percentage of users who
completed each task in less than 1 minute.

DISTRIBUTIONS AND OUTLIERS
Whenever analyzing time data, it's critical to look at the distribution. This is
particularly true for time-on-task data collected via automated tools (when the
moderator is not present). Participants might take a phone call or even go out to
lunch in the middle of a task. The last thing you want is to include a task time of
2 hours among other times of only 15 to 20 seconds when calculating an aver-
age! It's perfectly acceptable to exclude outliers from your analysis, and many
statistical techniques for identifying them are available. Sometimes we exclude
any times that are more than two or three standard deviations above the mean.
Alternatively, we sometimes set up thresholds, knowing that it should never take
a user more than x seconds to complete a task. You should have some rationale
for using an arbitrary threshold for excluding outliers.

The opposite problem—participants apparently completing a task in unusu-
ally short amounts of time—is also common in online studies. Some partici-
pants may be in such a hurry or only care about the compensation that they
simply fly through the study as fast as they can. In most cases, it's very easy
to identify these individuals through their time data. For each task, determine
the fastest possible time. This would be the time it would take someone with
perfect knowledge and optimal efficiency to complete the task. For example, if
there is no way you, as an expert user of the product, can finish the task in less
than 8 seconds, then it is highly unlikely that a typical user could complete the
task any faster. Once you have established this minimum acceptable time, you
should identify the tasks that have times less than that minimum. These are
candidates for removal—not just of the time but of the entire task (including
any other data for the task such as success or subjective rating). Unless you can
find evidence suggesting otherwise, the time indicates that the participant did
not make a reasonable attempt at the task. If a participant did this for multiple
tasks, you should consider dropping that participant. You can expect anywhere