Performance Metrics - Measuring the User Experience: Collecting, Analyzing, and Presenting Usability Metrics - page 68

The most common way to analyze and present binary success rates is by task.
This involves simply presenting the percentage of participants who completed
each task successfully. Figure 4.1 shows the task success rates for the data in
Table 4.1 . This approach is most useful when you want to compare success rates
for each task. You can then do a more detailed analysis of each task by looking
at the specific problems to determine what changes may be needed to address
them. For example, Figure 4.1 shows that the “Find Category” and “Checkout”
tasks appear to be problematic.

Percent Correct, by Task
(Error bars represent the 90% confidence interval)

100%

90%

80%

70%

60%

50%

40%

30%

20%

10%

0%

Login Navigate Search Find

Category

Find
Author

Read
Review

Add to
Cart

Update
Address

Checkout Check

Status

Task

Figure 4.1 Task success rates for the data in Table 4.1 , including a 90% confidence interval for each task.

TYPES OF TASK FAILURE

There are many different ways in which a participant might fail a task, but they tend to
fall into a few categories:

• Giving up —Participants indicate that they would not continue with the task if they

were doing this on their own.

• Moderator “calls” it —The study moderator stops the task because it's clear that the

participant is not making any progress or is becoming especially frustrated.

• Too long —The participant completed the task but not within a predefined time

period. (Certain tasks are only considered successful if they can be accomplished
within a given time period.)

• Wrong —Participants thought that they completed the task successfully, but they

actually did not (e.g., concluding that Neil Armstrong was the Commander of
Apollo 12 instead of Pete Conrad). In many cases, these are the most serious kinds
of task failures because the participants don't realize they are failures. In the real
world, the consequences of these failures may not become clear until much later
(e.g., you intended to order a copy of “Pride and Prejudice” but are rather surprised
when “Pride and Prejudice and Zombies” shows up in the mail several days later!)