Performance Metrics - Measuring the User Experience: Collecting, Analyzing, and Presenting Usability Metrics - page 85

Error Rate by Condition

40%

35%

30%

25%

20%

15%

10%

5%

0%

Control Condition
1

Condition
2

Condition
3

Condition
4

Condition
5

Condition
6

Condition
7

Condition
8

Figure 4.6 An example showing how to present data for single error opportunities. In this study, only
one error opportunity per task (entering a password incorrectly) was possible, and the graph shows the
percentage of participants who made an error for each condition.

errors based on a single opportunity per task. In this example, they were inter-
ested in the percentage of participants who experienced an error when using dif-
ferenttypesofon-screenkeyboards(Tullis,Mangan,&Rosenbaum,2007).The
controlconditionwasthecurrentQWERTYkeyboardlayout.

In many situations, there are multiple opportunities for errors per task (e.g.,
multiple input fields in a “new account” application). Here are some of the
common ways to analyze data from tasks with multiple error opportunities:

• Agoodplacetostartistolookatthefrequencyoferrorsforeachtask.You

will be able to see which tasks are resulting in the most errors. But this
may be misleading if each task has a different number of error opportu-
nities. In that case, it might be better to divide the total number of errors
for the task by the total number of error opportunities. This creates an
error rate that takes into account the number of opportunities.

• Youcouldcalculatetheaveragenumberoferrorsmadebyeachpartici-

pant for each task. This will also tell you which tasks are producing the
most errors. However, it may be more meaningful because it suggests that
a typical user might experience x number of errors on a particular task
when using the product. Another advantage is that it takes into account
extremes. If you are simply looking at the frequency of errors for each
task, some users may be the source of most of the errors, whereas many
others are performing the task error-free. By taking an average number of
errors for each user, this bias is reduced.