Performance Metrics - Measuring the User Experience: Collecting, Analyzing, and Presenting Usability Metrics - page 95

learnability of different tasks compare, but it also can also make the chart harder
to interpret.

The first aspect of the chart you should notice is the slope of the line(s).
Ideally, the slope (sometimes called the learning curve) is fairly flat and low on
the y axis (in the case of errors, time on task, number of steps, or any other met-
ric where a smaller number is better). If you want to determine whether a statis-
tically significant difference between the learning curves (or slopes) exists, you
need to perform an analysis of variance and see if there is a main effect of trial.
(See Chapter 2 for a discussion of analysis of variance.)

You should also notice the point of asymptote, or essentially where the line
starts to flatten out. This is the point at which users have learned as much as
they can, and there is very little room for improvement. Project team mem-
bers are always interested in how long it will take someone to reach maximum
performance.

Finally, you should look at the difference between the highest and the lowest
values on the y axis. This will tell you how much learning must occur to reach
maximum performance. If the gap is small, users will be able to learn the prod-
uct quickly. If the gap is large, users may take quite some time to become pro-
ficient with the product. One easy way to analyze the gap between highest and
lowest scores is by looking at the ratio of the two. Here is an example:

• Iftheaveragetimeonthefirsttrialis80secondsandonthelasttrialis60

seconds, the ratio shows that users are initially taking 1.3 times longer.

• Iftheaveragenumberoferrorsonthefirsttrialis2.1andonthelasttrial

is 0.3, the ratio shows a 7 times improvement from the first trial to the
last trial.

Task Time (secs) Across Trials

It may be helpful to look at how
many trials are needed to reach max-
imum performance. This is a good
way to characterize the amount of
learning required to become profi-
cient in using the product.

30

25

Control
Condition 1
Condition 2
Condition 3
Condition 4
Condition 5
Condition 6
Condition 7
Condition 8

20

In some cases you might want to
compare learnability across differ-
ent conditions, as shown in Figure
4.12 . In this study (Tullis, Mangan, &
Rosenbaum, 2007), they were inter-
ested in how speed (efficiency) of
entering a password changed over
time using different types of on-
screen keyboards. As you can see
from the data, there is an improve-
ment from the first trial to the second
trial, but then the times flatten out

15

10

5

0

Tr ial 1 Trial 2 Trial 3 Trial 4

Figure 4.12 Looking at the learnability of different types of on-screen
keyboards.