Performance Metrics - Measuring the User Experience: Collecting, Analyzing, and Presenting Usability Metrics - page 94

4.5.1 Collecting and Measuring Learnability Data
The process of collecting and measuring learnability data is basically the same as
it is for the other performance metrics, but you're collecting the data at multiple
times. Each instance of collecting the data is considered a trial. A trial might be
every 5 minutes, every day, or once a month. The time between trials, or when
you collect the data, is based on expected frequency of use.

The first decision is which type of metrics you want to use. Learnability can
be measured using almost any performance metric over time, but the most com-
mon ones are those that focus on efficiency, such as time on task, errors, num-
ber of steps, or task success per minute. As learning occurs, you expect to see
efficiency improve.

After you decide which metrics to use, you need to decide how much time
to allow between trials. What do you do when learning occurs over a very long
time? What if users interact with a product once every week, month, or even
year? The ideal situation would be to bring the same participants into the lab
every week, month, or even year. In many cases, this is not very practical. The
developers and the business sponsors might not be very pleased if you told them
the study will take 3 years to complete. A more realistic approach is to bring in
the same participants over a much shorter time span and acknowledge the limi-
tation in the data. Here are a few alternatives:

• Trials within the same session . The participant performs the task, or set of

tasks, one right after the other, with no breaks in between. This is very easy
to administer, but it does not take into account significant memory loss.

• Trials within the same session but with breaks in between each task . The break

might be a distracter task or anything that might promote forgetting. This
is fairly easy to administer, but it tends to make each session relatively long.

• Trials between sessions : The participant performs the same tasks over

multiple sessions, with at least 1 day in between. This may be the least
practical, but most realistic, if the product is used
sporadically over an extended period of time.

Task Time (secs) Over Trials

60

4.5.2 Analyzing and Presenting
Learnability Data
The most common way to analyze and present learn-
ability data is by examining a specific performance
metric (such as time on task, number of steps, or
number of errors) by trial for each task or aggregated
across all tasks. This will show you how that perfor-
mance metric changes as a function of experience,
as illustrated in Figure 4.11 . You could aggregate all
the tasks together and represent them as a single
line of data or you could look at each task as sepa-
rate lines of data. This can help determine how the

50

40

30

20

10

0

Tr ial 1Trial 2Trial 3Trial 4Trial 5Trial 6Trial 7

Figure 4.11 An example of how to present learnability data
based on time on task.