Performance Metrics - Measuring the User Experience: Collecting, Analyzing, and Presenting Usability Metrics - page 90

Home
Page

1

7

3

2 8

4 6

Category
Page A

Category
Page B

Category
Page C

Product
Page A1

Product
Page A2

Product
Page A3

Product
Page B1

Product
Page B2

Product
Page B3

Product
Page C1

Product
Page C2

Product
Page C3

9
Figure 4.8 Actual number of steps a user took in getting to the target item on Product Page C1. Note that
each revisit to the same page is counted, giving a total of nine steps.

5

A perfect lostness score would be 0. Smith (1996) found that participants
with a lostness score less than 0.4 did not exhibit any observable characteris-
tics of being lost. However, she reported that participants with a lostness score
greater than 0.5 definitely did appear to be lost. Note that additional measures
of lostness have been proposed by Otter & Johnson (2000) and Gwizdka &
Spence (2007).

Once you calculate a lostness value, you can easily calculate the average lost-
ness value for each task. The number or percentage of participants who exceed
the ideal number of actions can also be indicative of the efficiency of the design.
For example, you could show that 25% of the participants exceeded the ideal or
minimum number of steps, and you could break it down even further by say-
ing that 50% of the participants completed a task with the minimum number
of actions.

BACKTRACKING METRIC

Treejack ( http://www.optimalworkshop.com/treejack.htm ) is a tool from Optimal
Workshop for testing information architectures (IAs). Participants in a Treejack study
navigate an information hierarchy to indicate where in the hierarchy they would expect
to find a given piece of information or perform some action. Participants can move
down the hierarchy or, if they need to, they can move back up it. Several useful metrics
come out of a Treejack study, including traditional ones, such as where participants
indicated they would expect to find each function. But a particularly interesting metric
is a “backtracking” metric that indicates cases where a participant went back up the
hierarchy. You can then look at the percentage of participants who “backtracked” while
performing each task. In our IA studies, we've found this was often the most revealing
metric.

4.4.3 Efficiency as a Combination of Task Success and Time
Another view of efficiency is that it's a combination of two of the metrics dis-
cussed in this chapter: task success and time on task. The Common Industry