Performance Metrics - Measuring the User Experience: Collecting, Analyzing, and Presenting Usability Metrics - page 69

Another common way of looking at binary success is by user or type of user.
As always in reporting usability data, you should be careful to maintain the ano-
nymity of users in the study using numbers or other nonidentifiable descriptors.
The main value of looking at success data from a user perspective is that you can
identify different groups of users who perform differently or encounter different
sets of problems. Here are some of the common ways to segment different users:

• Frequencyofuse(infrequentusersversusfrequentusers)

• Previousexperienceusingtheproduct

• Domain expertise (low-domain knowledge versus high-domain

knowledge)

• Agegroup

Task success for different groups of participants is also used when each group
is given a different design to work with. For example, participants in a usability
study might be assigned randomly to use either Version A or Version B of a pro-
totype website. A key comparison will be the average task success rate for partici-
pants using Version A vs those using Version B.

If you have a relatively large number of users in a usability study, it may be
helpful to present binary success data as a frequency distribution ( Figure 4.2 ).
This is a convenient way to visually represent the variability in binary task success
data. For example, in Figure 4.2 , six users in the evaluation of the original web-
site completed 61 to 70% of the tasks suc-
cessfully, one completed fewer than 50%,
and only two completed as many as 81 to
90%. In a revised design, six users had a suc-
cess rate of 91% or greater, and no user had
a success rate below 61%. Illustrating that
the two distributions of task success barely
overlap is a much more dramatic way of
showing the improvement across the itera-
tions than simply reporting the two means.

Frequency Distribution of Task Success Rates

8

7

6

5

4

3

Original
Redesign

2

1

0

CALCULATING CONFIDENCE
INTERVALS FOR BINARY SUCCESS
One of the most important aspects of analyz-
ing and presenting binary success is includ-
ing confidence intervals. Confidence intervals
are essential because they reflect your trust or
confidence in the data. In most usability studies, binary success data are based
on relatively small samples (e.g., 5 to 20 users). Consequently, the binary success
metric may not be as reliable as we would like it to be. For example, if 4 out of 5
users completed a task successfully, how confident can we be that 80% of the larger
population of users will be able to complete that task successfully? Obviously, we
would be more confident if 16 out of 20 users completed the task successfully and
even more confident if 80 out of 100 did.

Task Success Rate

Figure 4.2 Frequency distributions of binary success rates from
usability tests of the original version of a website and the redesigned
version (data from LeDoux, Connor, & Tullis, 2005).