Performance Metrics - Measuring the User Experience: Collecting, Analyzing, and Presenting Usability Metrics - page 67

Table 4.1 Task success data for 10 participants and 10 tasks.
Task
1

Task
2

Task
3

Task
4

Task
5

Task
6

Task
7

Task
8

Task
9

Task
10

Average

Participant 1 1 1 1 0 1 1 1 1 0 1 80%

Participant 2 1 0 1 0 1 0 1 0 0 1 50%

Participant 3 1 1 0 0 0 0 1 0 0 0 30%

Participant 4 1 0 0 0 1 0 1 1 0 0 40%

Participant 5 0 0 1 0 0 1 0 0 0 0 20%

Participant 6 1 1 1 1 1 0 1 1 1 1 90%

Participant 7 0 1 1 0 0 1 1 1 0 1 60%

Participant 8 0 0 0 0 1 0 0 0 0 1 20%

Participant 9 1 0 0 0 0 1 1 1 0 1 50%

Participant 10 1 1 0 1 1 1 1 1 0 1 80%

Average 70% 50% 50% 20% 60% 50% 80% 60% 10% 70% 52.0%

Table 4.1 Task success data for 10 participants and 10 tasks.

than one participant and more than one task, there are always two ways you can
calculate task success:

• Bylookingattheaveragesuccessrateforeach task across the participants

• By looking at the average success rate for each participant across the

tasks

As an example, consider the data in Table 4.1 . Averages across the bottom
represent the task success rates for each task . Averages along the right represent
the success rates for each participant . As long as there are no missing data, the
averages of those two sets of averages will always be the same.

DOES TASK SUCCESS ALWAYS MEAN FACTUAL SUCCESS?

The usual definition of task success is achieving some factually correct or clearly defined
state. For example, if you're using the NASA site to find who the Commander of Apollo
12 was, there's a single factually correct answer (Charles “Pete” Conrad, Jr.). Or if you're
using an e-commerce site to purchase a copy of “Pride and Prejudice,” then purchasing
that topic would indicate success. But in some cases, perhaps what's important is not
so much reaching a factual answer or achieving a specific goal, but rather users being
satisfied they have achieved a certain state. For example, just before the 2008 U.S.
presidential election, we conducted an online study comparing the websites of the two
primary candidates, Barack Obama and John McCain. The tasks included things such as
finding the candidate's position on Social Security. Task success was measured by self-
report only (Yes I Found It, No I Didn't Find, or I'm Not Sure), as for this kind of site the
important thing is whether users believe they found the information they were looking
for. Sometimes it can be very interesting to look at the correlation between perceived
success and factual success.