Self-Reported Metrics - Measuring the User Experience: Collecting, Analyzing, and Presenting Usability Metrics - page 134

A score of 100 would mean that the site was perfect and would require
absolutely no improvement.” This condition was loosely based on a
method called Usability Magnitude Estimation (McGee, 2004) in which
test participants are asked to create their own “usability scale.”

These techniques were compared in an online study. The participants per-
formed six tasks on a live application used to look up information about
employees (phone number, location, manager, etc.). Each participant used only
one of the five self-report techniques. A total of 1131 people participated in the
online study, with at least 210 participants using each self-report technique.

The main goal of this study was to see if these rating techniques are sen-
sitive to detecting differences in perceived difficulty of the tasks. But we also
wanted to see how the perceived difficulty of the tasks corresponded to the task
performance data. We collected task time and binary success data (i.e., whether
users found the correct answer for each task and how long that took). As shown
in Figure 6.4 , there were significant differences in the performance data across
the tasks. Task 2 appears to have been the most challenging, whereas Task 4 was
the easiest.

Number Correct per Minute by Task
(Error bars represent the 95% confidence interval)

2.0

1.8

1.6

1.4

1.2

1.0

0.8

0.6

0.4

0.2

0.0

Task 1 Task 2 Task 3 Task 4 Task 5 Task 6

Figure 6.4 Performance data showing that users had the most difficulty with Task 2 and the least
difficulty with Task 4. Adapted from Tedesco and Tullis (2006); used with permission.

As shown in Figure 6.5 , a somewhat similar pattern of the tasks was reflected
by the task ratings (averaged across all five techniques). In comparing task per-
formance with task ratings, correlations were significant for all five conditions
( p < 0.01). Overall, Spearman rank correlation comparing performance data and
task ratings for the six tasks was significant: Rs = 0.83.