Planning - Measuring the User Experience: Collecting, Analyzing, and Presenting Usability Metrics - page 45

the user experience. We were curious about the correlations we've seen between two
measures of performance (task success and task time) and one measure of satisfaction
(task ease rating). We looked at data from 10 online usability studies we've run.
The number of participants in each of these studies ranged from 117 to 1036. The
correlations between task time and task rating were mostly negative, as you would
expect (the longer it takes, the less satisfied you are), but ranged from −0.41 to +0.06.
The correlations between task success and task rating were at least all positive, ranging
from 0.21 to 0.65. Together, these results suggest that a relationship exists between
performance and satisfaction, but not always.
3.3 CHOOSING THE RIGHT METRICS: TEN TYPES OF
USABILITY STUDIES
Some of the issues you should consider when choosing metrics for a usability study
include the goals of the study and the user, the technology that's available to collect
the data, and the budget and time you have to turn around your findings. Because
every usability study has unique qualities, we can't prescribe the exact metrics to
use for every type of study. Instead, we've identified 10 prototypical categories of
usability studies and developed recommendations about metrics for each. The rec-
ommendations we offer are simply suggestions that should be considered when
running a usability study with a similar set of characteristics. Conversely, metrics
that may be essential to your study may not be on the list. Also, we strongly recom-
mend that you explore your raw data and develop new metrics that are meaningful
to your project goals. Ten common usability study scenarios are listed in Table 3.1 .
The metrics that are used commonly or are appropriate for each of the scenarios are
indicated. The following sections discuss each of the 10 scenarios.
3.3.1 Completing a Transaction
Many usability studies are aimed at making transactions run as smoothly as pos-
sible. These might take the form of a user completing a purchase, registering a
new piece of software, or resetting a password. A transaction usually has a well-
defined beginning and end. For example, on an e-commerce website, a transac-
tion may start when a user places something in his shopping cart and ends when
he has completed the purchase on the confirmation screen.
Perhaps the first metric that you will want to examine is task success. Each
task is scored as a success or failure. Obviously the tasks need to have a clear end
state, such as reaching a confirmation that the transaction was successful.
Reporting the percentage of participants who were successful is an excellent
measure of the overall effectiveness of the transaction. If the transaction involves
a website or some live website metrics, such a drop-off rate from the transaction
can be very useful. By knowing where users are dropping off, you will be able to
focus your attention on the most problematic steps in the transaction.