Performance Metrics - Measuring the User Experience: Collecting, Analyzing, and Presenting Usability Metrics - page 71

optical zoom instead of 12×? According to a strict binary success approach, that
would be a failure. But you're losing some important information by doing that.
The user actually came very close to completing the task successfully. In some
cases, this might be acceptable to a user. For some types of products, coming
close to fully completing a task may provide value to the user. Also, it may be
helpful for you to know why some users failed a task or with which particular
tasks users needed help.

SHOULD YOU INCLUDE TASKS THAT CAN'T BE DONE?

An interesting question is whether a usability study should include tasks that can't be
done using the product being testing. For example, assume you're testing an online
bookstore that only carries mystery novels. Would it be appropriate to include a task that
involves trying to find a topic that the store doesn't carry, such as a science-fiction novel?
If one of the goals of the study is to determine how well users can determine what the
store does not carry, we think it could make sense. In the real world, when you come to
a new website, you don't automatically know everything that can or can't be done using
the site. A well-designed site not only makes it clear what is available on the site, but
also what's not available. However, when tasks are presented in a usability study, there's
probably an implicit understanding that they can be done. So we think if you do include
tasks that can't be done, you should make it clear up front that some of the tasks may
not be possible.

HOW TO COLLECT AND MEASURE LEVELS OF SUCCESS
Collecting and measuring levels of success data is very similar to binary suc-
cess data except that you must define the various levels. There are a couple of
approaches to levels of success:

• Basedontheuser'sexperienceincompletingatask.Someusersmight

struggle or require assistance, while others complete their tasks without
any difficulty.

• Basedontheusersaccomplishingthetaskindifferentways.Someusers

might accomplish the task in an optimal way, while others might accom-
plish it in ways that are less than optimal.

Levels of success based on the degree to which users complete a task typically
have between three and six levels. A common approach is to use three levels:
complete success, partial success, and complete failure.

Levels of success data are almost as easy to collect and measure as binary suc-
cess data. It just means defining what you mean by “complete success” and by
“complete failure.” Anything in between is considered a partial success. A more