Information Technology Reference
In-Depth Information
Although the second task may be perfectly appropriate in certain types of
usability studies, it's not appropriate for measuring task success.
The most common way of measuring success in a lab-based usability test is
to have the user articulate the answer verbally after completing the task. This
is natural for the user, but sometimes it results in answers that are difficult to
interpret. Users might give extra or arbitrary information that makes it difficult
to interpret the answer. In these situations, you may need to probe the users to
make sure they actually completed the task successfully.
Another way to collect success data is by having users provide their answers
in a more structured way, such as using an online tool or paper form. Each task
might have a set of multiple-choice responses. Users might choose the correct
answer from a list of four to five distracters. It's important to make the distracters
as realistic as possible. Try to avoid write-in answers if possible. It's much more
time-consuming to analyze each write-in answer, and it may involve judgment
calls, thereby adding more noise to the data.
In some cases the correct solution to a task may not be verifiable because it
depends on the user's specific situation, and testing is not being performed in per-
son. For example, if you ask users to find the balance in their savings account, there's
no way to know what that amount really is unless we're sitting next to them while
they do it. So in this case, you might use a proxy measure of success. For example,
you could ask users to identify the title of the page that shows their balance. This
works well as long as the title of the page is unique and obvious and you're confi-
dent that they are able to actually see the balance if they reached this page.
4.1.1 Binary Success
Binary success is the simplest and most common way of measuring task success.
Users either completed a task successfully or they didn't. It's kind of like a “pass/
fail” course in college. Binary success is appropriate to use when the success of
the product depends on users completing a task or set of tasks. Getting close
doesn't count. The only thing that matters is that they accomplish a goal with
their tasks. For example, when evaluating the usability of a defibrillator device
(to resuscitate people during a heart attack), the only thing that matters is being
able to use it correctly without making any mistakes within a certain amount of
time. Anything less would be a major problem, especially for the recipient! A
less dramatic example might be a task that involves a goal of purchasing a topic
on a website. Although it may be helpful to know where in the process someone
failed, if your company's revenue depends on selling those topics, that's what
really matters.
Each time users perform a task, they should be given a “success” or “failure”
score. Typically, these scores are in the form of 1's (for success) and 0's (for fail-
ure). (Analysis is easier if you assign a numeric score rather than a text value of
“success” or “failure.”) By using a numeric score, you can easily calculate the per-
cent correct as well as other statistics you might need. Simply calculate the aver-
Search WWH ::




Custom Search