Information Technology Reference
In-Depth Information
When using this scoring system, it's important to remember that these data
are ordinal (see ChapterĀ 2). Therefore, you should not report an average score.
Rather,presentthedataasfrequenciesforeachlevelofcompletion.Thisscor-
ing system is relatively easy to use, and we usually see agreement on the various
levels by different usability specialists observing the same interactions. Also, you
can aggregate the data into a binary success rate if you need to. Finally, this scor-
ing system is usually easy to explain to your audience. It's also helpful to focus
on the 3's and 4's as part of design improvements; there's usually no need to
worry about the 1's and 2's.
HOW TO ANALYZE AND PRESENT LEVELS OF SUCCESS
In analyzing levels of success, the first thing you should do is create a stacked
bar chart. This will show the percentage of users who fall into each category or
level, including failures. Make sure that the bars add up to 100%. Figure 4.3 is
an example of a common way to present levels of success.
Levels of Success, by Task
100%
90%
80%
70%
60%
50%
Failure/Quit
Some Problem
No Problem
40%
30%
20%
10%
0%
Login
Search
Find Book
Add to Cart
Checkout
Tasks
Figure 4.3 Stacked bar chart showing different levels of success based on task completion.
4.1.3 Issues in Measuring Success
Obviously, an important issue in measuring task success is simply how you
define whether a task was successful. The key is to clearly define beforehand
what criteria are for completing each task successfully . Try to think through the
various situations that might arise for each task and decide whether or not they
constitute success. For example, is a task successful if the user finds the right
answer but reports it in the wrong format? Also, what happens if he reports the
right answer but then restates his answer incorrectly? When unexpected situ-
ations arise during the test, make note of them and try to reach a consensus
among the observers afterward about those cases.
Search WWH ::




Custom Search