Database Reference
In-Depth Information
Success Rates and Statistics
You may have noticed that so far I've said little about calculating success rates, error rates, and the
like. Although I'm not opposed to metrics and statistics, I have not found them to be especially useful in
analyzing the results of paper prototype tests. For example, it's hard to understand the meaning of the
statement "Five out of seven users completed task 3 successfully" if the interface was different each
time.
But sometimes simple statistics can be helpful in determining the importance of an issue or how to
solve it. For example, I might be curious to know how many users looked for a link to the return policy
on the shopping cart page, so I might go through my notes and count how many times it happened.
Although it's often difficult to know ahead of time what issues you'll want to count (the most interesting
ones tend to be the ones you didn't anticipate), if your notes are sufficiently detailed you may be able to
extract some useful information. However, I have done this less often in paper prototype studies than
when I'm testing a working version of the interface.
Action Plan
I don't believe it's an overstatement to say that you'll never fix all the problems you find from a usability
study. It's just not practical to make an interface work perfectly for everyone, all the time; eventually you
reach a point of diminishing returns. Your project also has constraints on the time and people resources
that can be devoted to making changes.
I'm assuming that you already have some kind of process for managing your product development. I
don't want to turn this into a topic on project management, but here are a few thoughts to keep in mind
as you discuss your action plan with your fellow team members.
Consider practicality.
One drawback of paper prototyping is that you can sometimes fix an issue
in the prototype in a way that's not practical to implement. It's worth discussing the successful
changes to the interface to ensure they are doable. If not, since usability testing gave you a better
understanding of the problem, you'll at least be in a position to come up with an alternative idea.
Funnel usability issues into your regular process.
As a rule, I don't believe there's much
benefit in setting up a separate system for managing usability issues if the people who fix them
also do other development activities. Usability issues should be managed the same way as the rest
of the work for the project. For example, if you've got the equivalent of a bug database (one
company I worked for preferred the more delicate term
incident),
you'll probably want to funnel the
unsolved usability issues into it.
Don't start tracking too early.
Because paper prototyping is a fast-moving, iterative activity, it can
be premature to record every issue in a database—some problems will be fixed before you've even
entered them. One company I know of doesn't bother trying to capture the issues until the design
has shown signs of stabilizing, at which point they begin entering usability issues in their bug-
tracking database.
Divide issues into "now" and "later" piles.
If your next release is quickly approaching, you may
have to draw firm lines. One useful question is, "Which of these issues are important to the
success of the next release?" Anything that doesn't pass that test goes into the bug database at a
lower priority.
Consider allocating time for usability issues.
One valid concern about usability testing is that
it's hard to know ahead of time how long it will take to fix the issues you find. One way to manage
this is to set aside a predetermined amount of time to focus on the issues from usability
testing—give each person a prioritized list and have them fix as many as they can in a certain
amount of time. The interface may still go out the door with known usability problems, but at least
you've given it your best shot. (Caveat: This approach only works for non-critical issues, not
showstoppers such as "No one can successfully install the product.")
Search WWH ::
Custom Search