What Paper is (and Isn't) Good For - Paper Prototyping: The Fast and Easy Way to Define and Refine User Interfaces - page 214

What Usability Testing Won't Find (Real-Life Situations)
Usability testing, whether of paper prototypes or real products, has blind spots of its own. So far, this
chapter has mostly been about the method of paper prototyping and what problems it will and won't
find. A similar analysis could be done for other methods of prototyping, but we're still talking about how
well the methods do at eliciting information in the context of a usability test. Although a full discussion
of this topic is beyond the scope of this topic, I'd like to touch on some of the things that are difficult to
discover in a usability test.
Long-term use. Once people learn to use it, how well can they remember how to do things? How
efficiently can they get things done? How easily can they focus on the task and forget about the
interaction with the system? What can be streamlined? What "little" things become huge
annoyances when users encounter them every day?
Integration and compatibility. How does this interface fit with others that the target market is
using? Does it "play nice" with the operating system and other hardware and applications? Does it
have inconsistencies with those things that bother users?
Real-life context and needs. Do users' goals and tasks really correspond to those things we
asked them to do in usability testing? How often do users get interrupted in the middle of using the
interface or have to leave it to go look something up? How much work do they lose as a result? Do
they use all the functionality? Is the product really making users' lives better in the way we
envisioned? Especially if the users' goals are complex or take a long time to accomplish (for
example, using chemical modeling software to create a new drug), you aren't going to get the
whole picture in a 2-hour usability test.
Showstoppers can lurk beneath the surface of any of these questions, but you aren't going to find them
in a lab. When I claim that paper prototyping can find many important issues, sometimes people
respond "But not all of them." And that's true. But it's true of any method. Although paper prototyping
and usability testing are marvelously useful techniques, they are a complement, never a substitute, for
other methods of understanding users and their needs. (See the References section for more
information.)