Database Reference
In-Depth Information
analysts who conducted the cognitive walkthrough. Novick did note some of the deficiencies of the
paper prototype approach—its timing wasn't realistic, it didn't show color accurately, and it probably
wouldn't scale up to an entire airplane cockpit—but concluded that the overall approach was practical.
Strictly speaking, these studies don't "prove" that a paper prototype will find the same set of problems
as a more realistic one. Rather, they indicate that the sets of problems overlap to a considerable
degree and there don't appear to be important differences in the problems that are found with one
method versus another. However, these studies don't really scrutinize whether paper prototyping is less
useful in finding certain kinds of problems, as I described in the
previous chapter
(although Novick
seems to have theories similar to mine). I believe that such effects do exist, but research studies to
date have not been designed to reveal them. As with any field of study, subsequent research should be
able to fine-tune the questions under investigation and yield more detailed insights into exactly what
paper prototyping is and isn't good for. As it stands, the evidence suggests that paper prototyping is as
valid a technique for uncovering problems in an interface as usability testing the real thing.
Case Studies
The problem with finding real-life examples of paper prototyping's validity is that there's usually no point
of comparison—because companies are busy trying to get products out the door, they don't have the
luxury of conducting usability studies with a paper prototype and comparing the results to those
obtained from testing the real interface. The From the Field box by M. David Orr (on p. 291) provides
an interesting anecdotal example of someone who found himself in a position to do such a comparison.
Here's a slightly more scientific example:
Chapter 4
described a study where researchers
Säde,
Nieminen, and Riihiaho (1998)
tested a 3D mock-up of a can-recycling machine. Later in the study they
tested a functional prototype with a larger number of users (50 instead of 10) and found no statistically
significant differences in the severity of problems encountered in using the interface—the mock-up had
accurately predicted that most users would have little or no difficulty with the "automatic" version of
interface. (Note that the functional prototype had some improvements over the 3D mock-up, so this is
not entirely an apples-to-apples comparison, but at least they didn't find any worms in their apple.)
This is just one example—as shown in the References section, there are many papers and articles
about real-world experiences with paper prototypes. In most cases there is no point of comparison
because paper prototyping was the only method used, but I have yet to find an article that concludes,
"We tried paper prototyping, and it was a waste of time."
Summary: Does Paper Prototyping Find Real Problems?
As you can see, the evidence of paper prototyping's validity ranges from scientific to anecdotal, and I'll
add my own experience to the latter. Over a 10-year period, I have conducted several hundred usability
tests of software and Web sites, at least 100 using paper prototypes. At the end of each project, I meet
with the development team to categorize and prioritize the problems, then I write a report. When I go
back and reread those reports, I don't see any systematic differences in the type of findings. (A better
test would be to see if someone else could find a difference.) To a large extent, I believe that I have
found the same kinds of problems with paper prototyping as I've found when testing the real thing.
The real question, however, is what does it take to convince
you
(or your co-workers)? Some people
don't care what research says; they need to try a method and see for themselves. Other people may
need far more detail about the research than I've presented in this chapter, including a scrutiny of the
methods used to conduct it. Still others may make up their minds about paper prototyping based on the
opinions of people they trust. Whenever questions of paper prototyping's validity arise, the best way to
answer them depends on what type of person you're dealing with; I've provided you several different
approaches but you may only need to use one of them.
From the Field: Comparing Paper Prototypes and Reality—An Informal Case Study
"I'm a tech writer, instructional designer, and usability specialist who works for a consulting firm.
My company had been contracted to design and develop a software system for customizing
documentation. Our client manufactured 2-way radios. Each customer purchased only a subset
of the radio's features, so the manufacturer needed a way to generate documentation based on
Search WWH ::
Custom Search