The Politics of Paper Prototyping - Paper Prototyping: The Fast and Easy Way to Define and Refine User Interfaces - page 221

Bias
Another common concern is whether paper prototyping might introduce usability problems where none
exist. In essence, they're talking about bias. Does paper prototyping bias the results from usability
testing? You betcha ... and so does everything else.
Usability testing is an inherently unnatural (although useful) activity that always contains multiple forms
of bias. The trick is to recognize various forms of bias that may be present and consider their effects on
the data collected from usability testing. Let's first consider several forms of bias that are often present
in usability tests: users, tasks, test setting, facilitator, methodology, data analysis, observers, and bugs.
Last but not least, I'll zero in on paper prototyping.
Bias: Users Who Don't Fit the Profile
If a test participant doesn't fit the profile of the target market, you can get misleading results from any
kind of usability testing. Let's look at two examples: subject matter expertise and familiarity with the
technology.
Subject Matter Expertise
If you test with users who don't know or care enough about the subject matter, they might not reveal
important problems. For example, when I tested a Web site that sells toys, one of the tasks asked
users to buy a beanbag toy for a 2-year-old. Some of the users did not have children (I was testing a
variety of sites in this study), and the users without children found some issues with the site's
navigation and checkout process. But it was a parent who found the showstopper: "It doesn't say
whether beanbags are suitable for children under 3—no way would I buy it without knowing that."
People who have in-depth knowledge of a topic are more likely to realize when functionality or content
are missing, problems that are often difficult for non-subject matter experts to find.
There's a flip side to this problem: Users who lack the necessary background in the subject matter may
uncover "problems" that aren't really problems for the intended audience. Many professions have jargon
and concepts that are incomprehensible to an outsider. For example, when I first became involved in
usability, I went on a crusade to remove the word node from some of the applications at my company,
on the theory that it was an overly geeky term. Well, I was wrong; the users, who were engineers, had
no problem at all with the term node and would have been confused if I'd succeeded in getting it
changed to something else.
Familiarity with Technology
If you are developing an interface for people who work outside of high-tech, it's very risky to usability
test with tech-savvy co-workers or friends because they know too much about the ins and outs of the
type of platform you're developing for. I've seen on many occasions that users who know something
about Web site design can readily use things like rollover menus and multiple browser windows that
often stump a less technical audience. If your test population has more technical knowledge than your
typical user, you're likely to mask out a technology-related subset of problems. On the other hand, if
you're developing a product for network administrators, you don't have to worry about whether they can
use a mouse.
I once received a contemptuous email from someone who completely disbelieved an article I had
written called "Seven Tricks That Web Users Don't Know" (available at
www.ibm.com/developerworks/usability/library-us-tricks ). He was a system administrator in
Silicon Valley. He contended that everyone he knew, including his 10-year-old daughter, was
smarter than the people I'd tested with and concluded that I should "get out more" into the real
world. In my reply, I explained to him that all the usability tests I conduct are with strangers who
are recruited by an outside agency in accordance with the user profile determined by the client. I
refrained from touching the point about which of us should get out more.
As discussed in Chapter 5 , deciding on the profile of test participants is a necessary prerequisite for
usability testing. Conducting a test with a user who doesn't fit this profile carries a significant risk of