Observers - Paper Prototyping: The Fast and Easy Way to Define and Refine User Interfaces - page 169

Figure 10.1: Answers to the survey question, "What is the maximum number of observers that you
feel are appropriate to have in the same room with the user(s) during paper prototype tests?"
Learning from Disaster
On weekends I teach beginners how to ride motorcycles according to a curriculum developed by the
Motorcycle Safety Foundation (MSF). The MSF course draws heavily from the findings of a 1981
research study of motorcycle accidents in California. [ 1 ] For every significant accident cause, the course
has corresponding material aimed at reducing that risk. For example, the study found that in two thirds
of multiple-vehicle accidents, the driver of the other vehicle violated the motorcyclist's right of way.
Thus, the course discusses several ways for a motorcyclist to be more visible to other drivers.
Although I haven't done a scientific study of in-room observers, I try to copy the MSF
approach—whenever I hear about a "crash" (unpleasant incident) that was caused by an observer, I
ask myself whether my methods (the observer rules discussed later) are theoretically able to prevent it.
In doing research for this topic, I deliberately set out to collect the worst examples I could find about in-
room observers, and the three on pp. 227-228 are typical.
From the Field: In-Room Observer War Stories
Anecdote
"I remember one unfortunate incident. One of the observers was a technical documentation
person who had written the installation manual for a product whose installation we were testing.
He got very frustrated that the users were having trouble, but were not using the manual. At one
point he said loudly and angrily, 'But they're not even using the manual!' The users felt bad and
started using the manual (obviously a departure from their natural behavior)."
Betsy Comstock, Polycom
"A couple of years ago, one of the stakeholders insisted on being in the room while the user
testing took place. The test team didn't think it was a good idea, and we did caution the person
not to interrupt what was going on and not to react to anything that happened. During one of the
tests, a user hesitated and then chose an option that we hadn't anticipated. The facilitator was
waiting to see what happened next. The stakeholder got up, strode across the room and
informed the user, That's not how it's done— here's how it's done' and proceeded to demonstrate
the interaction as we had designed it."
Professor Rosalee Wolfe,