Observers - Paper Prototyping: The Fast and Easy Way to Define and Refine User Interfaces - page 170

DePaul University
My Comments
This observer hadn't been told that it's possible to gather data about what should go in the manual
without having the users actually use it. I also would have discussed the "game plan" with him to get
his agreement on when/if users would specifically be asked to use the manual.
And Betsy concurs: "I learned to more carefully instruct the observers about the purposes of the
testing, how to behave during a session, and what their expectations should be about the use of the
data."
Here's my favorite line for preventing observers from helping: "Once you start explaining the interface,
you no longer have a usability test but a very expensive training session."
And this observer learned this. As Rosalee reports, "To the stakeholder's credit, this person has since
embraced our efforts after we explained the value of letting a user struggle. Now the person is able to
be in the room and observe without interrupting the test. As a bonus, this person now takes excellent
notes ."
"I often conduct usability studies with observers in the room. One day we were testing a feature
that 'Joe' had put together. The participant was running into all the same problems as the
previous participants. At some point, Joe just couldn't take the fact that people didn't know how to
use his feature at all. He stood up, pointed at the participant, and started yelling, saying that he
was too stupid to use our software or to have the job he was in. I was finally able to get Joe to sit
down and be quiet and work on calming down the participant and building his ego back up. In the
end, everything was fine but it was an eye opener for me on what can happen during a study like
that. (Yes, Joe had been briefed that he wasn't allowed to talk.) The participant is still a customer,
luckily."
Anonymous
This story underlines how stressful usability testing can be for the observers. Part of the facilitator's
responsibility includes not making the observers feel picked on. After seeing the same issues a few
times, I might have asked Joe if we had enough data from that task and then substituted a different
one. Although Joe's frustration is understandable, his means of expressing it is extreme—I've worked
with hundreds of observers and never seen behavior like this. Joe may have had an anger
management problem or otherwise lacked the social skills to relate well to people. [ 2 ] If I knew there
was a "Joe" on the team, I would ask some of the observers sit in another room and tactfully suggest
to Joe that he might find it less frustrating to observe from afar.
Keep in mind that the potential for observers to wreak havoc doesn't magically vanish when you confine
them to an observation room. Colleagues have told me about observers who disrupted usability tests
by laughing loudly (most observation rooms aren't soundproof), barging into the test room to explain
something to the user, or even throwing M&M candies at the one-way mirror. So no matter where the
observers are located, it's important to brief them on appropriate behavior.
Who Shouldn't Be in the Room
It may not be appropriate to have a particular observer present if that person's presence might be
intimidating to users. In one usability study I conducted, the users were subscribers to a professional
magazine and one of the observers was a highly regarded editor of that magazine. The users knew the
editor by reputation, although none had actually met him. We decided to have the editor observe the
tests from the room next door so that his demigod presence wouldn't cow the users.
Similarly, you should tread carefully in any situation in which a user and observer know each other,
especially if there is a difference in power (manager-subordinate) or an ongoing relationship (account
manager-customer). In these situations, unless both parties agree that they're fine with the idea, it's
best to ask the potentially intimidating observer not to attend that test.
[ 1 ] The study's informal name, "The Hurt Report," refers not to its ominous subject matter but to its lead