Observers - Paper Prototyping: The Fast and Easy Way to Define and Refine User Interfaces - page 168

Concerns about In-Room Observers

Figure 10.1 shows how 169 usability professionals answered a question about in-room observers. As
you can see, most respondents feel that it's acceptable to have a few in-room observers, but not a lot.
As Table 10.1 shows, the concerns I've heard about in-room observers fall into three categories.

Table 10.1: Concerns about In-Room Observe r s

Concern Explanation Discussion

Disruptive
observer
behavior

Observers may not behave
appropriately, interrupting to
help the user, defending the
design, expressing dismay
that the person doesn't "get
it," etc.

This can indeed be a problem, but it's a
solvable one. Observers can be trusted to
behave appropriately, especially when they
are taught about the effects of their behavior
on users. After all, we're talking about fully
functioning adult members of society here,
not a troupe of baboons.

Changing
user
behavior

In a paper prototype test, the
Computer and facilitator are
necessarily present. But
having nonparticipating
observers may cause users
to behave differently than
they normally
would—inhibiting their
willingness to explore,
causing them to be less
forthcoming with criticism,
etc.

It's always good to question how your
methods may affect your results. Effects due
to observers can't be eliminated because
users must be told if the session is
videotaped and/or that people are watching
from elsewhere. In practice I find that properly
briefed users are quite willing to offer
feedback, so I'm not worried about that issue.
Other effects attributable to in-room
observers are certainly possible, but
practically speaking I don't think they would
reverse many of the findings I've seen from
paper prototype usability tests. Chapter 13
talks more about various sources of bias,
including some that I think are more
problematic than in-room observers.

User
stress
(a.k.a.
ethics)

Users may feel
uncomfortable being under
close scrutiny, especially
when (not if, since it almost
always happens) they are
confused about something or
get stuck.

True, and this is the bottom line. Causing the
users more stress than they have consented
to would be at odds with the ethics of usability
testing. You have to make your decision
about in-room observers based on your best
understanding of the benefits/risks, the
personalities of the individuals involved, and
your own comfort level (it's difficult to make
someone feel at ease in a situation where
you're uncomfortable yourself). Ethical
questions are complicated, so talking them
over with colleagues is a good idea.