Introduction to Usablity Test Facilitation - Paper Prototyping: The Fast and Easy Way to Define and Refine User Interfaces - page 141

effects on the data being collected. People who are unfamiliar with usability test facilitation may
mistakenly think that this is the facilitator's primary purpose, but in paper prototype usability tests, the
scientist role is usually the third priority—there are many circumstances where the scientist takes a
back seat to the other roles.
I don't want to imply that being scientific isn't important; professionals in any field should always
understand their methods, apply them appropriately, and seek ways to improve them. But keep in mind
that paper prototyping is not a technique that's used when precise usability measurements are required.
It's more of a blunt instrument. Most of the time, there isn't a need to calculate statistics. You don't use
an eye-tracker or spend hours analyzing videotapes. Most of the data from a paper prototype usability
test is qualitative, not quantitative.
Is objectivity important? Yes and no. Clearly, the facilitator has the opportunity to introduce all kinds of
bias into the testing process, through something as obvious and deliberate as a hint or as subtle and
unintentional as a smile when the user makes a correct choice. It's virtually impossible to remove the
human element from paper prototype testing. Thus, the scientist's focus isn't on minimizing interaction
with the users but rather on monitoring how it might affect the results. For example, if the users are
given a hint, the scientist makes note of it so that the team can discuss later how the users' behavior
might have been affected. For example, "They found the answer in the help, but only after a suggestion
that they look there. This is evidence that the help content works, but we can't conclude whether users
in real life will see it." However, it is usually best for the facilitator to avoid asking leading questions or
revealing his or her own opinions of the interface, so the scientist strives to maintain this objectivity.
Chapter 13 revisits the topic of bias in more detail.
For beginning facilitators, it is often hard enough to fulfill the flight attendant and sportscaster
responsibilities without adding note taking to the mix. I recommend that new facilitators skip the note-
taking until they're comfortable managing the action. Instead, invite plenty of observers and have them
take notes (which they should be doing anyway). I also know some facilitators who feel, even after
several years of experience, that they do their best work when they leave the note-taking to others.
[ 1 ] A friend once asked me point-blank after helping me with a pilot test, "Companies pay you to do that?
"
[ 2 ] Just don't get too carried away. I once commented that a user was scratching his head, whereupon
he fixed me with an unreadable expression and replied, "Yes, it itches."