Database Reference
In-Depth Information
testing, so it's good to be aware of how they're affecting your data. (Caveat: Self-awareness can be
difficult, both cognitively and emotionally, so think of it as a process rather than an event.)
A usability test is a social setting, and that can influence users' behavior, as illustrated by the following
From the Field box.
From the Field: Usability Testing in a Social Setting
"I was running people through a study on finding information on a Palm PDA. The tasks involved
reading a question and looking through the text until you found the answer. One of the answers
was located in the first sentence of the file. (Q: What was earnings per share? Text: Earnings per
share from continuing operations were $3.05 compared to ... ) Most of the people found that
number right away and made comments like 'Maybe ... but I better scroll down just to make sure.'
The times for this question were way out of line with the other tasks. In a real-world setting, the
person probably would have grabbed the answer and gone with it right away. But they wanted to
make sure they answered the question right. Their goal was not 'find earnings per share' but
rather 'answer this question correctly for the nice experimenter,' a distinction that makes a big
difference in how to interpret results."
Loel Kim and Michael Albers, the University of Memphis
(From "Web Design Issues When Searching for Information in a Small Screen Display."
SIGDOC'2001. © 2001 Association for Computing Machinery, Inc. Reprinted with permission.)
In Michael's case there was some concrete evidence (the task times) that pointed to a false problem. In
a paper prototype test the evidence may not be as clear, but you can still use the qualitative analysis
method that I describe later in this chapter to help you decide whether you've found a valid issue or not.
Bias: Test Methodology
There are many possible ways to conduct a usability test. In the classic "think aloud" method, you ask
the users to articulate what they're doing and thinking. In co-discovery, the users are working together
and talking to each other. You might interact with the users a lot or very little—at one extreme, you
could simply let users work uninterrupted, postponing all discussion until the end of the session. You
might script every word you say or not use a script at all. You might give the users questionnaires to
gather additional information. You might ask the users to review the videotape of themselves working
and explain the things that confused them.
These are all different—and valid—methods of conducting usability tests. And each method carries its
own risk of bias. For example, if you ask users to think aloud, it may change their behavior if it causes
them to weigh their decisions more carefully. On the other hand, if you let users work uninterrupted and
have them review the video later (a technique known as retrospective analysis), users may not
remember perfectly what was going through their minds at the time or may even invent reasons for their
behavior.
Bias: Data Analysis and Reporting
Human beings filter and prioritize information, and to some extent this is unavoidable. When we have a
set of usability test observations in front of us, we continue the filtering and prioritizing process and may
or may not be aware of the methods we are using to do so. For example, in the
previous chapter
I
described several interaction problems that are hard to find with a paper prototype. Let's say that you
found the "false top" problem particularly interesting because the possibility hadn't occurred to you
before. Next week a friend asks you to review his Web site and you notice it has a false top. It's very
likely that you will mention that problem, and perhaps even assign more importance to it than it
deserves, simply because you recently heard about it and found it interesting. That's a reporting bias.
I suspect that all humans are guilty of this type of bias, and I'm dead certain that usability professionals
are! The abstract of the Catani and Biers paper quoted earlier concluded by saying, "Other noteworthy
results were that: (1) usability professionals (through informal heuristics analysis) and users differed in
Search WWH ::
Custom Search