Database Reference
In-Depth Information
bugs, the effect can range from benign to completely undermining a conclusion.
And Yes, the Paper Prototype Introduces Bias Too
Last but not least, let's look at how a paper prototype may change what happens in a usability test. It's
possible that paper prototypes may:
Cause false problems.
1.
2.
Slow down the action, possibly inhibiting users from exploring the interface.
3.
Affect users' impressions of the interface.
False Problems
Sometimes the paper prototype can introduce some confusion into a usability test if the users have to
decipher the nature of what they're looking at. I once conducted a test of a prototype that had an
expandable list where items representing different types of devices were indented in a three-level
hierarchy. We had used removable tape for each line item, but due to sloppy placement it wasn't
always clear what lined up with what. And each line item had a hand-drawn icon to represent the device
type, but the icons looked different depending on who had done them. In testing, we saw users puzzle
over the structure that the paper prototype was trying to show them. In this case, because both the
alignment and icons were part of the data, it's likely that the paper prototype did introduce some degree
of confusion.
Another example comes from the Pingtel case study described in Chapter 2 . Hal Shubin explains: "On
the paper prototype, test participants tried pressing button labels instead of the buttons. That happened
in the 'soft phone' [emulator software] too. In the 3D version of the phone, there are cues not present in
the paper or software versions; the screen is recessed and the buttons stand out." In both these cases,
the development teams decided to downplay or dismiss these particular "problems" found in the paper
prototype.
Sometimes it's unclear whether it's the medium of paper or the human Computer that's responsible for
false problems. In an experiment, Uceta, Dixon, and Resnick (1998) created a hand-drawn paper
prototype of an interface for ordering fast food. They scanned their sketches and made a linked
PowerPoint presentation—in other words, a prototype that still looked hand-drawn but could be tested
on a computer. In testing they found a few more problems with the paper version, but eventually
decided the extra problems were false ones. (As you read the following explanation from their paper,
note that the person they refer to as the facilitator was also the Computer.)
"The results of the analysis indicated that the paper prototype found 55% of the usability
problems with the interface while the computer based prototype found 35% of the usability
problems. Given that both interfaces were exactly the same, (besides the medium of
presentation), the results were surprising. However, upon further analysis, the discrepancy was
attributed to the presentation environment. We suspect that the users were more frustrated at
the level of fidelity and the lack of feedback with the paper-based prototype. This became evident
after careful review of the videotapes, which showed that between the five participants for the
paper prototype test, there were 45 separate instances where the user was distracted in
some way by the facilitator [emphasis added]. It is ironic that this method can introduce non-
existing usability problems. It seems the facilitator's presence can also create a highly artificial
environment in which the user behavior can be manipulated (albeit unintentionally to the benefit
or deficit of the design). Taking this condition into consideration, statistical analysis indicated that
no differences were found between the sensitivity of both methods for finding usability problems."
Slower Action, Inhibiting Exploratory Behavior
Bob Virzi explained to me that he noticed some evidence of a behavioral effect in the data from the
study he conducted with Sokolov and Karis. On one task that was particularly difficult, some users of
the high-fidelity prototype started "thrashing"—rapidly and repeatedly taking various incorrect actions.
This behavior was not in evidence with the paper prototype, where users were more deliberate. I have
witnessed this effect myself on occasion, where users have said that they normally try lots of things
Search WWH ::




Custom Search