Preparing the Prototype - Paper Prototyping: The Fast and Easy Way to Define and Refine User Interfaces - page 130

Pilot Tests
Unlike a usability test rehearsal, a pilot test is not something you'll do for every usability study, although
there are situations in which one can be useful. A pilot test is a cross between an internal walkthrough
and a usability test. Its main purpose is to refine the tasks and methodology used in the main tests (as
opposed to a usability test, which is used to refine the interface). Unlike in a walkthrough, in a pilot test
you bring in a representative user. Unlike in a usability test, you ask this user to give you feedback on
the tasks and instructions, not just to do them. You can also time the pilot test to get a better idea of
how long it will take users to accomplish the tasks. In a pilot test, it's appropriate to interrupt the user
(politely, of course) to have a discussion with your teammates about how you want to change the tasks
or methodology—this is not something you want to spend time doing in a usability test.
A pilot test is useful when you don't want to change the tasks during the usability study. For example, I
tested a live Web site in the United States that was also being tested in Denmark. My Danish
counterpart created the tasks, and I conducted the pilot test. I then reported back to her my results
such as, " Task 2 will take more time than we thought, so we should allow up to 40 minutes. But task 3
and task 5 cover similar ground, so let's eliminate task 5." The pilot test helped us coordinate our
methods so that our test findings would be comparable.
I reserve pilot tests for situations in which there is a high degree of rigor needed in the testing
methodology, such as a competitive evaluation of two released products. Because a paper prototype is
continually evolving, there is less need to hold the methodology constant, so I don't bother with pilot
tests. However, there is no reason not to conduct a pilot test if you feel the need for one. In particular, a
pilot test can be a good opportunity for a new facilitator to practice under the guidance of a more
experienced one. Speaking of new facilitators, the next chapter explains the basics of usability test
facilitation.