Methodology III: Empirical Evaluation - Foundations for Designing User-Centered Systems - page 359

13.4 What to Evaluate?

In general, the sooner evaluation is done during development, the sooner you will
get feedback, and the more likely it is that the delivered product will be both
usable and acceptable to users. There is a trade-off here between the need for
evaluation and how closely related the current version is to the final version. If an
iterative development life cycle approach is used, this means that evaluation
should be carried out as part of each iteration. Obviously, the earlier in develop-
ment that evaluation takes place, the less developed the system that is being
evaluated will be. The way that evaluation is often carried out during development
is using some sort of prototype, until eventually the full system can be evaluated.
The prototype usually starts out as something with low fidelity (possibly just one
or more sketches), and increases in fidelity as the project progresses.

13.4.1 Pencil and Paper Prototypes

At the earliest stages of development, pencil and paper mockups of the interface
can be shown to the users, who are then asked how they would carry out particular
tasks. This technique, which is often described as storyboarding should ideally be
carried out with real users, although other designers, and even hostile users, could
be employed.
Using pencil and paper sketches is cheap, and can be used very early in design.
At this stage, ideas are usually still being explored, so the evaluations are usually
formative. The data collected are normally qualitative rather than quantitative,
with the results being used to inform the design of the artifact.

13.4.2 Computer-Based Prototypes

The next level of sophistication involves building a computer-based prototype.
There are now several tools around that can be used for prototyping at several
levels, from pencil and paper style sketches to interactive working prototypes.
At the earliest stages of computer-based prototyping you can employ Wizard of Oz
techniques, where the user will interact with a prototype (computer-based) interface.
The full functionality of the rest of the system is usually not available at this point, so
a human acts behind the scenes to process user input and actions and provide the
required responses (like the Wizard of Oz did in the film!). This technique has been
used very effectively for evaluating the potential of speech-based systems.
The level of sophistication of the prototype should naturally increase as
development progresses, and the prototype becomes closer to the finished product.
How it develops will depend mostly on which basic method of prototyping you
use:

evolutionary or revolutionary. In evolutionary prototyping, the original