Methodology III: Empirical Evaluation - Foundations for Designing User-Centered Systems - page 346

involves some sort of acceptance test with the customer to make sure that the
requirements have been met. You should make sure that you have the distinction
between these two clear in your own head.
In this chapter we provide an introduction to the topic of evaluation and the
methods that can be used to collect data. For a much fuller discussion about which
analytical methods to use, you should consult an appropriate textbook such as
Lazar et al.'s ( 2010 ) book Research methods in human-computer interaction.We
do not go into the details of the methods you need to use to analyze your data. The
way that you analyze the data is important, but is highly dependent on the type of
data that you collect, and the overall purpose of the evaluation. More information
about how to analyze your data can be found in statistics books, such as Howell
( 2010 ) or, for qualitative data Miles et al. ( 2013 ) and Todd ( 2004 ).
13.1.1 Why Do We Need User Testing?
If we could always guarantee that systems would be built correctly, based on a
complete and accurate analysis of user requirements, then there would not (the-
oretically, at least) be a need for user testing. There are several reasons why we
cannot give such guarantees, however. Perhaps the most pertinent one here is what
has been called the envisioned world problem (Carroll and Rosson 1992 ; Woods
and Dekker 2000 ), which is based on the fact that the world will inevitably change
between the time when the user requirements are analyzed and the time when the
final system is delivered. Often the changes are in ways that cannot be (fully)
controlled by the users. So, when the requirements analysis is carried out, the users
are effectively being asked to define what the system will need to do to work
acceptably in some future context which they cannot fully predict or envision.
There are still some system developers who believe that as long as a system
does what they consider to be the right thing, then that is enough. Their attitude is
that if the users cannot make the system work, it is due to the users' lack of
knowledge or ability, rather than being due to some fault of the developers. The
fundamental problem with this view is that if the delivered system does not fit with
the way that users normally carry out their work, there is a risk that it may not be
accepted (e.g., see Berg 1997 ). The acid test of acceptability often comes when the
users have to try to use the new system in their normal working environment.
The purpose of user testing is not to understand users but to evaluate how
particular users can carry out particular tasks (using your system) in a particular
context. There are still some designers who mistakenly believe that they know
exactly how users carry out their work. In practice, however, when you observe
users you see that they are often very creative, and use systems in ways that were
never conceived of by their designers. The use of spreadsheets to format text or to
make art, the use of thumbs to press cell phone buttons, and the use of social media
broadcast tools like Twitter to organize political activities are just a few examples
of creative use that were never imagined by their designers.