Methodology III: Empirical Evaluation - Foundations for Designing User-Centered Systems - page 365

the behavior, and there are less situational pressures on the behavior (Hock 2002 ,
pp 281-288). Questionnaires and surveys allow you to gather large amounts of data
in a relatively short period of time, as long as you distribute them appropriately.
Designing questionnaires and surveys is an art in itself, as great care needs to be
exercised to make sure that any potential biases are avoided. It is also important to
make sure that the questionnaires are well structured and tested, as this helps to
ensure the validity of the resulting data. For this reason, it is almost invariably a
good idea to carry out a pilot study on a small sample of users, and then refine the
questionnaires appropriately. Having a pilot study is also very useful for deter-
mining how long it will take to complete the survey. As a rule of thumb, most
people are relatively happy with filling in surveys that take 10-15 min to com-
plete, without any reward.
The questions need to be carefully designed, because you will not have a chance
to explain them to respondents. So they need to be clear, unambiguous, and easy to
understand. It is also important that you do not ask leading questions that reflect
any biases that you may have. You also need to think about the answers that you
require. In some cases it may be a simple ''Yes/No/Don't Know,'' or it may be
''select one (or more) options'' from a possible list. In other cases (and quite often
in usability surveys) you will be trying to gauge people's opinions about some-
thing, in which case you are more likely to use rating scales, such as a five-point
Likert scale, where you will ask respondents how much they agree with a particular
statement, such as ''I found it easy to locate the home page button.'' In this case the
response scale would normally be from Strongly Disagree to Strongly Agree.
Distribution of questionnaires and surveys requires careful thought. Usability
surveys are frequently handed out on paper to participants as part of a usability
study (often at the end). There are some standard usability rating scales that you
could use or adapt for your own purposes, such as the System Usability Scale
(SUS, Brooke 1996 ). More generally, however, you may want to use electronic
surveys, in which case you need to think about how you will attract people from
your target audience to complete the survey.
Note that if you intend to use follow-up surveys at the end of a test, you need to
be aware of what is called the media equation (Reeves and Nass 1996 ). This refers
to the fact that if you give people the survey on the same machine as the one on
which you give them the test, they rate things more highly than if they complete
the survey on a different machine! They treat the machine they used as an agent
that needs to be treated socially.
13.5.7 Interviews and Focus Groups
Interviews can take three different forms: structured, unstructured, and semi-
structured. Whichever type you decide to use, it is often a good idea to record
them, with the written consent of the interviewees. It is also a good idea to make
some written notes. These will help add extra context to help interpret the content