Methodology III: Empirical Evaluation - Foundations for Designing User-Centered Systems - page 352

• Maturation of the participants: if their condition changes over the duration of the
study, e.g., they become more tired.
• The types of participants: it is often impossible to use randomly selected par-
ticipants, so you need to make sure that you do not end up with unbalanced
groups of participants (e.g., all males, or all people aged over 50 in one of the
groups).
• Testing effects: if you give participants in the study the same test at two different
times, they may find it easier the second time because they already know the
questions.
These potential confounding effects are well known. In many cases there are
well documented solutions too which can be found in books on experimental
psychology methods such as those by Calfe ( 1985 ), Ray ( 2008 ), and Campbell and
Stanley ( 1963 ).
External validity relates to how far the results of the study can be generalized to
other populations (such as different user groups), other places, and other times.
One of the main factors that needs to be considered is the choice of participants
used in the study. If all your participants are university students, for example, how
can you be sure that the findings will apply to people who are aged over 60? To get
good external validity you need to be aware of other effects that could have an
impact on the results that you obtain, and provide some way of alleviating them.
These effects include:
• Awareness of anticipated results: if participants can guess what they think the
outcome of the study should be, they may adapt their behavior to what they
think you expect them to do.
• The Hawthorne effect: people's performance can change simply as a function of
being watched or recorded.
• Order effects: if you test people on artifact A first and then artifact B, there may
be some carryover effect which means that their results with artifact B are better
than they would otherwise have been.
• Treatment interaction effects: it may be the case that the participants in your
study are motivated differently, and the way they are allocated to groups for
testing means that you have one group that is more highly motivated than the
other, which hence performs at a higher level.
Again, these potential confounding effects and solutions to the problems caused
by them can be found in the literature mentioned above.
One way to increase external validity is to make sure that you use a wide range
of users, stimuli, and contexts. The downside is that this will increase costs (time
and money) and make it more difficult to detect real differences. Care is also
needed to make sure that you do not reduce reliability (which is discussed below).
Ecological validity refers to the extent to which your results can be applied to
real world settings. You should be able to see that it is closely related to external
validity. For an evaluation study to have high ecological validity, the methods,
materials, and setting of the study must approximate the real-life situation that is