Experimental Setups for User Evaluation of Mobile Devices and Ubiquitous Systems - Human-Computer Interaction and Innovation in Handheld, Mobile and Wearable Technologies - page 157

direct interaction with the user; the tasks directly
related to the evaluated devices; and the context,
which is defined as a complementary set of the
three preceding elements, i.e., other users, other
devices, and tasks with no direct link with the
evaluation.
Usually, it is straightforward to determine
the elements of the first three categories because
they are in direct relation with the objectives of
the evaluation. On the contrary, for the last cat-
egory, making the distinction between relevant
and non-relevant elements of the context can be
difficult. Indeed, by definition, mobile devices
and ubiquitous systems form a unit with their
environment. Elements of the context have a
direct influence on the interaction, whereas the
others contribute to create a general atmosphere.
For instance, bad weather conditions in a ski re-
sort or a crowded museum can have a significant
impact on interaction, and so, are relevant. On the
contrary, traffic jams at a ski resort entrance or
bad weather condition around a museum building
have little impact on the interaction.
The literature distinguishes two main ap-
proaches to set up a realistic interactive environ-
ment (Hagen, Robertson, Kan, & Sadler, 2005). A
first approach -usability laboratory- simulates the
aspects of the environment with varying degrees
of realism. A second approach -in-situ- places
users in the real world, where all the aspects of
the environment are naturally present. Overall,
it may seem “obvious” that the in-situ approach
have greater validity than the laboratory approach,
because all the expected aspects of the environment
must be present. We will see that the literature is
not unanimous on this point.
Laboratory. A first approach consists in simu-
lating the elements of the interactive environment,
with more or less realism, in a usability laboratory.
This traditional approach is based on well-known
methodologies.
The user mobility is the crucial element that
researchers seek to simulate. Various simulation
techniques were used: the user could walk around

buildings (Brewster & Walker, 2000), inside cor-
ridors (Pirhonen, Brewster, & Holguin, 2002) or
on a course in the usability laboratory (Kjeldskov
& Stage, 2004). Mini-steppers (Pirhonen, et al.,
2002) or treadmills (Kjeldskov & Stage, 2004)
could also be used to simulate the walk. In ad-
dition, others researchers tried to recreate the
“general atmosphere” of the environment with
real furniture in a usability laboratory divided into
several rooms, for instance to simulate hospital
rooms (Kjeldskov, et al., 2004).
These simulation techniques are well adjusted
for the simulation of short-duration experiments
for small-size environments (for instance: offices,
shops, apartments). However they usually require
a large usability laboratory and can be quite expen-
sive. They are not well adapted to simulations of
long-duration experiments in large-size environ-
ments (for instance: ski resorts, museums). And,
more generally speaking, a high level of simulation
fidelity is difficult to achieve if the system under
test is designed for large or complex environments.
Moreover, even if the simulation is very realistic,
the user always perceives it as a simulation, and
should not behave as usual. Furthermore, a lot of
real-world elements, for instance adverse weather
conditions, are missing. As a consequence, some
usability issues, for instance a non-waterproof
touch-screen, are hidden.
In-situ. In contrast, a second approach consists
of replacing simulated elements by real world
elements (i.e., experimenting in-situ). A priori,
more reliable results must be obtained, since the
experimental setup is supposed to be as close
to the future usage environment as it can be.
Regrettably, in-situ experiments are known to
be complex to set-up, and suffer from important
variability of the experimental conditions due to
the natural variability of the real world (Kellar,
et al., 2005). Maybe as a consequence of it, they
are used very infrequently in the field of mobile
usability research (Kjeldskov & Graham, 2003).
The literature uses several different terms to
name in-situ experiments. The non-exhaustive list