Experimental Setups for User Evaluation of Mobile Devices and Ubiquitous Systems - Human-Computer Interaction and Innovation in Handheld, Mobile and Wearable Technologies - page 160

tion of the experiments is usually composed of a
camcorder, and sometimes, a gooseneck camera,
fastened to the device, is also used to capture the
interactions between the user and the device.
From a methodological perspective, the in-situ
experimental setup is similar to the laboratory
one. The in-situ protocol can be seen as a varia-
tion of the laboratory protocol, because the user,
the device, the facilitator, the observers and the
instrumentation are simply “moved” from the
artificial context of the laboratory, to the real
context in the field. In other words, the stage
changed, but the actors, the decors and the script
are identical. So users acts in a context that is far
from reality. Moreover, he/she may suffer, even
without realizing it, from some influence of the
presence of the observers.
Finally, we hypothesize that, in-situ, the user
acts in a “bubble” where his/her tasks, general
activity, social relationships, etc. are constrained
by the experimental protocol. We assume that
the expected benefits of the in-situ are partly an-
nihilated by these constraints, and therefore, the
detected differences between the two configura-
tions are partly artifacts. In other words, we hy-
pothesize that the location of the experiment for the
mobile/ubiquitous system is not the key element
that makes the difference between laboratory and
in-situ experimental setups. We assume that the
people (e.g., facilitator, cameraman) as well as the
instrumentation devices (e.g., gooseneck camera,
wireless microphone) have an important effect on
the user behavior because they alter the realism
of the environment. Moreover, the level of the
instructions given to the user -tasks prescribed or
not- change significantly the experimental setups.
Our assumptions are consistent with Thomas
and Kelloggs ones about the minimization of the
“ecological gap” [Thomas, 1989 #120]. At last,
we hypothesize that there are more than one in-
situ experimental setup.
In-situ with or without ecological context.
We propose to split the in-situ experimental
setups into two opposite setups. The first setup

is similar to laboratory experiment. The only sig-
nificant difference is the location: the laboratory
is replaced by the real world, but the people and
the instrumentation are still present. The second
one tends to place the user as close as possible to
reality, in other words in an ecological context.
This experimental setup is characterized by the
absence of visible elements related to the obser-
vation -people and instrumentation- in order to
preserve the self-determinism of the user.
More formally, compared to the laboratory
experimental setup, the realistic ecological context
experiment takes place in-situ if (1) the context is
real and (2) the device and the data are perceived
as real by the user (however the Wizard of Oz
technique could be used). The ecological context
is preserved if, moreover, (3) the user is free to
do what he/she wishes -no prescribed tasks- and
(4) all observation elements -people and instru-
mentation- are invisible. We now name this latter
experimental setup “in-vivo”.
Very few in-vivo experiments are reported
in the literature. The Demumieux and Losquin
publication that focuses on usage statistics of
mobile phones (Demumieux & Losquin, 2005)
or the more recent work of Jensen and Larsen
(Jensen & Larsen, 2007) are the rare exceptions.
This quasi-absence of in-vivo experiments can
be explained by the technical difficulties that
must be overcome to set up these experiments.
For instance, the observation is difficult because
video-recording systems cannot easily be placed
in the context. Moreover, the added value of
these experiments is difficult to estimate, even if
instinctively one could think that the ecological
context must increase the validity of the results.
Uncertainty Principle. In-vivo experiments
face another critical issue. It is quite impossible
to know whether users act in a different way in
ecological context or not, because classic observa-
tion techniques, for instance the video recording
technique, alter the ecological context. Alterna-
tives to the video recording technique, for instance
diaries or critical incidents technique, cannot be