Experimental Setups for User Evaluation of Mobile Devices and Ubiquitous Systems - Human-Computer Interaction and Innovation in Handheld, Mobile and Wearable Technologies - page 165

Figure 3. Postcard with a hidden RFID tag (on the left) and one pair of antennas of the RFID based
tracking system (on the right)

by the visitors. We determined that about half of
the visitors carried out a complete visit (they use
both information kiosks) as it were envisaged by
the scenario writer. The lack of video records
made the interpretation of abnormal visitors tracks
difficult. In most cases, the comments of staff, in
particular the guards, allowed us to discover the
reasons of the abnormal behaviors detected. This
problem seems to be generic with experiment
in-situ in ecological context. The results could be
consulted in French in (Jambon, Mandran, &
Perrot, 2007).
Synthesis. From a technical standpoint, the
experiments in the field are more difficult to set
up than laboratory experiments. Experimenters
must be aware that “out of the lab,” the expression
“it's the jungle out there” (Kellar, et al., 2005) is
right. The situation is worse in ecological contexts
because there is no facilitator to solve minor pro-
totype bugs, thus the device must be reliable. In
our opinion, there is no major difference between
laboratory experiments and in-situ experiments,

because the “protection bubble” isolates the user
from a part of the variability of the real world.
However, these experiments are interesting if the
context is difficult to simulate.
From a methodological standpoint, we make
a strong distinction between in-situ and in-vivo
experiments. The difference is not only a conse-
quence of the absence of observers and instrumen-
tation, but also a consequence of the absence of
instructions. Without prescribed task, test users
use available systems functions as they wish.
Because they use only a limited set of functions,
it is coherent to detect less usability problems in
ecological context. However, the usability prob-
lems detected must be more relevant because they
appear in the real usage context of the system. In
addition, in-vivo experiments are well adapted to
the evaluation of usage. For instance, in the E-
skiing experiment, the most important result was
about the real usage of the system. In the Museum
experiment, reliable statistics about kiosks usage
could be collected.