Experimental Setups for User Evaluation of Mobile Devices and Ubiquitous Systems - Human-Computer Interaction and Innovation in Handheld, Mobile and Wearable Technologies - page 162

be implemented. In general, it is not possible to
provide a believable justification for the record-
ing device in situations where the device may
seem incongruous. For instance, it is be difficult
to justify a camera in hospital rooms.
Moreover, researchers must balance between
the ethical and the efficiency aspects of the Tro-
jan Horse technique. It is imperative to inform
users of what is recorded, even if, this technique
requires to partly hide the real use of the data at
the beginning of the experiment. In all cases, the
users must be informed at the end of the experi-
ment about the Trojan Horse and how data was
collected and used.
Roadmap . We must continue to explore if
in-situ experiments are worth the cost. More
formally, our objective is to determine the added
value, in terms of correctness and completeness,
of in-situ experiments compared to laboratory
experiments. Articles dealing with comparisons
between experiments in laboratory and in-situ
without ecological context already exits in the
literature (see paragraphs about methods). Our
roadmap is to complement these results with the
analysis of experiments in-vivo.
In ecological contexts, the video-observation
of users is not always possible, so the computer
logs collected by the devices are the primary
source of information. These logs are related
with interactions between user and device -user
actions and interface feedbacks- and also with
the more general user activities -geo-localization,
network logs, etc.
We first created a meta-evaluation (MapMobile
experiment) in a quasi-realistic context in order to
compare findings obtained with two different sets
of observation data: video records versus computer
logs. The objective of the meta-evaluation was to
determine to what extent the analysis of computer
logs could replace the analysis of video records.
In other words, this experiment was necessary
to “calibrate” the observation technique we have
been using until now in ecological context.

Then, we tested the Trojan Horse technique
with two experiments in-situ with ecological
context. The first test was the user evaluation of a
mobile service for skiers in a ski resort (E-skiing
experiment). The second one was the user evalu-
ation of a ubiquitous environment that was set
up for a temporary museum exhibition (Museum
experiment).

EXPERIMENTS

MapMobile experiment. This experiment was
a meta-evaluation, so it had two objectives: a
basic objective, the device usability evaluation,
and a meta-objective, the observation techniques
comparison.
The basic objective was to evaluate the usability
of context-aware personal digital assistant that
also gave geo-localized information to the user.
The scenario simulated the indoor guiding of an
executive in a professional context. The experi-
ments took place in a real professional building
during work hours. Twelve users participated
to the experiments. Due to technical failures or
organizational mistakes, the records of two users
were to be removed from the data set. Each experi-
ment lasted about thirty minutes. A facilitator and
cameraman accompanied the user, as shown in
Figure 1. The facilitator was in charge of giving
instructions and asking specific questions during
the scenario execution.
The meta-objective was to compare two dif-
ferent observation techniques. The first technique
was representative of experiments in-situ (without
ecological context). We recorded the video of the
context (camcorder held by a cameraman), the
comments of the user (wireless microphone) and
the PDA screen film (screen grabber tool). The
second technique mimicked observations that
could take place in-vivo (with ecological context).
We recorded the user actions, the interface feed-
backs and the geo-localization of the device. Both
sets of data were record at the same time for all