Experimental Setups for User Evaluation of Mobile Devices and Ubiquitous Systems - Human-Computer Interaction and Innovation in Handheld, Mobile and Wearable Technologies - page 158

of references and used terms includes: “field ex-
periments” (Hertzum, 1999), “field trails” (Jensen
& Larsen, 2007), “experimentations in the wild”
(Waterson, Landay, & Matthews, 2002), “reality
testing” (Bennett, Lindgaard, Tsuji, Connelly, &
Siek, 2006), and “quasi-experimentation” (Roto,
Oulasvirta, Haikarainen, Lehmuskallio, & Nyys-
sönen, 2004). In our knowledge, there is no attempt
of classification of these terms. To standardize
the vocabulary, we use the generic term “in-situ”
throughout this article even if the authors of cited
papers use another term.
Laboratory versus in-situ. In the literature,
numerous publications compare laboratory ver-
sus in-situ experiments (Fields, Amaldi, Wong,
& Gill, 2007). These comparisons between the
experimental protocols are mainly based on two
criteria: the number of problems detected and their
severity, generally as defined by Molich (Molich,
2000). In our knowledge, there is no attempt to
use the Hartson et al. criteria for the evaluation
of usability inspection methods (Hartson, Andre,
& Williges, 2003). The results published in the
literature vary: several publications claim that
there is no difference between the experimental
setups, while others argue the opposite.
If we study the publications, we first note
that most of them detect no or few differences
between laboratory and in-situ experiments. Beck
et al. (Beck, Christiansen, Kjeldskov, Kolve, &
Stage, 2003), Kjeldskov et al. (Kjeldskov, et al.,
2004), (Kjeldskov, et al., 2005), Betiol and Cybis
(Betiol & Cybis, 2005) as well as Kaikkonen et
al. (Kaikkonen, Kekäläinen, Cankar, Kallio, &
Kankainen, 2005) conclude that the differences
are not significant. However, while Betiol and
Cybis detect problems of greater severity in the
laboratory, Kaikkonen et al., indicate that problems
detected in the field are slightly more severe. The
Kjeldskov et al. publication (Kjeldskov, et al.,
2005) gives complementary results: the intersec-
tions between the issues detected by four methods
(including laboratory and in-situ) were important

for the critical problems, partial for the serious
ones, and weak for the cosmetic ones.
On the contrary, another evaluation, concern-
ing six methods tended to prove that the static
evaluation -the test user is sitting at a table- could
detect more usability issues than all the others,
in particular the in-situ evaluation (Kjeldskov &
Stage, 2004). These results are consistent with
the Baillie and Schatz conclusions (Baillie &
Schatz, 2005). As the in-situ experiments are
more complex to set up than laboratory ones, it
seems preferable to only use this technique. More
recently, Duh et al. called the Kjeldskov al. results
into question since they detected more usability
issues in-situ than in the laboratory (Duh, Tan, &
Chen, 2006). The Duh et al. experiments could
be easily compared to Kjeldskov et al. ones since
they used the same classification of ergonomic
problems severity: the one defined by Molich
(Molich, 2000). Moreover, the Duh et al. experi-
ment protocol was very similar to the Kaikkonen
et al. one who concluded, on the contrary, that the
differences were tiny (Kaikkonen, et al., 2005).
These results are consistent with the Po et al. con-
clusions about heuristic evaluations: they suggest
that heuristic evaluations detect more usability
issues in-situ than in the laboratory (Po, Howard,
Vetere, & Skov, 2004).
Discussion about possible biases. These results
lead to the conclusion that not only is there no
consensus concerning the added value of in-situ
experiments, but also, that the comparisons be-
tween laboratory and in-situ experiments are not
easily reproducible. A closer look on the experi-
mental setups gives interesting indications about
possible biases.
In all the experiments, at least one person -
usually a facilitator- is present close to the user.
In the majority of the experiments, a gooseneck
camera is used to record the user interaction with
the mobile device. To capture the context in the
laboratory, ceiling cameras or cameras on tripods
are used. In-situ, cameras are fixed on the user's
shoulder or cameramen are in charge of this task.