Experimental Setups for User Evaluation of Mobile Devices and Ubiquitous Systems - Human-Computer Interaction and Innovation in Handheld, Mobile and Wearable Technologies - page 166

GENERALIZATION in-vitro, but will be detected in-situ, providing it

shine... This is a limitation of this configuration.
The elements of the environment are inherently
unpredictable. So, there is no guarantee that they
will occur.
We name this configuration “in-situ”, which
is consistent with the terminology proposed by
Kjeldskov et al. (Kjeldskov & Skov, 2007).
The real world (in-vivo). The experimental
setup changes radically. The facilitator and ob-
servers are absent and so do not influence the
user, who can use the system under test as he/
she whishes. Two types of results are obtained:
(1) usage statistics of the device, i.e. the tasks
that the user perform naturally; and (2) usability
issues “filtered by the real usage”, i.e. usability
issues related to the user activity in the real life.
By definition, these tasks are only a subset of
the possible tasks that can be performed with the
device. So, it is consistent that the number of us-
ability issues detected in-vivo is much less than
in-vitro or in-situ. However, ergonomic problems
detected would gain in relevance, as they are trig-
gered by the actual use of the device.
However, the results obtained with this con-
figuration should be taken with caution since the
tasks performed by the subjects and events from
the context do not occur in a deterministic man-
ner, and can be viewed as hidden independent
variables. To minimize this risk, it can be inter-
esting -if feasible- to significantly increase the
number of subjects and the experiment duration to
statistically increase the probability of occurrence
of the events. Moreover, the fact that the user is
aware of being part of an experiment provides a
bias, known as the “Hawthorne effect” (Mace-
field, 2007). Although controversial, this effect
suggests that the user performance may improve
by the only fact that the user is aware of being
part of an experiment.
We name this configuration “in-vivo”, which is
consistent with our terminology. Note that Kjeld-
skov et al. did not mention this configuration in
their article (Kjeldskov & Skov, 2007).

Taking as starting point the distinction between
laboratory and in-situ experimental setups in the
literature, we proposed the concept of in-vivo
as a specialization of the in-situ configuration.
However, the search for a classification system of
these configurations, as well as recent literature
results (Kjeldskov & Skov, 2007), led us to define
another configuration, the “realistic simulation”.
In addition, we standardize the names of these
configurations, based on a terminology inspired
by biology, which is consistent with the terminol-
ogy proposed by Kjeldskov et al. (Kjeldskov &
Skov, 2007).
The laboratory (in-vitro). The usability labora-
tory is the reference configuration. It is character-
ized by an artificial context. The usability problems
detected are directly dependent on the tasks per-
formed by the user, specified in the scenarios. This
configuration is suitable for the study of usability
issues. It is therefore unlikely that usage issues or
patterns could be indentified with this configura-
tion. We now mane this configuration “in-vitro”
to be consistent with the terminology proposed
by Kjeldskov et al. (Kjeldskov & Skov, 2007).
The field (in-situ). Compared to the in-vitro
configuration, the main interest of the in-situ
configuration is to have more realistic contextual
elements. The experimental setup is only slightly
different from the in-situ configuration. Notably,
user tasks are also prescribed. So, it is not sur-
prising that usability issues detected are similar
in type and number in these two configurations.
Moreover, the presence of a facilitator and observ-
ers is a known bias.
If the environment is somewhat constrained,
even if the users are often isolated from the real
world by a “bubble”, unplanned events from
the environment can make the evaluation more
relevant. The usability issues detected are “local-
ized” in the real context. For instance, if a smart-
phone screen is not readable in the direct sunlight
brightness, this usability issue will not be detected