Information Technology Reference
In-Depth Information
Moreover, the same tasks are prescribed to the us-
ers both in the laboratory and in-situ. In summary,
most of the experimental setups are identical or
very similar in the laboratory and in-situ. This
methodological approach is interesting because
the experimental setups can be removed from the
set of independent variables. As a consequence,
the results of the experiments can be compared
according to the context -the unique independent
variable.
However, this in-situ approach has important
consequences because the social activities of
the user are constrained by the experiment. For
instance, Kjeldskov et al. observed that the set
{user, facilitator and cameraman} could create a
“group effect”: the people of the user's entourage
deviated from the group track and did not try to
interrupt the activity of the user (Kjeldskov, et
al., 2005). We detected a similar effect (Jambon,
Golanski, & Pommier, 2007). So, in our point
of view, this approach makes the user walk in a
“protection bubble” that isolates him/her from
the natural context.
Baillie and Schatz found that users performed
their tasks faster and made fewer mistakes in-
situ. They suggested that users might feel more
“relaxed” outside of the laboratory (Baillie &
Schatz, 2005). Similarly, Schulte-Mecklenbeck
et al. pointed out that, for a task of information
retrieval on the web, users did not finish all their
tasks less frequently and retrieved more informa-
tion in the laboratory than in-situ. The authors
speculated that a “psychological pressure”, due
to the presence of the observer, is the main reason
(Schulte-Mecklenbeck & Huber, 2003).
On the contrary, when users are left alone,
Isomursu et al. observed significant improvements
in the quality of the self-reported usability prob-
lems (Isomursu, Kuutti, & Väinämö, 2004). The
authors used the “experience clip technique” that
consisted in charging one of the users to play the
role of the cameraman. In this experiment, with
couples of users, one of the users was in charge
of recording a video-clip (with a camera phone)
of the other user (interacting with the mobile
device) whenever a usability or usage problem
occurred. Isomursu et al. detected that when the
users did not know each other (in this case, the
video-shooter was a researcher), the quality and
pertinence of the video-clips illustrating the us-
ability or usage problems reported in the field
dropped significantly. Surprisingly, when asked,
the users “explained that the presence of the
researcher did not have any effect on the usage
situation” (Isomursu, et al., 2004). Moreover, in
this latter situation, the user did not try to correct
the device related problems, but instead usually
asked the researcher for help.
We argue that from a methodological point
of view, the in-situ experiments described here
can be viewed as laboratory experiments that are
“relocated” in the field. As a consequence, the
supposed added value of the real context may
be partially lost due to the “protection bubble”
effect. In these articles, no formal distinction is
made between laboratory and in-situ experimen-
tal setups. The in-situ experimental setups are
usually defined as the opposite of the laboratory
experimental setups. There is a tacit consensus to
consider in-situ experimental setups as placing the
user in a “natural” context or in an “ecological”
situation. As a consequence, authors make only
one distinction between laboratory and in-situ
experimental setups. In our opinion, there are two
kinds of in-situ experimental setups.
RESEARCH AGENDA
Hypothesis. The latter remarks and anecdotes led
us to study more precisely the experimental pro-
tocols used in laboratory and in-situ. For all these
experiments, the protocols are almost identical in
laboratory and in-situ. The tasks performed by the
users are prescribed. Usually, a facilitator stays
in the immediate vicinity of the user. In addition,
cameramen and/or observers may be required to
facilitate the data collection. The instrumenta-
Search WWH ::
Custom Search