Information Technology Reference
In-Depth Information
Within each configuration, all situations are
theoretically possible. However, we make the
distinction between the situations that we found
most relevant (black circles), intermediate ones
(gray circles) and those that seemed less attractive
(white circles). The least interesting situations are
extreme situations when the experimental proto-
col seemed impractical. Indeed, free tasks in an
artificial context seemed irrelevant because it
must be difficult for a user to act “naturally” in
the context of a usability laboratory. Similarly,
requesting users to carefully execute prescribed
tasks in a real context seemed unrealistic, because
interruptions are common in the real world.
In contrast, we believe the most interesting
situations are situations that present a good match
between the prescription of tasks and realism of
the context. Indeed, the artificial environment of
a usability laboratory is well suited to prescribed
tasks, the simulated environment allows users to
execute realistic scenarios, while free task are well
suited in a real context. Intermediate situations
are compromises between the two previous ones.
Note that the configuration in-situ does not include
one of the most interesting situations. Indeed,
we believe that this configuration is only a good
compromise when the configurations in-vivo or
in-simu cannot be implemented.
Possible approaches. From our point of view,
the four configurations defined above are more
complementary than competitive. The configu-
ration in-vitro is the reference tool to search for
usability issues. The configuration in-simu can
detect the features that make sense for the user
as well as some usability problems, but in a
controlled environment. Finally, the configura-
tion in-vivo can finalize the study with emphasis
on usage patterns. The configuration in-situ is a
special case that can substitute for in-simu or in-
vivo configurations if, respectively, the context
is difficult to simulate or letting the user perform
free tasks does not make sense. The choice of a
particular configuration can also be determined by
the objectives of the evaluation or by the technical
limitations of the system under test. For instance,
in-vitro, it is possible to use mock-ups via the
Wizard of Oz technique, which is quite impos-
sible in-vivo, with the notable exception of the
work of Consolvo et al. (Consolvo, et al., 2007).
For a given device, the “natural” order is to
perform the three types of evaluation in the se-
quence (in-vitro → in-simu → in-vivo) in order
to gradually identify usability issues and then
usages ones. The in-simu step is optional. This
approach however has a flaw: if some features
do not correspond to real usages, they have been
nevertheless developed, at great expense, during
the previous two steps. It is also possible to invert
this “natural” order and follow a reverse sequence:
(in-vivo → in-simu → in-vitro). The idea is to
identify usages patterns before focusing only on
the usability of the used features. The drawback
of this approach is that a critical usability issue
can mask an interesting feature, not used by users
only because of its poor usability.
CONCLUSION
In conclusion, we suggest that there are not only
two -laboratory and field- but at least four -labo-
ratory, simulator, field and real world- possible
configurations for the user evaluation of mobile
systems and ubiquitous environments. These
configurations are not competitive but comple-
mentary because they have varying abilities to
detect usability and usage issues.
ACKNOWLEDGMENT
The author would like to acknowledge the partners
and the founding organizations of the Franco-
Finnish project ADAMOS (Mobile Adaptive
Services - Design Parameters and User Experi-
ence Factors), and the Rhône-Alpes region project
IMERA (Mobile Interfaces for Augmented Reality
Search WWH ::
Custom Search