Large Scale User Trials - Human-Computer Interaction and Innovation in Handheld, Mobile and Wearable Technologies - page 150

Figure 2. Categorisation of Ubicomp system evaluations by time and space

phy—such as ongoing observation—which were
lost by adopting more summative methods. This
resulted in evaluators continually observing par-
ticipants' activities online in order to inform later
face-to-face interviews or questionnaires, as well
as carrying out a form of orchestration, interven-
ing as and when required in order to fix technical
glitches and keep the system running smoothly.
Adaptive evaluation can thus come to employ
existing orchestration techniques applied to other
systems, as described earlier in this chapter. We
note that in these more performance-based sys-
tems (such as Uncle Roy All Around You or Day
of the Figurines), orchestration is geared towards
maintaining the performance as well as ensuring
the smooth running of technology. An adaptive
perspective on evaluation then couples these two
facets of orchestration, using the idea that the
continual involvement with a system's execu-
tion—part and parcel of general orchestration
duties—can then also be used to inform ongoing
evaluation.

In summary, this chapter argues against a nar-
row restriction to any one particular evaluation
perspective. Instead, we see different approaches
as being tools to use or combine, as appropriate to
the evaluation context. There are times when either
quantitative or qualitative approaches are most
useful, and different regions of the figure favour
different evaluation techniques. For systems that
involve smaller, room- or building-sized spaces
and shorter durations from minutes to hours, we
would recommend constructing a video record
from multiple angles, perhaps also embedding sen-
sors within the environment and also potentially
conducting more formal experimental setups.
Evaluator involvement within these spaces can or
should be high. Within such evaluations, logging
is important, although remote monitoring is less
vital than for other regions. Remote monitoring
is likely to be unnecessary given the increased
evaluator access. For systems run in much larger
(e.g., city-sized or global) spaces but over similar
time periods, logging comes into its own as a vital
tool, as does remote monitoring, however equally