Large Scale User Trials - Human-Computer Interaction and Innovation in Handheld, Mobile and Wearable Technologies - page 139

contextuality and appropriation that let users take
full advantage of new mobile devices. As Carter
and Mankoff (2007) put it, “Ubicomp systems
[are] more difficult to evaluate than desktop ap-
plications. This difficulty is due to issues like scale
and a tendency to apply Ubicomp in ongoing,
daily life settings unlike task and work oriented
desktop systems.” Many of these challenges have
already been faced by researchers studying the use
(rather than usability ) of Ubicomp technologies
in the wild. Observational techniques founded in
ethnography may be well suited in principle but
in practice are often hampered because of the
difficulty of actually observing users' activities.
Small devices such as mobile phones and PDAs
can easily be occluded from view, and people's use
may be intimately related to and influenced by the
activity of others far away (Crabtree et al., 2006).
In this chapter, we reflect on our studies of four
mobile multiplayer games: Treasure (Barkhuus et
al., 2005), Feeding Yoshi (Bell et al., 2006), Ego
and Hungry Yoshi (McMillan et al, 2010) and
of two everyday awareness applications: Shakra
(Maitland et al., 2006) and Connecto (Barkhuus
et al., 2008). The development of these systems
has spanned the last seven years, with user experi-
ence design and evaluation techniques evolving
over this time. We show a progression from early
trials lasting around a quarter of an hour and tak-
ing place within a specific confined area, to trials
months or years in length (indeed, often without a
specified end date) that explore users' integration
of technology into their everyday lives. Studying
system use over longer periods of time and in less
constrained settings provides greater opportunity
for witnessing unanticipated behaviour as users
take ownership of the system, but can leave the
evaluator more detached from the trial. Addi-
tionally, while many have studied the effects of
uncertainty with regard to positioning accuracy
and network connectivity on the user experience
e.g.,(Crabtree et al. (2004)), the impact these fac-
tors have on evaluators is not usually explicitly
acknowledged.

Here we discuss the strategies that we, as evalu-
ators, employed to discover participants' reactions
and experiences with regard to our five systems.
The studies are presented chronologically, as the
challenges faced in one study often influenced
design and evaluation of subsequent systems. We
suggest methods for keeping evaluators informed
of activity during a trial that might take place
over an extended period of time and over a wide
geographical area, and suggest that such informa-
tion is of crucial importance to adaptation of an
ongoing evaluation based on evaluators' continual
involvement with it or, in more extreme cases,
immersion in it. Such adaptation may be done in
order to inform and improve ongoing and post-hoc
analysis. To conclude the paper we discuss the
temporal and geographic scale of each study as
contributing factors to the complexity of running
such studies, and of gathering and interpreting
evaluation data.

RELATED WORK

Researchers have examined how a particular
design (along with external factors) can encour-
age adaptive behaviour, and how that behaviour
can in turn inform the design process. Vougiazou
et al. (2006) discuss a number of systems that
feature evolving cooperation or the dynamic and
ongoing creation of authored game rules. It is
notable that the evaluation of each system, for
the most part, was based on direct observation
and was short-term in duration. While the authors
identify the challenges of long-term studies, no
strategies are offered to overcome them. Iterative
participatory design practices for developing the
usability of mobile devices offer to some extent
more 'agile' evaluation techniques (de Sá et al.,
2008), and there are some existing demonstrations
of remote, in situ data collection systems (Carter
et al., 2007; Consolvo & Walker, 2003; Froehlich
et al., 2007). Froehlich et al. (2007) have explored
context-dependent 'experience sampling' systems