Large Scale User Trials - Human-Computer Interaction and Innovation in Handheld, Mobile and Wearable Technologies - page 149

problems collecting this observational data are
typically mitigated by the limited duration, such
evaluation activity is often quite intensive due to
monitoring or shadowing participants over such
a large physical space.
Experiences involving fully unconstrained
spaces and long durations perhaps present the
greatest challenge analytically, even though
systems like these have become increasingly
common for researchers concerned with a key
characteristic of Ubicomp—the appropriation
of technology in everyday life. The player ex-
perience in games like MobiMissions (Grant et
al., 2007) and Day of the Figurines (Crabtree et
al., 2007) lasted a month or more, during which
time players could roam wherever they pleased.
Such systems are usually assessed without direct
observation, instead exploiting mixed methods of
interviews, questionnaires, usage diaries and so
on. We note that such conventional, static evalu-
ation techniques were employed in our earlier
systems like Feeding Yoshi. Connecto and Ego
followed a modified version of this approach
due to the trial lengths and unconstrained spa-
tial distributions of users, but, importantly, data
streamed via GPRS enabled evaluators to view
the moment-by-moment actions of the users and
adapt both the systems' orchestration and their
ongoing evaluations accordingly.
Figure 2 attempts to summarise the systems
mentioned above in terms of time and space.
Difficulty in exercising experimental control
increases as both geographic distribution ('space'
in Figure 2) and temporal duration increase. For
Treasure, it tended to be easier to mitigate uncer-
tainty about where and when interactions might
occur due to 'park-sized' interactions occurring
over minutes that could be covered by saturat-
ing the space with evaluators. Shakra and Yoshi
introduced uncertainty over where interactions
might happen due to 'city- or multi-city sized'
interactions as well as extending data collection
times from minutes into days. Connecto and Ego
stretch the boundaries of evaluation to a greater

degree by involving 'city-sized' interactions for
weeks or perhaps months. Finally Hungry Yoshi
pushed even further by extending the user trial
to a global audience, magnifying the uncertain-
ties over where and when interaction may occur.
Hungry Yoshi also pushed our adaptive evalua-
tion techniques by introducing the idea of mass
participation. As applications run on users' own
devices, trials do not require a specified end date
when equipment is retrieved, so trials can run for
as long as researchers are willing to gather and
study data, and interact with users.
As we increase the temporal and geographic
scale of user experiences, we gain a greater op-
portunity to see how ubiquitous computing may
become embedded into everyday life, subject both
to mundane routines of work and home, and to
possibilities for serendipitous and opportunistic
interaction. Through this we might explore more
fully how competence, system appropriation and
mastery, as well as strategies and tactics (particu-
larly in the case of Ubicomp games) develop in
use. As we have seen, however, this interest is in
tension with our ability to evaluate such uses and
environments. For summative methods, which as
we note may be more practical in longer term
trials of a greater spatial distribution, evaluators
must assess much data collected in a post-hoc
way, gleaning information from users during
interviews and system logs. In contrast, ethno-
graphic studies favour observation and rich de-
scription as a way of understanding system use.
By engaging and immersing oneself in the expe-
rience, evaluators may be able to observe many
of the more subtle interactions that take place.
Such techniques have been key to understanding
the nature of interaction either over a lengthy
duration or wide physical space. We faced diffi-
culties in adopting such ethnographic techniques
on interaction both unconstrained in space and
happening over a long time, and so we attempted
to produce a synthesis of both forms of technique.
As such, in the evaluation of Ego we attempted
to claw back some of the properties of ethnogra-