Large Scale User Trials - Human-Computer Interaction and Innovation in Handheld, Mobile and Wearable Technologies - page 142

the employed evaluation strategy focussed on
system logs and post-trial interviews.
This greater detachment of the evaluator from
the system use made the evaluation of Feeding
Yoshi more challenging than that of Treasure.
Although every participant was interviewed fol-
lowing the week's trial, the duration of the game
was such that particular instances of play or
players' motivations at specific times were often
forgotten. Other aspects were deemed irrelevant
by the players and therefore went unreported. As
the evaluators were unable to directly follow the
events of the trial, they were less able to focus
questioning on specific topics that emerged from
witnessed behaviour. Another issue arose when
some players reported using other forms of com-
munication technologies to discuss and encourage
play with other team members. This had not been
anticipated in advance and most of this information
was inaccessible for post-trial analysis.
If unable to directly observe the participants,
and not directly immersed in the game themselves,
evaluators might find it difficult to establish how
use of the system is developing. Opportunities to
observe or log unanticipated activity can be lost,
and without such behaviour being mentioned by
chance during interviews, its occurrence might go
completely undetected. We subsequently tried to
address these problems in the evaluation of later
systems.

a trial and to allow them to embark on interviews
with a better understanding of events in order to
tailor questioning to each individual participant's
experience.
Shakra was a mobile phone-based application
that analysed patterns of fluctuation in GSM cell
signal strength to provide summaries of the amount
of time a user spends walking. Users could view
their daily activity levels in comparison with the
accumulated activity totals of their friends, with
the intention of making people more aware of their
activity levels and thereby hopefully encouraging
them to achieve the recommended 30 minutes per
day. In the evaluation of Shakra users were given a
usage diary—a printed form that they were asked
to fill in and return at the end of the trial. Shakra
was piloted with three groups of friends over the
course of one week, the aim being to examine
the impact the activity tracking and sharing of
activity levels had on users' self-awareness and
to discover whether this motivated any change in
attitude towards physical activity.
The evaluation of Shakra attempted to address
some of the problems experienced in evaluating
Feeding Yoshi. The usage diaries were an attempt
to overcome the issues of delayed reflection, with
participants encouraged to document any signifi-
cant happenings that occurred each day. The diary
had 19 questions and was returned after the trial
but before the participant was interviewed, so that
evaluators could familiarise themselves with the
individual's experience and tailor specific ques-
tions to draw out particular events. However, it
became apparent during interviews after the trial
that the players would spend less time on the
(static) diaries on each successive day, stating
that they felt they were repeating the same things
day in and day out. Additionally, our evaluations
were focussed on how user behaviour changes
over time, and a static diary is not a tool adept
at uncovering such information. While the diary
could capture some very common issues that
arise in such experiences, getting at the nuanced
behaviour in a particular experience was much

DYNAMIC QUESTIONING: FLEXIFILL

The previous sections have shown that the evalu-
ation of Feeding Yoshi was performed reflectively
without any direct observation of the play. In
subsequent systems such as Shakra (Maitland et
al., 2006) and Connecto (Barkhuus et al., 2008),
we tried to overcome this problem by introduc-
ing new techniques that enabled more informed
reflective evaluation. The goal was to find a tech-
nological means of providing evaluators with a
greater degree of insight into user activity during