Large Scale User Trials - Human-Computer Interaction and Innovation in Handheld, Mobile and Wearable Technologies - page 144

with both activities then shaping and informing
subsequent as well as ongoing analysis. A feature
of this continual connection is that it enables data
to be gathered in a less intrusive way than direct
observation or shadowing. Such direct techniques
are often impractical in experiences that span a
large area over a long duration. In Ego, the evalu-
ators were able to passively involve themselves
in players' experiences without heavily impacting
them, and yet gain understanding about interac-
tions between players.
However, on some occasions, the evaluators
were moved to make more active interventions
during the trial. For example, the users would
sometimes encounter technical difficulties, which
is to be expected with a system running over such
a large period of time and geographical distribu-
tion. In past trials it had been difficult for users
to understand when the system was not working
correctly, typically resulting in frustration and
disengagement from the experience. However,
through evaluators' continual awareness, the iden-
tification of such problems was no longer the sole
responsibility of the user. In Ego, by being able
to observe the system and immerse themselves
in the game as if they were playing, evaluators
could remotely identify occasions when patterns
of activity were potentially unusual and intervene
accordingly. For example, one participant who
had been an extremely active player had only
managed a very low score one day, which did
not fit with his usual pattern of play. He had not
contacted the evaluators since he was in the final
few days of the trial and he assumed he had done
something to break the device. Through their con-
tinued awareness of the system and individuals'
play, evaluators were able to identify and fix this
problem. In this case the database on the mobile
client had been corrupted, but it was then possible
to fix this within an hour of the problem being
identified. This prevented this user's experience
from being cut short prematurely and therefore
generated more data for analysis.

It is worth noting that one possible side-effect
of the continual attention paid to user problems by
evaluators is an increased engagement in the trial
by those users, therefore opening up findings to the
potential criticism of an increased likelihood of the
Hawthorne effect. “Proponents of the Hawthorne
effect say that people who are singled out for a
study of any kind may improve their performance
or behavior not because of any specific condition
being tested, but simply because of all the attention
they receive.” (Rice, 1982). Such a view seems to
indicate that the degree of attention paid to those
participating in a study is positively correlated with
any subsequent Hawthorne effect—a commonly
held assumption being that the no human-centred
study is completely free from the Hawthorne effect
(Macefield, 2007). However, the generalisability
of the Hawthorne effect has recently been called
into question (Rice, 1982; Macefield, 2007).
Macefield (2007) presents a full discussion on
the limitations of such a generalisation with re-
spect to usability evaluations. Similarly, Crabtree
& Rodden (2004) propose that the Hawthorne
effect is often overestimated when considering
ethnographic studies in the workplace and home,
simply because when in these environments people
“have better things to do than impress or worry
about the ethnographer”.
The Ego evaluation also featured a new ver-
sion of the FlexiFill tool. Rather than relying on a
relatively broad sampling of user activity over the
course of the trial as the focus of questions, evalu-
ators used the improved FlexiFill which permitted
the addition and tailoring of questions in order to
make specific enquiries at any point, once again
moving a further step away from post-hoc static
diaries and questionnaires. Any observed actions
that seemed interesting or puzzling could be put
to the user the next time they logged in, without
waiting for a post-trial interview, reducing the
risk of player accounts becoming skewed over
time or, worse, forgotten.
One subtle example of this revealed a complex,
but somewhat misguided tactic employed by a