Information Technology Reference
In-Depth Information
Table 1. Summary of the evaluations of the discussed systems
Number of
users
Evaluator awareness and
involvement
System
Space
Time
Data Collected
20
Direct observation,
interviews,
video, system logs
Small
(Constrained)
Very short (min-
utes)
Direct observation of users'
actions
Treasure
Large (Uncon-
strained)
Short
(days)
16
Indirect post-event / reflec-
tive methods
Feeding Yoshi
Interviews, system logs
Large (Uncon-
strained)
Short
(days)
16
Interviews, usage diaries,
system logs
Shakra
Indirect / reflective
12
Interviews, semi-adaptive
usage diaries, GPRS,
system logs
Large (Uncon-
strained)
Indirect / reflective /
adaptive
Connecto
Medium (weeks)
20
Interviews, ongoing ob-
servation through GPRS
uploading, system logs
Large (Uncon-
strained)
Long
(Month)
Semi-direct / indirect /
adaptive
Ego
5000
(play > 5
days)
Interviews, ongoing ob-
servation through GPRS
uploading, system logs,
task mechanism to ask
questions directly
Large
(Unconstrained)
Long
(Year)
Semi-direct / indirect /
reflective / adaptive
Hungry Yoshi
that we may manage and control uncertainty in
evaluation procedures. Using time and space as a
basis to characterise evaluated user experiences,
we can distinguish between four rough categories
and draw out distinctions between the kinds of
evaluative techniques we found necessary for
each. The table below summarises much of the
prior discussion of our five systems, and serves
to help with the subsequent discussion.
Treasure is an example of a system in which
interaction is relatively
constrained with regard
to both duration and space
. Although there were
some difficulties in collecting data for evaluation,
and tight experimental control was limited to some
extent through the trial's openness to interruption
by people outside the trial (such as car drivers
passing through the area), Treasure's limited
spatial and temporal extents permitted close and
relatively comprehensive direct observation. On
the other hand, broader issues of how Ubicomp
technologies may be woven into everyday life are
not addressed easily when time and space are so
constrained.
User experiences of
longer duration but con-
strained space
tend towards more 'traditional'
Ubicomp systems such as the Active Badge Loca-
tion System (Want et al., 1992) and smart home
environments (Demiris et al., 2007). Evaluations
here have necessitated ongoing commitment to a
static setting like a home or an office building.
To a varying degree, the challenge for evaluators
in studying these scenarios is in maintaining this
continual involvement in the systems—rather
than an intense, spatially distributed involvement
(see below). Thus the problem becomes one of
determining when interesting interactions take
place rather than where.
A
shorter duration but a less constrained
space
suggests participants may move in a far less
restricted way—with no set boundary. City-scale
experiences such as Human Pac-Man (Cheok et
al., 2003) and Uncle Roy All Around You (Benford
et al., 2004) involved evaluating games that took
place over a large yet flexible area, with player
experiences lasting at most a few hours each. Direct
observation was still possible due to the temporally
focussed nature of player interactions. Although
Search WWH ::
Custom Search