Large Scale User Trials - Human-Computer Interaction and Innovation in Handheld, Mobile and Wearable Technologies - page 151

important is the use of mobile camera operators
recording video when and where possible. These
fragments are, currently, more feasibly pieced
together after the event although we note that
advances in wireless networking should make
real-time collation feasible very soon. Systems in
smaller spaces but over longer durations in turn
preclude more exhaustive video recordings, again
favouring remote monitoring and logging as useful
ways to enrich a video record consisting of short
samples across the duration of the system's run.
Finally, for systems in the more problematic larger
space, longer duration region, we recommend our
adaptive evaluation techniques.

adapted for a particular user or subset of users,
the subsequent findings remain comparable with
user group as a whole and that the nature of the
adaptation over time is well-documented.
As part of our ongoing work, we are creating
FlexKit, a platform for creating component-based
smartphone applications and for adapting the
software used in their evaluation. FlexKit is an
adaptive middleware infrastructure for creating
component-based systems with additional sup-
port for recommendations, reviewing and user
feedback and sharing of software components. The
infrastructure itself supports users, developers and
evaluators at different levels. FlexKit is a step on
from FlexiFill: where FlexiFill enabled evaluators
to edit the questions put to users, FlexKit enables
evaluators to provide new modules to users so as
to alter their applications. FlexKit also offers the
ability to update logging code, so as to capture
data that was either missed or deemed unneces-
sary based on earlier premises in the user trial.
In this section we are going to focus on its use
for evaluation.
FlexKit makes use of the Domino (Bell et
al., 2006), component architecture to support the
integration of software modules at runtime. The
current version of FlexKit is available for iOS and
it is currently being alpha tested in two systems,
World Paint and FlexBook. World Paint is a collab-
orative painting game that allows users to 'paint'
the world by tracking their movement and display-
ing paint blobs over a map of the world. While
FlexBook is an application for customising and
building Facebook posts. Within FlexKit, modules
are installed explicitly though user interaction or
implicitly though system updates. Modules can
be obtained in two ways: passed from peer-to-
peer over ad hoc data connections, or they can
be obtained from the shared module repository
available via the FlexKit module store. Once a
module exists on the local device, Domino can be
instructed to integrate the module. The module is
then available within the launched application and
a user can include it in the active configuration.

THE CHALLENGES OF USING
ADAPTIVE EVALUATION

There are three overarching and inter-related
challenges to consider in the design of adaptive
evaluations: firstly, the appropriate triangulation
of both evaluation methods and of evaluation
data; secondly, capturing adequate amounts and
types of data while avoiding data overload; and
thirdly, maintaining scientific rigour. Although the
triangulation of data from multiple sources and
of mixed methods is an approach (Denzin, 1978;
Mackay & Fayard, 1997; Wilson, 2006) rather
than a problem, a challenge lies in the appropri-
ate selection of methods and/or data to be logged
in order to avoid data overload when attempting
to formulate reliable findings which help answer
the research questions of concern. Data overload
may occur due to the sheer volume of data being
captured, as could potentially be the case in any
long-term or large-scale evaluation. It is exacer-
bated somewhat during adaptive evaluations when
ongoing analysis of data is required in order to
maintain an awareness of what is happening in
the field. While the question of scientific rigour
should be of concern to any researcher, when
employing adaptive evaluation methods, research-
ers must strive to ensure that if an evaluation is