Information Technology Reference
In-Depth Information
tend to evaluate an ALS as a whole , focusing on an end value delivered by the sys-
tem such as the overall user's performance or the user's satisfaction...Evaluating
a system as a whole can be acceptable in the field where no acceptable compo-
nent model of a system can be identified. However, it is not the case for adaptive
systems...'.
This paper suggests using a layered evaluation process , in which one of the layer
is represented by the learning material and an other by GUIs. The authors intro-
duced such an approach to guide designers in the ALS development process. Such a
layered approach is in line with the User-Centred Design (UCD), used in the TER-
ENCE project, where the evaluation is used iteratively and incrementally to refine
the requirements, the design or the development of the system. Moreover, all the
evaluation studies reported in [3] stressed the fact that the usability issues of the
ALS interfaces have to be solved before starting the evaluation of the ALS in order
to minimize bias in the evaluation study of the ALS' usability as “a whole”, that is
the ALS' pedagogical effectiveness. A thing which is mandatory in UCD as well.
The TERENCE project took up such a two layer up for the learning material and
the GUIs, before the large scale evaluation, in two main manners:
the TERENCE team evaluated the learner material and the GUI prototypes via
expert-based evaluations reported in [9],
the TERENCE team evaluated the refined version of the learner material and the
first releases of the GUIs via user-based evaluation, reported in [5].
In this paper, we focus on the user-based evaluation, mainly reported in [5], of the
most complex GUI, namely TERENCE learner GUI. The entire learner GUI is avail-
able at http://hixwg.univaq.it/learner-gui.html; its design is described in [10].
2
Experiment Description
For the experiments we here describe, we adopted user-based criteria methods like
observational evaluation [1], semi-structured interviews [7] and think-aloud proto-
col [6]. In fact, the approaches used in the literature for evaluating TEL projects
are mainly user-based (see [8]). An important reason justifying the usage of user-
based approaches in TEL projects is the fact that users are often involved in the
design of the projects. Like the other TEL projects, the TERENCE project involves
users in the evaluation process. In doing so, the TERENCE team opted for meth-
ods that are adequate to the TERENCE main users, that is, 7-11 year old children,
and prone to being used in numerous but short inexpensive evaluation sessions. In
fact, observational evaluation, semi-structured interviews, and think-aloud protocol
are semi-structured methods for examining and reporting problems with the learner
GUI in qualitative and quantitative ways.
The reports of the assessments for the learner GUIs usability evaluations and the
learning material in [5] is divided as follows: (1) goals of the assessment; (2) par-
ticipants , that is, the description of the involved users; (3) tasks and material ,thatis,
 
Search WWH ::




Custom Search