Evaluation - Advances in Personalized Web-Based Education - page 92

student model (Brusilovsky and Millán 2007). It affects automated tutoring sys-
tems in making instructional decisions (Li et al. 2011), since a student model ena-
bles understanding and identification of students' needs (Sucar and Noguez 2008).
Although, the adaptation generated by student modeling techniques often tend
to improve the interaction of the learner with the educational system, most of the
time the exploitation of such techniques makes the system more complex, less
predictable and buggier. As a consequence, it should be evaluated whether or not
the student model really improves the system (Gena 2005; Chin 2001). Therefore,
the evaluation of a student model is a crucial factor. Even though the evaluation
of adaptive systems is a difficult task due to the complexity of such systems, as
shown by many studies (Lavie et al. 2005; Markham et al. 2003; Del Missier and
Ricci 2003), several researchers have attempted to assess the student model of
their adaptive system. An assessment of the student model that SQL-Tutor uses is
presented in Mitrovic et al. (2002). Also, Weibelzahl and Weber (2003) performed
the evaluation of the accuracy of the student model of an adaptive learning system,
called the HTML-Tutor. A more recent attempt to assess the effectiveness and the
accuracy of the student model, which was applied in an ITS for learning software
design patterns, was done by Jeremi´ et al. (2009).
Although, there are many evaluation methods available in literature review,
as Mulwa et al. (2011) have mentioned, there is no a standard agreed measure-
ment framework for assessing the value and effectiveness of the adaptation yielded
by adaptive systems. The most common practice of evaluation is experiments.
However, there is not an accurate, clear and agreed framework in which an experi-
ment for the assessing of a student model should be performed. Furthermore, it
is important to not only evaluate but also to ensure that the evaluation uses the
correct methods, since an incorrect method can lead to wrong conclusions (Gena
and Weibelzahl 2007). Besides a well-designed evaluation framework underlines
the success of an approach and its potential value to others (Dempster 2004).
For this reason, the presented evaluation process is performed applying the
evaluation framework PERSIVA (Chrysafiadi and Virvou 2013a), which includes
both questionnaires and observations through experiments. Applying the particular
evaluation framework, either the educational impact (i.e. performance, satisfaction,
change of learners' attitudes) or the adaptation of the personalized and/or adap-
tive tutoring system is assessed. The evaluation of the educational impact is based
on the Kirkpatrick's model (1979). Furthermore, experiments play a major role in
the particular evaluation method, as they are appropriate for the evaluation of the
effectiveness and successfulness of user models (Chin 2001; Virvou and kabassi
2004).
The remainder of this section is organized as follows. Initially, the evaluation
framework PERSIVA is presented. Then, the evaluation criteria, process and popu-
lation are described. The presentation of the evaluation's results follows. Finally,
the conclusions drawn from the evaluation process are presented.