Evaluation - Advances in Personalized Web-Based Education - page 94

4.2.3 The Evaluation Process

The presented evaluation process includes experiment research and questionnaires.
Below, the particular evaluation processes that were used to assess each individual
evaluation criteria were described.

• Learners' general satisfaction : For gathering this kind of information a ques-
tionnaire (Questionnaire A, Appendix B) was used. The questions were close-
ended based on Likert scale with five responses ranging from “Very much”
(5) to “Not at all” (1). The questions were divided into six categories based on
the type of information that were evaluated. The questions of the first category
are related to the quality of the content. The questions of the second category
concern the quality of instruction. The third and fourth categories concern the
friendliness and usefulness of the programming tutoring system correspond-
ingly. The questions of the fifth category are aimed to evaluate the adaptivity of
the system. The final question concerns the overall rating of the system.
• Learners' performance : The learner's performance is defined by either her/his
degree of success in tests and exercises or the times that s/he needed to read
a particular domain concept. Therefore, factor p was calculated to assess the
learners' performance. In particular, the value of factor p is derived if the learn-
er's degree of success in a particular domain concept is divided with the times
that s/he needs to read the particular concept (Eq. 4.1 ). The better the degree
of success is, the higher the value of p is. The lower the value of times that
the learner read the concept is, the higher the value of p is. Also, the lower the
degree of success is, the lower the value of p is. The higher the value of times
that the learner read the concept is, the lower the value of p is. Learners' perfor-
mance was measured by conducting an experiment with an experimental group
(the group of students which used the presented programming tutoring system)
and a control group (the group of students which used a similar programming
tutoring system from which the presented student model was absent). Factor p
was calculated and compared for both groups.

degree of success in Ci i
reading times of Ci i

p =

(4.1)

• Changes on learners' behavior and thoughts about computer program-
ming : For gathering this kind of information a questionnaire (Questionnaire B,
Appendix B) was used. The questions were close-ended based on Likert scale
with five responses ranging from “Very much” (5) to “Not at all” (1).
• Changes on learners' behavior and thoughts about e-learning : For gather-
ing this kind of information a questionnaire (Questionnaire C, Appendix B) was
used. The questions were close-ended based on Likert scale with five responses
ranging from “Very much” (5) to “Not at all” (1).
• Results on learners' further studies : The assessment of that criterion is per-
formed with both questionnaire and experiment. In particular, a questionnaire
(Questionnaire D, Appendix B), which included five close-ended questions