Information Technology Reference
In-Depth Information
Similarly, the assignment of questions to the set of topics
T
and learn-
ing objectives
was performed manually in order to maintain robustness.
However, not all topics or learning objectives were identified within the three
question sets. Table 2 gives an overview of the mappings' statistics. Column
avg ˄ ∈T (
L
( q, ˄ )
|{
∈M ˄ }|
) contains the average number of question per topic,
( q, ˆ )
while avg ˆ ∈L (
|{
∈M ˆ }|
) represents he average number of question per
learning objective (12 in total).
Table 2. Statistics on the topic and learning objective mappings
avg ˄ ∈T ( |{ ( q, ˄ ) ∈M ˄ }| )
avg ˆ ∈L ( |{ ( q, ˆ ) ∈M ˆ }| )
Data set
Earth sciences
8.46
34.78
Nutrition
13.25
26.5
Homeschooling
4.89
19.1
3.2 Experiment
The evaluation of the recommender models introduced in Subsections 2.2 and
2.3 is not an easy task for several reasons. First, to learn such models, a history
of user interactions with the QA system is needed. Without any kind of recom-
mendation engine behind the search or browsing functionality, such interactions
would not be possible, or even reliable, since the user is not aware of the possible
question choices.
Secondly, if suggestions are provided, even in their simplest form, the result-
ing browsing log would not reflect the users natural learning process, but rather
a learning process influenced by the capabilities of the used recommendation
engine. Therefore, the recorded question sequences would still not be suitable
to be used for training a new recommender model which relies on the natural
learning process of the user.
In order to evaluate the performance of the LoR, due to the lack of resources,
a scenario of user interaction with a QA system was simulated, where recommen-
dations were not provided at all. Having an overview of the available questions
is not feasible, given the size of the datasets. Therefore, for each domain, five
subsets of 20 questions were randomly generated and users were asked to order
each of these 20 question-sets in the sequence that they, personally, would want
to ask them or would want learn about.
Table 4 shows, for each of the three domains, the number of collected ques-
tion sequences, i.e. the total number of user responses, the number of distinct
questions within the collected sequences and their percentage with respect to
the total number of questions.
Overall, about 13 male and female users participated in this survey, but
not all of them provided an ordering for each of the question sets. The
obtained number of sequences are generally balanced between male and female
participants.
 
Search WWH ::




Custom Search