Evaluating Self-Explanations in iSTART: Word Matching, Latent Semantic Analysis, and Topic Models - Natural Language Processing and Text Mining - page 97

ilarity is computed in two ways: (1) using cosines — comparing the explanation
and the benchmark using the cosine formula (Referred as TM1) and (2) using KL
distances — comparing the explanation and the benchmark using the KL distance
(Referred as TM2). As before, formulae are constructed using Discriminant Analysis
in order to categorize the quality of explanation as Levels 1, 2, or 3.
6.2.4 Metacognitive Statements
The feedback systems include a metacognitive filter that searches the trainees' self-
explanations for patterns indicating a description of the trainee's mental state such
as “now I see ...” or “I don't understand this at all.” While the main purpose of
the filter is to enable the system to respond to such non-explanatory content more
appropriately, we also used the same filter to remove “noise” such as “What this
sentence is saying is ...” from the explanation before further processing. We have
examined the effectiveness of the systems with and without the filter and found
that they all perform slightly better with than without it. Thus, the systems in this
chapter all include the metacognitive filter.
The metacognitive filter also benefits the feedback system. When a metacogni-
tive pattern is recognized, its category is noted. If the self-explanation contains only
a metacognitive statement, the system will respond to a metacognitive category such
as understanding, not-understanding, confirmation, prediction, or boredom instead
of responding irrelevantly. Regular expressions are used to define multiple patterns
for each metacognitive category. If any pattern is matched in the self-explanation,
words matching the pattern are removed before evaluation. Examples of regular ex-
pression are shown below:
NOTUNDERSTAND :i(?:.?m |\ W+am)(?: \ W+ \ w+)? \ W+ \ W+(?:(?:not
(?: \ W+ \ w+)? \ W+(?:sure | certain | clear)) |
un(?:sure | certain | clear))
UNDERSTAND :now \ W+i \ W+(?:know | knew | underst(?:an | oo)d |
remember(?:ed)? | recall(?:ed)? | recogniz(?:ed)? | get |
got | see)
CONF :(?:so \ W+)?i \ W+(?:was | got \ W+it) \ W+(?:right | correct)
The first pattern will include “I'm not sure,” “I am uncertain”; second pattern
includes “Now I understand,” “Now I remembered”; and the last pattern includes
“So, I was right.” We originally constructed over 60 patterns. These were reduced
to 45 by running them on a large corpus of explanations and eliminating those that
failed to match and adding those that were missed.
6.3 iSTART: Evaluation of Feedback Systems
Two experiments were used to evaluate the performance of various systems of al-
gorithms that vary as a function of approach (word-based, LSA, combination of
word-based and LSA, and combination of word-based TM). In Experiment 1, we