The Manual Revision of the TERENCE Italian Smart Games - 2nd International Workshop on Evidence-Based Technology Enhanced Learning - page 28

2 Manual Revision

The objective of the manual revision was to control the automatic generation and
ensure the formal, technical, and content correctness of all the components of the
game proposed for the stimulation of understanding of the written text. Details on
the guidelines and on the software system that supported the manual revision can be
found in [3].
The revision work was divided into 3 steps:

• Formal revision, i.e.: correction of grammatical and syntactic errors in the text,

correction of punctuation, correction of the verb (present tense, active form),
correction of referring expressions, check of sentence length and structure;

• Substantial technical revision, i.e.: check of game identification number, correc-

tion of the questions texts, correction of the solutions (by keeping fixed the main
event), selection of new fixed events for solutions;

• Construction of cause/effect games, i.e.: text proposal, check out of proposals,

games uploading.

Each operator studied the text of the story and reviewed all the games associated
with it.
By proceeding on the basis of the output of the automatic generation, the manual
review of TERENCE games was initiated through a software to speed up the au-
dit work and its monitoring. The operator, accessing the software, could select the
games by selecting the proper topic and story. The Italian revision was conducted
on 4 books: 3 books (16 stories) for students 7-9 years old and 1 topic ( 9 stories)
for students 9-11 years old.
The software shows on left of the interface the story text (so to let the operator
to have the context always available) and enables in a tabular fashion to view the
related games. After a game with its own ID was selected, it was possible to identify
the type of game on the screen (before/after, before/while, before/while/after, what,
while/after, who). Each game had its own fixed event (the event around which the
entire game is build), completed with the event description, the game question, as
well as a set of multiple-choice questions that depends on the selected game type.
The operator's job was firstly to verify the congruence between the fixed event
text and the text of the proposed questions and solutions.
In general, the revision was structured, with the general principle to modify as
little as possible the work of the automatic generation.
The revision consisted in correcting: (i) the grammatical and syntactic errors in
the text; (ii) punctuation; (iii) verbs (present tense, active form); (iv) referring ex-
pressions; (v) sentence length and structure; (vi) the questions' texts

2 ; (vii) the

possible solutions; (viii) eventually the fixed event.
Each operator studied the story text, reviewed all the games associated with it
and created from scratch the textual part of the causal games 3 .

2

With the aim of unambiguously identifying the event in the story. For instance, the same
event text could be referred to different story episodes.

3

The NLP module was in fact unable to detect any causal relation in the text.