Information Technology Reference
In-Depth Information
metrics are based on the Levenshtein Distance [72]. This calculates the minimal
distance between two strings by examining the minimal number of operations that
is required to transform one string into the other. An operation can be an insertion,
a deletion or a substitution of one single character.
The character error rate is calculated as follows:
+
+
I
S
D
CER
=
∗
100
N
where
•
I is the number of inserted characters
•
S is the number of substituted characters
•
D is the number of deleted characters
•
N is the maximum number of characters of both text strings
The word error rate (
WER
) uses the same formula, in which however the param-
eters
I
,
S
and
D
refer to operations on entire
words
and
N
is the maximum number
of
words
.
Baseline Performance
As shown in Table 5.2, the resulting word and character error rates elevated at
49.2 % and 18.6 % respectively. These error rates are even higher than the (already
unsatisfactory) rates reported in the literature: Koile et al. [65] used the same engine
and reported a word error rate of 27 % on handwritten answers that students wrote
on Tablet PCs as a response to specific questions of the instructor.
Our goal was to find out why in our case the error rate is that elevated. For this
reason, we manually analyzed all 679 annotations that were made by 10 users in
this lecture. The analysis showed that these annotations are much more complex to
recognize than handwritten notes or answers. In contrast to answers that are submit-
ted to the teacher, annotations serve a personal use. They have an informal character
and are often written in a hurry. The annotations heavily varied in size, position and
orientation. Even within one single annotation, the size of the characters can be very
different (Fig. 5.14). Moreover, a considerable number of annotations (29 % of our
test set) contained mixed text and drawings (Fig. 5.15). This is also due to to the
topic of the lecture - sorting algorithms - in which students sketched many tree-like
structures. Finally, annotations contain many (often personal) abbreviations as well
as domain-specific terms (like
O
(
nlogn
)
)orformulae.
Tabl e 5. 2
Baseline performance
of the handwriting recognition of lecture annotations
Percentage
Word error rate
49.2 %
Character error rate
18.6 %
