Mining Diagnostic Text Reports by Learning to Annotate Knowledge Roles - Natural Language Processing and Text Mining - page 57

the POS tags were often erroneously induced (in the sentence with only 8 words
of Figure 4.9 there are 3 such errors—CARD tags for 2 nouns and 1 adjective),
which then resulted in erroneous parse trees. But, in those cases when the tag-
ging was performed correctly, the parse trees were also correct. Still, the parser
could not parse long sentences, perhaps due to the fact that it was trained in
the part of the Negra corpus with sentences having up to 10 words. Trying the
parser with long English sentences instead, produced excellent results. We con-
cluded that at this phase of implementation, the Stanford parser could not be
used with our corpus of German sentences that contain an average of up to 18
words per sentence.
BitPar Parser - This parser is composed of two parts, the parser itself [27] and
the parameter files (chart rules, lexicon, etc.) from [25]. Published experimental
results claim robust performance, due to the use of sophisticated annotation
and transformation schemata for modeling grammars. Another advantage of
the parser is that its lexicon can be extended very easily with triples of domain-
dependent words, their tags, their frequency counts in a corpus, thus avoiding
the tagging errors typical for unlexicalised parsers. These tagging errors damage
the parse results, as can be seen from the results of the Stanford parser. Our
critique for the described BitPar is that it usually produces trees with more
nodes than the other parsers and the annotation of nodes contains specialized
linguistic information, not very appropriate for creating features for learning.
Sleepy Parser - This parser has been specifically tuned for the German language,
and while it is a statistical parser like the others, it uses different annota-
tion schemas and incorporates grammatical functions (SB-subject, OC-clausal
object, MO-modifier, HD-head, etc.) or long-distance dependencies between
terms. In constrast to the two other parsers, it also has a highly tuned su x
analyzer for guessing POS tags [8], which contributes to more accurate tagging
results than the other parsers, although some domain-dependent words are not
always correctly tagged. Erroneous parsing is also encountered for very long
sentences.
Choosing a Parser
All the tested parsers make errors during parsing. In the end, the criteria upon which
we based our choice of the parser were speed and output information. Sleepy was
the fastest and had the most informative output (it prints the log value expressing
the likelihood of parsing, and it labels the majority of nodes with their grammatical
function). Actually, choosing a parser upon these criteria instead of the accuracy
of parsing could be regarded as inappropriate. Our justification is that a metric to
measure the accuracy of parsing on new data does not exist. These parsers have all
been trained on the same corpus, and at least the two German parsers tuned up
to the point where their results are almost the same. Thus, a priori their expected
accuracy in a new corpus should be equal, and accuracy is not a criterion for choosing
one over the other. Given the di culty of evaluating the accuracy of the parse trees
and their presumed similarity, we based the choice of parser on the qualities that
contributed most to our task, namely speed and informative output.