Information Technology Reference
In-Depth Information
Table 1.2
Results for the quotation extraction
Reporting clause
Reported clause
Holder Ve r b
P R F1 P R F1 P R F1
All quotations 0.801 0.649 0.717 0.932 0.728 0.817 0.862 0.821
0.841
Direct quotations 0.791 0.672
0.727
0.914 0.679 0.779 0.89 0.895
0.892
Indirect quotations 0.852 0.596 0.701 0.989 0.815
0.893
0.747 0.782 0.764
Mixed quotations 0.727 0.653 0.688 0.852 0.767 0.807 0.913 0.505 0.65
The direct quotation extraction performs best. Our component achieves an F1-score of 0.89 for the
extraction of the reported clause and an F1-score of 0.73 for assigning a speaker. The extraction of
indirect quotations is more difficult. Still, achieving an F1-score of 0.76 for the extraction of the
reported clause, it produces reasonable results
quotations. In order to measure the performance of our approach we make use of
standard information retrieval measures and compute a token-based recall, precision
and F1-score. We regard all overlapping tokens as true positives. All missing tokens
are regarded as false negatives and all unnecessarily annotated tokens as false pos-
itives. We then calculate the overall performance by summing up all intermediate
results and by calculating final micro-averaged results for the subtasks of holder,
verb, and reported clause extraction.
Table
1.2
summarizes the obtained results. The results meet our expectations. We
achieve the best micro-averaged F1-score of 0.89 for the extraction of direct quotation
parts. With an F1-score of 0.76 our approach for extracting reported speech performs
less effective but still reasonable. Considering quotation holders, the proposed algo-
rithm behaves comparably for all quotation types. It achieves an F1-score of 0.72.
The detection of reporting verbs or clauses introducing a quotation performs quite
well with an F1-score of 0.82. It is striking that the extraction of reported clauses
of mixed quotations is most challenging. Here, our algorithm does not exceed an
F1-score of 0.65.
In order to facilitate a manual evaluation of our extraction approach and also
for its further refinement and improvement, we developed a web demonstrator that
visualizes the results calculated by our component (Fig.
1.5
). The upper field allows
to insert an arbitrary text
16
which our quotation component analyzes subsequently.
The demonstrator shows the identified quotations in the preview window below.
The detected text spans are highlighted in specific colors. At a glance the user sees
whether the extraction was successful or whether the algorithm provides erroneous
annotations.
16
Note that our approach is calibrated on news articles and could produce insufficient markup for
text types other than news articles.
Search WWH ::
Custom Search