Mining Diagnostic Text Reports by Learning to Annotate Knowledge Roles - Natural Language Processing and Text Mining - page 63

Table 4.4. Statistics for the benchmark datasets.

Subcorpus Cases No. Sentences No. Unique Sentences No. Annotated Roles No.

Isolation Current 491 1134 585 1862

Wedging System 453 775 602 1751

Table 4.5. Learning results for the benchmark datasets.

Subcorpus Recall Precision F β =1 measure

Isolation Current 0.913 0.934 0.92

Wedging System 0.838 0.882 0.86

4.5.2 Active Learning versus Uniform Random Selection

In order to evaluate the advantages of active learning, we compared it to the uni-
form random selection of sentences for manual annotations. Some results for both
approaches are summarized in Table 4.6 and Table 4.7. Recall, precision, and F β =1
measure were calculated after each iteration, in which 10 new sentences manually
labeled were added to the training set. The results of active learning (F β =1 measure)
are 5-10 points better than those of random learning. For this experiment, the step
d) of the active learning strategy was not applied, since it is very specific to our
corpus.

Table 4.6. Random Learning Results.

Sentences No. Recall Precision F β =1 measure

10 0.508 0.678 0.581

20 0.601 0.801 0.687

30 0.708 0.832 0.765

40 0.749 0.832 0.788

Table 4.7. Active Learning Results.

Sentences No. Recall Precision F β =1 measure

10 0.616 0.802 0.697

20 0.717 0.896 0.797

30 0.743 0.907 0.817

40 0.803 0.906 0.851