Information Technology Reference
In-Depth Information
Table 5.
Recall Values for threshold of 20 best terms for Rvar and MI based metrics, and
average recall
Czech
English
Portuguese
P(20)
P(20)
P(20)
LBM Rvar
0.20
0.16
0.13
LM Rvar
0.25
0.12
0.14
LBM MI
0.20
0.16
0.15
LM MI
0.25
0.11
0.13
Average
0.23
0.14
0.14
Table 6.
Kappa statistics-based agreement between the evaluators, for Portuguese and English,
for Tf-Idf and Phi-Square based metrics
Portuguese
English
tfIdf
0.57
0.35
LM tfidf
0.56
0.42
LB tfidf
0.67
0.38
LBM tfidf
0.64
0.40
L ϕ2
0.64
0.46
0.56
0.40
LM
ϕ
2
LBM ϕ2
0.54
0.31
Tables 6 and 7 depict the agreement between evaluators, for Portuguese and
English, by using Kappa statistics. It shows that for Portuguese we have higher levels
of agreement for the Tf-Idf and Phi-Square based metrics. For English agreement
achieved is not so high, but never the less, we consider it acceptable.
Disagreement was mainly due to acceptance of some adjectives as near good
descriptors by one of the evaluators, while the other systematically rejected them in
the sense that adjectives, by themselves, are not good descriptors. This means that,
if the evaluation phase had been preceded by identification of a couple of cases where
the evaluation would be dissimilar, the agreement obtained would have been higher.
Disagreement regarding Rvar and MI based metrics occurred mainly because selected
key terms occurred just once and it was very hard to agree on how such rare terms
could be key terms of those documents. We have not achieved to have the results for
Czech evaluated by two persons. But it should be mentioned that Czech poses yet
another problem when evaluation is at stake, due to its richer morphology. For the
example shown in table 1, one observes that multi-words extracted and ranked are
mostly sub-phrases of multi-word “group on ethics in science and new technologies”
if not of the 11-word term “members of the group on ethics in science and new
technologies”. While for Portuguese and English this has almost no consequences, for
Czech, “skupiny pro etiku ve vědě” is a translation of “of group on ethics in science”
which is not exactly a term. Corresponding term in nominative case would be
“skupina pro etiku ve vědě”. It was accepted as adequate (G) as it also translates as
