Information Technology Reference
In-Depth Information
Fig. 6.4
The interface used to judge Semantic Web results for relevancy
likelihood of purely coincidental agreement by the judges. Fleiss'
both corrects
for chance agreement and can be used for more than two judges (Fleiss 1971).
The null hypothesis is that the judges cannot distinguish relevant from irrelevant
results, and so are judging results randomly. Overall, for both relevance judgments
over Semantic Web results and web-page results,
κ
κ
=
.
<
.
0
5724 (
p
0
05, 95%
[
.
,
.
]
Confidence interval
), indicating the rejection of the null hypothesis
and 'moderate' agreement. For web-page results only,
0
5678
0
5771
κ
=
.
<
.
0
5216 (
p
0
05,
[
.
,
.
]
95% Confidence interval
), also indicating the rejection of the
null hypothesis and 'moderate' agreement. Lastly, for only Semantic Web results,
κ
=
0
5150
0
5282
.
<
.
[
.
,
.
]
), also indicating
the null hypothesis is to be rejected and 'moderate' agreement. So, in all cases there
is 'moderate' agreement, which is sufficient given the general difficulty of producing
perfectly reliable relevancy judgments. Interestingly enough, the difference in
0
5925 (
p
0
05, 95% Confidence interval
0
5859
0
5991
κ
between the web-page results and Semantic Web results show that the judges were
actually
slightly
more reliable in their relevancy judgments of information from
the Semantic Web rather than the hypertext Web. This is likely due to the more
widely varying nature of the hypertext results, as compared to the more consistent
informational nature of Semantic Web results.
Were judges more reliable with entities or concepts? Recalculating the
κ
for all
κ
=
.
<
.
results based on entity queries,
0
5989 (
p
0
05, 95% Confidence interval
[
.
,
.
]
κ
=
.
0
5923
0
6055
), while for all results based on concept queries was
0
5447
<
.
[
.
,
.
]
(
p
). So it appears that judges are
slightly more reliable discovering information about entities rather than concepts,
backing the claim made by Hayes and Halpin that there is more agreement in
general about 'less' abstract things like people and places rather than abstract
concepts (Hayes and Halpin 2008). However, agreement is still very similar and
'moderate' for both information about entities and concepts. It is perhaps due to the
entity-centric and concept-centric definition of relevance that the agreement was not
higher.
0
05, 95% Confidence interval
0
5381
0
5512
Search WWH ::
Custom Search