The Semantics of Search - Social Semantics: The Search for Meaning on the Web - page 175

Fig. 6.14 Summary of best average precision scores: relevance feedback from hypertext to
Semantic Web

We used the original ranking of the top 10 results given by FALCON-S to
calculate its average precision, 0.6985. We then compared both the best baseline,
rm , as well as the best system with feedback in Fig. 6.14 . As shown, our system
with feedback had significantly ( p 05) better average precision (0.8611) than

FALCON-S (0.6985), as well better ( p

< 0 .

05) than the 'best' language model
baseline without feedback (0.5043) as reported earlier as given in Sect. 6.5.1.1 .
Average precision does not have an intuitive interpretation, besides the simple
fact that a system with better average precision will in general deliver more accurate
results closer to the top. In particular, one scenario we are interested in is having
only the most relevant RDF data accessible from a single URI returned as the top
result, so that this result is easily consumed by some program. For example, given
the search 'amnesia nightclub,' a program should be able to consume RDF returned
from the Semantic Web to produce with high reliability a single map and opening
times for a particular nightclub in Ibiza in the limited screen space of the browser,
instead of trying to display structured data for every nightclub called 'amnesia'
in the entire world. In Table 6.2 , we show that for a significant minority of URIs
(42%), FALCON-S returned a non-relevant Semantic Web URI as the top result.
Our feedback system achieves an average precision gain of 16% over FALCON-S.
While a 16% gain in average precision may not seem huge, in reality the effect is
quite dramatic, in particular as regards boosting relevant URIs to the top rank. So in
Tab le 6.3 , we present results of how our best parameters tf with m

< 0 .

000 lead
to the most relevant Semantic data in the top result. In particular, notice that 89%
of resolved queries now have relevant data at the top position, as opposed to 58%
without feedback. This would result in a noticeable gain in performance for users,
which we would argue allows Semantic Web data to be retrieved with high-enough
accuracy for actual deployment.

= 10 ,