The Semantics of Search - Social Semantics: The Search for Meaning on the Web - page 176

Tabl e 6. 3 Table comparing
hypertext-based relevance
feedback and FALCON-S

Results Feedback FALCON-S

Top relevant 118 (89%) 76 (58%)

Non-relevant top 14 (11%) 56 (42%)

Non-relevant top entity 9 (64%) 23 (41%)

Non-relevant concept 5 (36%) 33 (59%)

While performance is boosted for both entities and concepts, the main improve-
ment comes from concept queries. Indeed, as concept queries are often one word
and ambiguous, not to mention the case where the name of a concept has been taken
over by some company, music band, or product, it should not be surprising that
results for concept queries are considerably boosted by relevance feedback. Results
for entity queries are also boosted. A quick inspection of the results reveals that the
entity queries were the most troublesome, and that these entity queries gave both
FALCON-S and our feedback system problems. These problematic queries were
mainly very difficult queries where a number of Semantic Web documents all share
similar natural language content. An example would be a query for 'sonny and cher,'
which results in a number of distinct Semantic Web URIs: one for Cher , another one
for Sonny and Cher the band, and another for 'The Sonny Side of Cher,' an album by
Cher. For concepts, one difficult concept was the query 'rock.' Although the system
was able to disambiguate the musical sense from the geological sense, there was
a large cluster of Semantic Web URIs for rock music, ranging from Hard Rock to
Rock Music to Alternative Rock . These types of queries seem to present the most
difficulties for Semantic Web search engines.
Although less impressive than the results for using hypertext web-pages for
relevance feedback for the Semantic Web, the feedback cycle from the Semantic
Web to hypertext does improve significantly the results of even commercial
hypertext web-engines, at least for our set of queries about concepts and entities.
Given the unlimited API-based access offered by Yahoo! Web Search in comparison
to Google and Microsoft web search, we used Yahoo! Web Search for hypertext
searching in this experiment, and we expect that the results in a coarse-grained
manner should generalize to other Web search engines. The hypertext results for our
experiment were given by Yahoo! Web Search, and we calculated a mean average
precision for Yahoo! Web Search to be 0.4039. This is slightly less than our baseline
language model ranking, which had an average precision of 0.4284. As shown in
Fig. 6.15 , given that our feedback-based system had an average precision of 0.6549,
our relevance feedback performs significantly ( p

< 0 . 05) better than Yahoo! Web

Search and ( p < 0 . 05) the baseline rm system.

6.8.2 Discussion

These results show our relevance feedback method works significantly better than
various baselines, both internal baselines and state of the art commercial hypertext
search engines and Semantic Web search engines. The parametrization of the precise