The Semantics of Search - Social Semantics: The Search for Meaning on the Web - page 178

relevance feedback help a fair amount in improving Semantic Web search using
hypertext results, and as relevance judgments can be approximated by click-through
logs of hypertext Web search engines, it is realistic and feasible to try to improve
semantic search using relevance feedback from hypertext search. In fact, it is simple
to implement pseudo-feedback from hypertext Web search using hypertext search
engine APIs, as no manual relevance judgments must be made at all and the API
simply can produce the top 10 results of any query quickly.

6.9 Future Work on Relevance Feedback

There are a number of areas where our project needs to be more thoroughly
integrated with other approaches and improved. The expected criticism of this work
is likely the choice of FALCON-S and Yahoo! Web search as a baseline, and that we
should try this methodology over other Semantic Web search engines and hypertext
Web search engines. Lastly, currently it is unknown how to combine traditional
word-based techniques from information retrieval with structural techniques from
the Semantic Web, and while our experiment with using inference as document
expansion did not succeed, a more subtle approach may prove fruitful. At this
point, we are currently pursuing this in context of creating a standardized evaluation
framework for all Semantic search engines. The evaluation framework presented
here has led to the first systematic evaluation of Semantic Web search at the
Semantic Search 2010 workshop (Blanco et al. 2011a). Yet in our opinion the most
exciting work is to be done as regards scaling our approach to work with live large-
scale hypertext Web search engines.
While language models, particularly generative models like relevance mod-
els (Lavrenko 2008), should have theoretically higher performance than vector-
space models, the reason why large-scale search engines do not in general imple-
ment language models for information retrieval is that the computational complexity
of calculating distributions over billions of documents does not scale. However,
there is reason to believe that relevance models could be scaled to work with Web
search if they built their language sample from a suitably large 'clean' sample of
natural language and also compressed the models by various means.
One of the looming deficits of our system is that for a substantial amount of
our queries there are no relevant Semantic Web URIs with accessible RDF data.
This amount is estimated to be 34% of all queries. However, these queries with
no Semantic Web URIs in general do have relevant information on the hypertext
Web, if not the Semantic Web. The automatic generation of Semantic Web triples
from natural language text could be used in combination with our system to create
automatically generated Semantic Web data, in response to user queries.
Another issue is how to determine judgments for relevance in a manner that
scales to actual search engine use. Manual feedback, while providing the more
accurate experimental set-up for testing relevance feedback, does not work in
real search scenarios because users do not exhaustively select results based on