The Semantics of Search - Social Semantics: The Search for Meaning on the Web - page 153

Tabl e 6. 1 Ten selected
entity and concept queries

Entity Concept

Ashville North Carolina Sociology

Harry Potter Clutch

Orlando Florida Telephone

Ellis college Ale

University of Phoenix Pillar

Keith urban Sequoia

Carolina Aster

El Salvador Bedroom

San Antonio Tent

Earl May Cinch

Web documents about concepts, for a total of 2,000 Semantic Web documents
for relevance judgments. Then, the same experimental query log was used to
retrieve pages from the hypertext Web using Yahoo! Web search, resulting in the
same number of web-pages about concepts and entities (2,000 total) for relevance
judgments. The total number of all Semantic Web documents and hypertext web-
pages gathered from the queries is 4,000.
The queries about entities and concepts are spread across quite diverse domains,
ranging from entities about both locations (El Salvador) and people (both fictional
such as Harry Potter and non-fictional such as Earl May) to concepts ranging over a
large degree of abstraction, from sociology to ale. A random selection of 10 queries
from the entity and concept queries is given in Table 6.1 . This set of 4,000 hypertext
web-pages and Semantic Web documents are then used to evaluate our results in
Sect. 6.5 .

6.2.3 Relevance Judgments

For each of the 200 queries selected in Sect. 6.2.2 , 10 hypertext web-pages and 10
Semantic Web documents need to be judged for relevance by three human judges,
leading to a total of 12,000 judgments for relevance for our entire experiment, with
the correct relevance determined by 'voting' amongst the three judges per document.
Human judges each judged 25 queries presented in a randomized order, and were
given a total of 3 h to judge the entire sample for relevancy. No researchers were
part of the rating. The judges were each presented first with 10 hypertext web-pages
and then with ten Semantic documents that could be about the same query. Before
starting judging, the judges were given instructions and trained on 10 sample results
(five web-pages and five Semantic Web documents). The human judges were forced
to make binary judgments of relevance, so each result must be either relevant or
irrelevant to the query. They were given the web-page selected by the human user
from the query log as a 'gold standard' to determine the meaning of the keyword.