Search Engine Performance Comparisons - Distributed Artificial Intelligence, Agent Technology, and Collaborative Applications - page 154

Table 2. Various single-value measures applied to the experiment data

AV MARS

Type 1 3.78 0.014

Type 2 32.7 25.7

ESL Type 3 124 113

Type 4 7.56 0.708

Type 5 25.7 17.3

Measured 82.2 77.6

ASL

Estimate 29.8 29.8

RankPower 3.29 2.53

Revised Rank Power 0.34 0.36

is. Revised RankPower has values between 0 and 1 with 0 being the least favorite and 1 being the most
favorite.
We can draw the following observations from the data shown in Table 2. Note that these observations
demonstrate the effectiveness of single-value measures, especially, the RankPower . The focus was not
on the comparison of the actual search engines since the experimental data is a few years old.

1. In ESL Type 1 comparison, AltaVista has a value of 3.78 which means on the average, one needs
to go through 3.78 irrelevant documents before finding a relevant document. In contrast, ESL Type
1 value for MARS is only 0.014 which means a relevant document can almost always be found at
the beginning of the list. MARS performs much better in this comparison because of its relevance
feedback feature.
2. ESL Type 2 counts the number of irrelevant documents that a user has to go through if she wants
to find six relevant documents. AltaVista has a value of 32.7 while MARS has a value of 25.7.
Again because of the relevance feedback feature of MARS, it performs better than AltaVista.
3. It is very interesting to analyze the results for ESL Type 3 request. ESL Type 3 request measures
the number of irrelevant documents a user has to go through if she wants to find all relevant docu-
ments in a fixed document set. In our experiments, the document set is the 200 returned documents
for a given query and the result is averaged over the 72 queries used in the study. Although the
average number of relevant documents is the same between AltaVista and MARS (see the values
of estimated ASL) because of the way MARS works, the positions of these relevant documents are
different. This results in different values of ESL Type 3. In order to find all relevant documents
in the return set which the average value is 29.8 documents, AltaVista would have to examine a
total of 124 irrelevant documents while MARS would examine 113 irrelevant documents because
MARS have arranged more relevant documents to the beginning of the set.