Search Engine Performance Comparisons - Distributed Artificial Intelligence, Agent Technology, and Collaborative Applications - page 149

information about numerous FTP sites and provides a searchable interface so users can easily retrieve
files through different FTP sites. Gopher provides search tools to large number of Gopher servers on
the Internet. WAIS has similar functionality to that of Archie, except that it concentrated on wide
variety of information on the Internet, not just FTP sites. With the fast development of Web, search
engines designed just for the Web started to emerge. Some of the examples include WWWW (World
Wide Web Worm), then-most-powerful search engine AltaVista, NorthernLight, WebCrawler, Excite,
InforSeek, HotBot, AskJeeves, AlltheWeb, MSNSearch, and of course, Google. Some of these search
engines disappeared in history; others were re-tooled, re-designed, or simply merged; yet others have
been able to stay at the front against all the competition. Google since its inception in 1998 has been
the most popular search engine mostly because of its early success in its core algorithm for search, the
PageRank algorithm (Brin & Page, 1998). Search engines today are generally capable of searching not
only free text, but also structured information such as databases, as well as multi-media such as audio
and video. Some of the representative work can be found in (Datta et.al ., 2008) and (Kherfi et.al ., 2004),
More recently some academic search engines start to focus on indexing deeper Web and producing
knowledge based on the information available on the Web, e.g., the KnowItAll project by Etzioni and his
team, see for example (Banko et.al , . 2007). In a relatively short history, many aspects of search engines
including software, hardware, management, investment and others have been researched and advanced.
Microsoft, though a later comer in the Web search business, tried very hard to compete with Google
and other leading search engines. As a result, Microsoft unveiled its own search engine on November
11th, 2004 with its Web site at http://beta.search.msn.com (Sherman, 2005). We refer to it as MSN
in this discussion. The beta version of the search has since evolved to what is now called Live search
engine (http://www.live.com). This chapter reports the results of a project attempting to assess the
performance of the Microsoft search engine when it was in its beta version from various perspectives.
Specifically the study collects statistics such as the average user response time, average process time
for a query reported by MSE itself, the number of pages relevant to a query, the quality of the search in
terms of RankPower , and comparisons with its competitors. The rest of the chapter is organized as fol-
lows: Section 2 provides an overview of search engine performance metrics. The goals and the metrics
of this study are described in Section 3. Section 4 discusses the method of study and the experimental
settings, followed by the results and their analysis in Section 5. Our thoughts and conclusions about the
study are presented in Section 6.
Perfor Mance Metrics for we B search engines
While user perception is important in measuring the retrieval performance of search engines, quanti-
tative analyses provide more “scientific evidence” that a particular search engine is “better” than the
other. Traditional measures of recall and precision (Baeza-Yates 1999) work well for laboratory stud-
ies of information retrieval systems. However, they do not capture the performance essence of today's
Web information systems for three basic reasons. One reason for this problem lies in the importance
of the rank of retrieved documents in Web search systems. A user of Web search engines would not go
through the list of hundreds and thousands of results. A user typically goes through a few pages of a
few tens of results. The recall and precision measures do not explicitly present the ranks of retrieved
documents. A relevant document could be listed as the first or the last in the collection. They mean the
same as far as recall and precision are concerned at a given recall value. The second reason that recall