Information Technology Reference
In-Depth Information
2
(3)
E
= −
1
1 1
+
R P
where recall has the most effect. The harmonic F measure (Shaw 1986) is essentially a complement of
the E measure ,
2
1 1
.
(4)
F
=
+
R P
The precision-recall measure and its variants are effective measures of performance of information
retrieval systems in the environment where the total document collection is known and the sub-set of
documents that are relevant to a given query is also known.
The drawbacks of the precision-recall based measures are multi-fold. Most noticeably, as Cooper
pointed in his seminal paper (Cooper 1968), it does not provide a single measure; it assumes a binary
relevant or irrelevant set of documents, failing to provide some gradual order of relevance; it does not
have built-in capability for comparison of system performance with purely random retrieval; and it does
not take into account a crucial variable: the amount of material relevant to the query which the user
actually needs. The expected search length ( ESL ) (Cooper 1968, Korfhage 1997) is a proposed measure
to counter these problems. ESL is the average number of irrelevant documents that must be examined
to retrieve a given number if of relevant documents. The weighted average of the individual expected
search lengths then can be defined as follows,
N
if e
*
if
(5)
if
=
1
ESL
=
N
if
if
=
1
where N is the maximum number of relevant documents, and e if the expected search length for if relevant
documents.
The average search length ( ASL ) (Losee 1998, Losee 1999, Losee 2000) is the expected position of a
relevant document in the ordered list of all documents. For a binary judgment system (i.e. the document
is either relevant or irrelevant), the average search length is represented by the following relation,
ASL = N[QA + (1-Q)(1-A)]
(6)
where N is the total number of documents, Q is the probability that ranking is optimal, and A is the
expected proportion of documents examined in an optimal raking if one examines all the documents
up to the document in the average position of a relevant document. The key idea of ASL is that one can
compute the quality of an IR system without actually measuring it if certain parameters can be learned
in advance. On the other hand, if one examines the retrieved documents, the value A can be determined
 
Search WWH ::




Custom Search