Adaptive Information Filtering - Text Mining: Classification, Clustering, and Applications - page 192

TABLE 8.1: The values assigned to
relevant and non-relevant documents that the
filtering system did and did not deliver. R − ,
R + , N + ,and N − correspond to the number
of documents that fall into the corresponding
category. A R ,A N ,B R ,and B N correspond to
the credit/penalty for each element in the
category.

Relevant Non-Relevant

Delivered R + ,A R N + ,A N

Not Delivered R − ,B R N − ,B N

much in common, especially if we treat a topic as a form of user information
need. Since a separate chapter in this topic is devoted to TDT, we refer the
readers to that chapter for research on TDT.
This chapter is organized as follows. Section 8.2 introduces the standard
evaluation measures used in the TREC adaptive filtering task. Section 8.3
introduces commonly used retrieval models and adaptive filtering approaches.
Section 8.4 describes how to solve the “cold start” problem for new users using
Bayesian prior learned from other users. Section 8.5 introduces techniques to
avoid redundant information while filtering. This chapter ends with discussion
and references to other important topics not covered in details in this topic.

8.2 Standard Evaluation Measures
In the information retrieval community, the performance of an ad hoc re-
trieval system is typically evaluated using relevance-based recall and precision
at a certain cut-off of the ranked result. Taking a 20-document cut-off as an
example:

precision = the number of relevant documents among the top 20
20

(8.1)

recall = the number of relevant documents in the top 20
all relevant documents in the corpus

(8.2)

What is a good cut off number is unknown. In order to compare different
algorithms without a specific cut off, the mean of the precision scores after
each relevant document is retrieved, which is called Mean Average Precision
(MAP), is often used.
However, the above evaluation measures are not appropriate for filtering.
Instead of a ranking list, a filtering system makes an explicit binary decision