Information Technology Reference

In-Depth Information

like regression and classification. Since the output space of ranking is much more

complex than those of regression and classification, and there is a hierarchical

structure in the ranking problem (i.e., query-document), the generalization ability

and statistical consistency in ranking should be revisited.

As a summary, we plot the representative algorithms of the three approaches

introduced in this topic in Fig.
19.1
. Due to space restrictions, we have not put al-

gorithms on data preprocessing, advanced ranking tasks, and theoretical work on

ranking in this figure. From the figure, one can find that learning to rank for infor-

mation retrieval has become hotter and hotter in recent years, and more and more

attention has been paid to the listwise approach.

At the end of this topic, let us come back to the questions we raised in the intro-

duction, and provide their possible answers.

(1)
In what respect are these learning-to-rank algorithms similar and in which

aspects do they differ? What are the strengths and weaknesses of each algorithm?

The answer can be found by the algorithm description and categorization. Ba-

sically algorithms belonging to the pointwise approach reduce ranking to either

regression, classification, or ordinal regression. Algorithms belonging to the pair-

wise approach reduce ranking to pairwise classification. The advantage of these two

approaches is that many existing theories and tools in machine learning can be di-

rectly applied. However, distinct properties of ranking have not been considered in

their formulations. For example, most evaluation measures for ranking are query

level and position based. However, neither the query information nor the positional

information is visible to the loss functions of these two approaches. The listwise

approach instead treats ranking as a new problem, and defines specific algorithms

for it. It can better leverage the concept of
query
, and consider the positional infor-

mation in the ranking results when training the ranking model. The problem with

the listwise approach is that it is in general more complex than the pointwise and

pairwise approaches, and a new theoretical foundation is needed.

According to the analysis in Chap. 5, the loss functions of many learning-to-rank

methods, no matter whether pointwise, pairwise, or listwise, are upper bounds of

(1

−

−

MAP). Therefore, the minimization of these loss functions

can lead to the minimization of (1

NDCG) and (1

−

NDCG) and (1

−

MAP), or the maximization

of NDCG and MAP.

(2)
Empirically speaking, which of those many learning-to-rank algorithms per-

form the best?

According to the discussions in Part IV, the LETOR benchmark datasets have

been widely used in the recent literature of learning to rank. Due to the standard data

collection, feature representation, dataset partitioning, and evaluation tools provided

in LETOR, it is possible to perform fair comparisons among different learning-to-

rank methods. Empirical studies on LETOR have shown that the listwise ranking

algorithms seem to have certain advantages over other algorithms, especially for the

top positions of the ranking results, and the pairwise ranking algorithms seem to