Information Technology Reference
In-Depth Information
outperform the pointwise ranking algorithms. These results are in accordance with
the discussions in this topic.
Potential problems with the aforementioned experimental results are as follows.
As pointed out in Chap. 11, these experimental results are still primal and by
carefully tuning the optimization process, the performance of every algorithm
can be further improved.
The scale of the datasets is relatively small. One may obtain more convincing
experimental results if the algorithms are compared using a much larger dataset.
The area of learning to rank will benefit a lot from the release of such large-
scale datasets. It is good news that the newly released Yahoo! learning-to-rank
challenge datasets and Microsoft learning-to-rank datasets both contain tens of
thousands of queries. We can foresee that with the release of these large datasets,
a new wave of learning-to-rank research will emerge.
(3) Theoretically speaking, is ranking a new machine learning problem, or can
it be simply reduced to existing machine learning problems? What are the unique
theoretical issues for ranking that should be investigated?
According to the discussions in Part VI, we can clearly see that it is better to
regard ranking as a new machine learning problem, rather than reducing it to ex-
isting problems. As compared to classification and regression, the evaluation of a
ranking model is performed at the query level and is position based. Mathematically
speaking, the output space of ranking contains permutations of documents, but not
individual documents, and there is a hierarchical structure (i.e., query-document)
in the learning process. Therefore, the “true loss” for ranking should consider the
positional information in the ranking result, but not as simple as the 0-1 loss in
classification. The generalization in ranking should be concerned with both the in-
creasing number of training queries and that of training documents. In this regard, a
novel theoretical framework is sorely needed to perform formal analysis on ranking.
(4) Are there many remaining issues regarding learning to rank to study in the
future?
This is not a simple question to be answered in one or two sentences. We will
present more discussions on the future work on learning to rank in the next chapter.
References
1. Burges, C.J., Ragno, R., Le, Q.V.: Learning to rank with nonsmooth cost functions. In: Ad-
vances in Neural Information Processing Systems 19 (NIPS 2006), pp. 395-402 (2007)
2. Burges, C.J., Shaked, T., Renshaw, E., Lazier, A., Deeds, M., Hamilton, N., Hullender, G.:
Learning to rank using gradient descent. In: Proceedings of the 22nd International Conference
on Machine Learning (ICML 2005), pp. 89-96 (2005)
3. Cao, Y., Xu, J., Liu, T.Y., Li, H., Huang, Y., Hon, H.W.: Adapting ranking SVM to docu-
ment retrieval. In: Proceedings of the 29th Annual International ACM SIGIR Conference on
Research and Development in Information Retrieval (SIGIR 2006), pp. 186-193 (2006)
Search WWH ::




Custom Search