Information Technology Reference
In-Depth Information
Table 11.8
Winning number of each algorithm
Algorithm
NDCG@1
NDCG@3
NDCG@10
P@1
P@3
P@10
MAP
Regression
4
4
4
5
5
5
4
RankSVM
21
22
22
21
22
22
24
RankBoost
18
22
22
17
22
23
19
FRank
18
19
18
18
17
23
15
List
et
29
31
33
30
32
35
33
AdaRank
26
25
26
23
22
16
27
SVM
map
23
24
22
25
20
17
25
Table 11.9
Results on the MQ2007 dataset
Algorithm
NDCG@1
NDCG@3
NDCG@10
P@1
P@3
P@10
MAP
RankSVM
0.410
0.406
0.444
0.475
0.432
0.383
0.464
RankBoost
0.413
0.407
0.446
0.482
0.435
0.386
0.466
ListNet
0.400
0.409
0.444
0.464
0.433
0.380
0.465
AdaRank
0.382
0.398
0.434
0.439
0.423
0.374
0.458
4. In terms of MAP, the listwise ranking algorithms are in general better than the
pairwise ranking algorithms. Furthermore, the variance among the three pairwise
ranking algorithms in terms of MAP is much larger than that in terms of other
measures (e.g., P@1, 3, and 10). The possible explanation is that since MAP
involves all the documents associated with a query in the evaluation process, it
can better differentiate algorithms.
To summarize, the experimental results indicate that the listwise algorithms have
certain advantages over other algorithms, especially for the top positions of the rank-
ing result.
11.3 Experimental Results on LETOR 4.0
As for the datasets in LETOR 4.0, not many baselines have been tested. The only
available results are about Ranking SVM, RankBoost, ListNet, and AdaRank, as
shown in Tables
11.9
and
11.10
.
From the experimental results, we can find that the differences between the
algorithms under investigation are not very large. For example, Ranking SVM,
RankBoost, and ListNet perform similarly and a little bit better than AdaRank on
MQ2007. RankBoost, ListNet, and AdaRank perform similarly on MQ 2008, and
Ranking SVM performs a little bit worse. These observations are not quite in accor-
dance with the results obtained on LETOR 3.0.
The possible explanation of the above experimental results is as follows. As we
know, the datasets in LETOR 4.0 have more queries than the datasets in LETOR 3.0;
