Database Reference
In-Depth Information
Table 1.
Pearson's correlation coecients for rankings of retrieval systems in TREC
9 by different metrics (ED(L): Euclidean Distance with the linear model; ED(C):
Euclidean Distance with the cubic model)
Metric AP RP NDCG P10
ED(L) .928 .905
.973
.884
ED(C) .702 .663
.815
.624
AP
.992
.976
.962
RP
.969
.973
NDCG
.941
Table 2.
Pearson's correlation coecients for rankings of retrieval systems in TREC
2001 by different metrics (ED(L): Euclidean Distance with the linear model; ED(C):
Euclidean Distance with the cubic model)
Metric AP RP NDCG P10
ED(L) .872 .799
.894
.696
ED(C) .952 .932
.981
.869
AP
.952
.975
.872
RP
.955
.918
NDCG
.897
Table 3.
Linear regression of different metric values in TREC 9 (dependent variable
is ED(L))
R
2
Metric Constant
Linear
Significance
coecient
level
AP
18.597
-3.313
0.862
.000
RP
18.616
-2.785
0.820
.000
NDCG
18.724
-1.771
0.946
.000
P10
18.593
-3.377
0.782
.000
3.2 Experiment 2
Next we carried out an experiment to compare the overall quality of all metrics.
In [2,8], the stability of several metrics was investigated. Here we took a slightly
different approach, which we think is more reliable. Using a certain metric, we
compare the average effectiveness of two runs
A
and
B
over 50 queries to see if
the difference between them is above a given threshold (
T
). If it is true, or
e
(
A
)
>
e
(
B
)and(
e
(
A
)
−e
(
B
))
/e
(
A
)
>T
, then we look at every query to see how many
of them will contradict the above conclusion, or (
e
(
B
)
− e
(
A
))
/e
(
B
)
>T
.Thus
error rate can be calculated as the percentage of queries which are contradicted
to the overall conclusion. On the other hand, for a given threshold (
T
), we
calculate how many pairs of runs can be distinguished from all possible pairs.
The percentage of these (differentiation rates) is used to represent the sensitivity
of a metric. Since the Euclidean distance and all ranking based measures are very
different, It is not a good idea to use the same thresholds for them. Instead, we
