Database Reference
In-Depth Information
0.3
0.25
0.2
0.15
0.1
ED_linear
ED_cobic
AP
RP
NDCG
P10
0.05
0
0.2
0.25
0.3
0.35
0.4
0.45
0.5
0.55
0.6
Differentiaation rate
Fig. 3.
Average error rates of the TREC 2001 in the condition of fixed differentiation
rates
Form this experiment, we confirm that the Euclidean distance with the linear
model is the best among all the metrics involved. However, this time the Eu-
clidean distance with the cubic model becomes the worst. Those ranking based
metrics are close. However, in TREC 9, NDCG is better than the three others;
in TREC 2001, P10 is better than the three others. If considering the average of
the two year groups, the order from best to worst is P10, NDCG, RP, and AP.
It is exactly the same as that in the above experiment.
4 Conclusions
From our experiments, we demonstrate that the Euclidean distance with the
linear model is a very good metric. It has the ability of keeping good balance
between sensitivity and reliability. On the other hand, its cousin, the Euclidean
distance with the cubic model, does not seem to be as good as it. It suggests
that the Euclidean distance can be a good metric if relevance scores are properly
estimated. In addition, the linear model is easy to apply and does not need any
training data as the cubic model. As a matter of fact, it can be used in exactly
the same way as those ranking based metrics. We believe it is an attractive
option for retrieval evaluation. However, for the Euclidean distance, there is one
problem: though the cubic model is a more sophisticated model than the linear
model, why the linear model is better than the cubic model ? This is not clear
and remains to be our future investigation problem.
