Information Technology Reference
In-Depth Information
5.4.2 Measure-Specific Loss
The other sub-category of the listwise approach optimizes a measure-specific loss
function. Therefore, the discussion on the relationship between such a loss function
and the corresponding evaluation measure will be more straightforward, since they
have natural connections. For example, some algorithms are explicitly designed as
optimizing an upper bound of the measure-based ranking error. However, also be-
cause of this, one may not be satisfied with the relationship of an “upper bound”
and want to know more about these algorithms. For this purpose, a new quantity
named “tendency correlation” is introduced in [
5
]. Basically, the tendency correla-
tion measures the relationship between a surrogate measure (e.g., SoftNDCG and
AppNDCG) and the corresponding evaluation measure. Note that in the definition,
for simplicity, we assume the use of a linear scoring function, i.e.,
f
w
T
x
.
=
Definition 5.1
(
ε
Tendency Correlation) Given an evaluation measure
M
and a sur-
rogate measure
M
, the documents
x
associated with a query, and their ground-truth
labels
y
, for two ranking models
w
1
and
w
2
, denote
β
M(w
1
;
x
,
y
)
−
M(w
2
;
ε(β,w
1
,w
2
,
x
,
y
)
=
x
,
y
)
x
,
y
)
,β
−
M(w
1
;
x
,
y
)
−
M(w
2
;
≥
0
.
M
and
M
is defined as
Then the tendency correlation between
ε
=
min
β
w
1
,w
2
,
x
,
y
ε(β,w
1
,w
2
,
x
,
y
),
max
M
has
ε
tendency correlation with
M
.
and we say
When
ε
is small, we say that the surrogate measure has a strong tendency cor-
relation with the evaluation measure. According to [
5
], the tendency correlation as
defined above has the following properties.
1. The tendency correlation is invariant to the linear transformation of the surrogate
measure. In other words, suppose a surrogate measure has
ε
tendency correla-
tion with the evaluation measure, and we transform the surrogate measure by
means of translation and/or scaling, the new measure we obtain will also have
ε
tendency correlation with the evaluation measure.
2. If
ε
is zero, the surrogate measure will have the same shape as the evaluation
measure (i.e., their only differences lie in a linear transformation). If
ε
is very
small, the surrogate measure will have a very similar shape to the evaluation
measure.
3. It has been theoretically justified that when
ε
approaches zero, the optimum
of the surrogate measure will converge to the optimum of the evaluation mea-
sure [
5
].
One may have noticed that the definition of tendency correlation reflects the
worst-case disagreement between an evaluation measure and a surrogate measure.
