Digital Signal Processing Reference
In-Depth Information
7.5.2 Evaluation Measures
In the following, evaluation criteria for classifiers are considered at first. These will be
followed by such for regressors where a continuous relation between the output of the
learning algorithm and the target needs to be evaluated. In the case of classification,
however, we need to compare discrete predicted class labels and compare these with
the ground truth target. For simplification— without limitation of the general case—
let the rejection class be assumed to be inherently modelled, i.e., rejection is one
of the target classes. By that, we can consider the classification task as a mapping
X
→{
y
.
Evaluation criteria are defined as related to the test set's
1
,...,
C
}
,
x
→ˆ
instances, and the
individual instances are each assigned to exactly one target class
i
T
∈{
1
,...,
C
}
:
C
C
T
=
1
T
i
=
1
{
x
i
,
n
|
n
=
1
,...,
T
i
}
,
(7.85)
i
=
i
=
where
T
i
is the number of instances in the test set that belong to class
i
. By that, the test
set has the size
|
T
|=
i
=
1
T
i
. Note, however, that attempts exist to find evaluation
criteria where several classes may be assigned to one instance. This requirement is
for example given in the case of the classification of a speaker's emotion, where one
is not only 'surprised', but e.g., 'happily surprised' or 'angrily surprised' which led
to the introduction of soft emotion profiles [
51
]. Similarly, music genre or ballroom
dance style are often ambiguous in music analysis, cf. musical pieces that allow for
either Rhumba or Foxtrott as choice of dance.
We will first consider evaluation measures for classification in the general case of
two or more classes (i.e.,
M
2) [
1
]. The most common measure is the probability
that an instance of the test set is classified correctly. This is usually referred to as
(weighted) accuracy
WA
, or weighted average recall or recognition rate.
≥
# correctly classified test instances
# test instances
WA
=
i
=
1
x
i
∈
T
i
|ˆ
y
=
=
.
(7.86)
|
T
|
If this rate is given per class
i
, one speaks of the class-specific recall
RE
i
:
x
i
∈
T
i
|ˆ
y
=
RE
i
=
.
(7.87)
T
i
With
p
i
=
T
i
/
|
T
|
as the prior probability of class
i
in the test set further holds:
M
WA
=
p
i
RE
i
.
(7.88)
i
=
1
