Digital Signal Processing Reference
In-Depth Information
Table 5.4
Mean kappa values over the raters (A-D) for four different calculations of gold standard
obtained either by employing rounded mean or median of the labels per song
# Classes
1
2
Gold Standard
κ
κ
κ
Arousal
5
mean
0.328
0.477
0.634
5
median
0.415
0.518
0.626
3
mean
0.475
0.496
0.533
3
median
0.526
0.545
0.578
Va l e n c e
5
mean
0.307
0.453
0.602
5
median
0.411
0.510
0.604
3
mean
0.440
0.461
0.498
3
median
0.519
0.535
0.561
Fig. 5.5
5 class distrib-
ution of the music database
(2,648 total instances) after
annotation based on rounded
median of all raters [
14
]
5
×
year. The distributions of instances per partition are displayed in Fig.
5.6
following
the three degrees per dimension.
To reveal the impact of limiting to musical pieces with clear agreement by a
majority of raters, one can additionally consider the sets Min2/4 for the case of
agreement of two out of four raters, while the other two have to disagree among each
other, resembling unity among two and draw between the others, and the set Min3/4,
where three out of four raters have to agree. Note that the minimum agreement is
based on the original five degrees per dimension and that this sub-set is only used
for the testing instances, to keep training conditions fixed for better transparency of
effects of such 'prototypisation'. The numbers of instances per degree of agreement
are shown in
5.7
.
For 1 937 of 2 648 songs in the NTWICM database lyrics can automatically be
collected from two on-line databases: In a first run LyricsDB
6
was applied, which
6
LyricsDB
(
http://lyrics.mirkforce.net
)
