Conflict Cues in Call Center Interactions - Conflict and Multimodal Communication: Social Research and Machine Intelligence - page 422

Fig. 20.2 Distribution of emotionally annotated units according to gender ( left ) and speaker role

( right )

Table 20.3 Agreement
results between annotators
within the same language
(GReek and ITalian) and
between the naive annotators
and the expert one

Agreement results
GR1 vs. GR2 Expert vs. GR1 Expert vs. GR2 Average

80.4 % 89.3 % 79.8 % 83.2 %

IT1 vs. IT2 Expert vs. IT1 Expert vs. IT2 Average

40 % 44.6 % 65.30 % 50 %

20.2.2 Perceptual Experiments and Inter-annotator Agreement

To measure inter-annotator agreement, the extracted and annotated units were
assigned to two Greek nonexpert annotators (GR1, GR2) to label them as positive,
negative, or neutral. The “neutral” value was given as an option so as not to bias
the annotators toward the positive or negative direction. The annotators had no prior
knowledge of (a) the context of the discussion and (b) the content of the files the
units were extracted from. The two annotators agreed on the 74 % of speech units,
showing a high agreement with the expert annotator. The out-of-context assessment
was then compared to the expert annotator's assessment, resulting in an agreement
average of 83.2 %, cf. first row of Table 20.3 .
In parallel, driven by works reporting on psychological experiments carrying
out a comparative analysis of subjective perceptions of emotional states (Riviello
et al. 2011 ; Esposito and Riviello 2011 ), a cross-cultural pilot experiment was
carried out to investigate the role of paralinguistic information and language in the
perception of emotional information. Specifically, our goal was to test through a
small-scale experiment the human ability to infer emotional information through
only perceptual cues and how effective it is compared to the knowledge of the
linguistic content.
The experiment involved the assessment of the speech units by two Italian
annotators (IT1, IT2) who do not have any knowledge neither of the Greek language
nor of the context of the discussions. Inter-annotator agreement was measured
among the two Italian annotators as well as between each of them and the expert
annotator (cf. Table 20.3 ).