Multi-Modal Classifier-Fusion for the Recognition of Emotions - Coverbal Synchrony in Human-Machine Interaction - page 79

Figure 3. Depiction of the individual channels in an affectively colored human-computer
interaction. Furthermore, a semi-automatic annotation process is depicted: The labels
encircled in green are set by the human annotator and the system proposes the red labels
additionally.
(Color image of this fi gure appears in the color plate section at the end of the topic.)
be robust against missing data. This can be achieved by temporal
smoothing of the results as seen in Figure 4. There, the blue short
lines represent decisions for the audio channel in word granularity,
the orange dots represent decisions for video frames and the short
green lines are decisions based on physiological signals such as skin
conductance. When a line is drawn in a lighter color, the sample is
rejected due to a low respective confidence. But still a decision for
every time step is returned by exploiting the hypothesis that the lateral
differences over time are low. Further, it is possible to stack multiple
layers of classifiers in order to assess more complex categories based
on simpler observations. When using, for example, statistical models
that can incorporate time series, this architecture can reflect high level
concepts that are not directly observable in the data.
Based on this, we propose a classifi cation architecture, as depicted
in Figure 5, for the recognition of affective states in human-computer
interaction. Here, every individual channel is classifi ed separately with
the usage of an uncertainty measure. Based on this, a sample reject
mechanism is applied in order to prevent false classifi cations. The
subsequent temporal integration is used not only to further improve