The Situated Multimodal Facets of Human Communication - Coverbal Synchrony in Human-Machine Interaction - page 193

depending on the native language (Italian and American English)
of the subjects involved and the cultural context (American English
and Italian movies) associated to the stimuli. For example, for the
Italian subjects tested on the Italian stimuli, the combined audio/
video and the audio alone conditions conveyed the same amount of
emotional information (F 1,8 = .004, ρ = .95), while such information was
significantly different from the mute video and the combined audio/
video (F 1,8 = 10.414, ρ = .01) as well as from the mute video and the
audio alone (F 1,8 = 13.858, ρ = .005) conditions. This was not the case
for the American English subjects tested on the American English
stimuli, where the combined audio/video and the mute video, as well
as the mute video and the audio alone conditions conveyed the same
amount of emotional information (F 1,8 = 2.323, ρ = .16; (F 1,8 = 1.696, ρ =
.22) respectively) whereas such information was significantly different
for the audio alone and the combined audio/video conditions (F 1,8 =
9.031, ρ = .01).
The data discussed above suggest that in decoding emotional
states, humans do not sum up linearly the cues provided. The
combined audio/video may convey the same amount of emotional
information of the mute video or the audio alone, depending on the
language, the expressed emotion and the cultural specific background.
The visual channel may facilitate under certain conditions a cross-
cultural decoding of emotional states but this is not always the case
and cannot be applied to all emotional categories. For example, the
decoding of irony is strictly language and cultural dependent. It has
been suggested that the process to decode emotional information is
driven by a “reducing cognitive load” strategy (Esposito, 2007, 2009)
where in an attempt to reduce the amount of emotional cues to be
processed for the task, humans select the cues that are more close to
their personal/cultural competences, to their daily experience, and
more congruent with the context of the contemporary interactional
instance. Support to these speculations comes from literature on non-
human animals where it has been shown that direct experience of male
rhesus macaques with specific females allows them to interpret their
facial signals more accurately (Higham et al., 2011). Unfortunately, to
our knowledge, there are no data on human animals that report on
similar experiments and this is surely another research aspect to be
accounted for as a situated multimodal facet of communication.
6. Discussion
How can verbal and nonverbal emotional expressions be connected
with the situated multimodal facets of human communication?