The Situated Multimodal Facets of Human Communication - Coverbal Synchrony in Human-Machine Interaction - page 194

Communication is an emergent behavior and its success strongly
depends on “readiness to act” and “prompting of plans” in order to
appropriately handle the incoming communicative event, producing
suitable mental states, feelings, and expressions, under conditions of
limited time and resources.
In humans, emotions cannot be separated from communication,
and communicative signals that bring meanings to the interactional
instance also bring emotional contents. I have observed interactions
in different languages, only among male or only female, or both, and
I was always amazed by the amount of emotional content portrayed
by the interactants through their body poses, facial, vocal, and gestural
expressions, as well as head movements and gaze directions.
Communication is not neutral, it cannot be, since neutrality would
not allow humans to rapidly and continuously adapt to the dynamical
context of the communicative instance and the dynamic changes that
it produces on our knowledge acquisition and use.
Communication is an embodied process that transforms the
information and the meanings gathered in the course of everyday
activities and builds up from them new knowledge and practical ability
to render the world interpretable while interacting with others attuned
to behavioral sequences that underpin collaboration.
Communication is a multimodal, multifunctional, multi-determined
phenomenon and its components iterate its complexity. Therefore, in
order to understand the multimodal facets of communication it is
necessary to approach it holistically, including the emotional states
that are an integrant part of such a process.
To do so, multiple theoretical investments are needed, ranging
from mathematical models of interaction and dynamics of signal
exchanges (in terms of shared meanings, emotional states, cultural
differences, social, organizational and physical context effects), to
social intelligence, behavioral analyses, action selection, and cognitive
processes of cooperation, decision making and resource management.
The benefits of such an approach are threefold:
THEORETICAL: A holistic approach will allow the development of
new mathematical models for representing data, reasoning, learning,
planning, and decision making, as well as new individual/group
behavior analysis models of social interaction in multilingual and
cross-cultural contexts. This will produce new socio-psychological and
computational approaches to model interaction, starting from existing
cognitive frameworks such as the emotion model by Ortony et al. (1990)
and from existing algorithmic solutions such as dynamic Bayesian