A Framework for Studying Human Multimodal Communication - Coverbal Synchrony in Human-Machine Interaction - page 23

Table 2. Sensory modalities and production modalities in multimodal communication.

Sensory Modalities Production Modalities

Sight

Communicative body movements/gestures/writing

Hearing Voice, speech

Touch Touch

Smell Smell

Taste Taste

lexicon, syntax, semantics and pragmatics) combined with prosody
and communicative body movements (Kendon, 2004; Argyle, 1988;
Allwood, 2008a).
Multimodality implies multimediality, since the sensory modalities
also involve physical media, i.e. optical (sight), acoustic energy
(hearing), pressure (touch) as well as molecules affecting taste and
smell. In addition, there are other perceptual modalities than the
traditional five senses that may be relevant, e.g. modalities for
temperature, color, shape, movement, etc.
Studying communication from a multimodal perspective leads
to a deeper understanding of many processes connected with
communication. Two of these are:
(i)

Multimodal integration (sometimes, with a metaphor from
physics, also called information fusion). Multimodal integration
concerns how we can integrate information from our separate
sensory modalities with our memory sources to form a common
complex multimodal experience. For instance, in a normal
conversation, we integrate what we see (colors, shapes and
movements), what we hear, touch and smell with what we
epistemically and emotionally experience. Furthermore, this is
all integrated with other, already stored information we have in
our memory concerning our interlocutor and about what he/she
is saying or doing.

(ii) Multimodal distribution (with another metaphor from physics,

this is sometimes called information fission). Multimodal
distribution concerns how we distribute what we want to
communicate or do, using several production modalities. If I
want to tell you that I am happy to see you, my message will
be distributed into a vocal verbal aspect, a prosodic (intonation,
tone of voice) aspect, a gestural aspect (face, head, arms, torso,
etc.) and possibly a touch and smell aspect. How exactly these
different aspects are related to each other is one of the questions
still to be resolved in the study of multimodal communication.