A Framework for Studying Human Multimodal Communication - Coverbal Synchrony in Human-Machine Interaction - page 24

In many contexts, flexibility in the choice of modality is needed.
Multimodality gives us this flexibility, and also the possibility of
being redundant when this is needed, for example, in a complex noisy
environment.
Building on the definition of communication given above and
building on the notion of multimodality, we can now give a definition
of multimodal communication.
Multimodal communication = co-activation, sharing and co-
construction of information simultaneously and sequentially through
several modes of perception (and production) (Allwood, 2008a).
In Table 3, we give an overview of how dimensions of production
and perception can be related in multimodal communication.
The combination of dimensions can be simultaneous or sequential,
occur on varying levels of consciousness and intentionality and involve
several communicative orientations (see below).
As we can see from the table, communication can involve many
types of communicative expressions over and above the auditory aspects
of speech. Table 4 gives an overview (Allwood, 2002).
Most of these expressions can supplement auditory aspects of
speech or play an autonomous role in communication.

Table 3. Multimodal face-to-face communication—Perception and production.

PERCEPTION Hearing Vision Touch Smell Taste

Understanding + Attitudinal Reactions

PRODUCTION
SPEECH:
Prosody/Phonology x x

Vocabulary x

Grammar x

GESTURES:
Facial gestures x

Manual gestures x

Body movements x

Posture x

Touch x

Smell x

Taste x

Medium acoustics optics physiology molecules molecules