Multimodal Fusion in Human-Agent Dialogue - Coverbal Synchrony in Human-Machine Interaction - page 390

objects and specifies the intended referent by verbally providing a
category that distinguishes it from alternatives.
Cooperation by transfer means that a chunk of information
produced by one modality is used by another modality. Transfer is
typically used in hypermedia interfaces when a mouse click triggers
the display of an image.
Different modalities may also be used concurrently , i.e. produce
independent chunks of information, i.e. chunks without any semantic
overlap, at the same time. For example, a user may say “Hello” and
at the same time point to an object. Here the chunks of information
should not be merged. Earlier systems usually did not allow for
a concurrent use of modalities, but required an exclusive use of
modalities. For example, the user may utter a greeting and point to
an object, but not at the same time.
While the relationship between modalities has mainly been
discussed for multimodal user commands, little attempts have been
made to specify the relationship between social signals. However,
modalities that convey social signals may cooperate in a similar
manner as modalities that convey semantic information. For example,
different dimensions of emotions, such as valence and arousal, may be
expressed by different channels of communication, such as the face or
the voice. It is important to note, however, that it is hard to deliberately
employ given channels of communication for the expression of social
signals.
Different modalities do not always convey congruent pieces
of information. In the case of semantic information, little robust
input processing components typically lead to incongruent pieces
of information. In the case of social signals, incongruent pieces of
information often result from the fact that users are not equally
expressive in all modalities. In particular, the attempt to conceal social
signals may result into an inconsistent behavior.
Another classification concerns the timing of modalities. Here,
we may basically distinguish between the sequential use of modalities
and the parallel use of modalities which overlap in time. Semantically
related modalities may overlap in time or may be used in sequence.
If they are merged, the temporal distance should, however, not be too
large. Algorithms for the fusion of social signals usually start from
the assumption that social signals that refer to particular user state,
such as frustration, emerge at exactly the same time interval. We will
later see that such an assumption may be problematic.
Other researchers use similar terms to describe relationships
between modalities. See Lalanne et al. (2009) for an overview.