The Situated Multimodal Facets of Human Communication - Coverbal Synchrony in Human-Machine Interaction - page 187

the speaker is transmitting. Assuming that speech and gestures are
employed together in the transmission of information, our research
questions were:

a) Do gestures have any gestural entity equivalent to speech pauses?

b) To what degree do these entities synchronize with speech pauses?

The synchrony of gestures with pausing strategies engages them
in the recovery and lexicalization process needed to formulate the
utterance plan and in the cognitive process undertaken to convey the
added information to the interactional instance. The meaning of this
synchrony will attribute to the conveyed message both temporal and
spatial dimensions, making the utterance a more complex entity than
the simple verbal and phonological phrase. An utterance, as reported in
Kendon (2004), is a “ dynamic visible action ” and gestures as speech are
an expressive resource that can take on different functions depending
on the communicative demand.
To answer the fi rst question, a pilot study (Esposito et al., 2001)
analyzing video-recordings of narrative discourse data identifi ed a
gestural entity named “hold”. A gestural “hold” is detected when the
arms and hands engaged in gesticulation remain still in the gesture
space for at least three video frames (i.e. approximately 100-120 ms)
in whatever position. As defi ned in Esposito et al. (2002), a hold is an
active confi gured gestural state where no intended (hand) motion is
perceived. Note that the absence of movement is judged perceptually
by an expert human coder. Therefore, the concept of hold is ultimately
based on the receiver's perception. A hold may be thought to be
associated with a particular level of discourse abstraction. In producing
a sentence, the speaker may employ a metaphoric gesture (McNeill,
1992) with a hold spanning the entire utterance. The speaker may also
engage in a slight oscillatory motion centered around the original hold,
without any change in hand shape, or add emphatic beats (McNeill,
1992) coinciding with peaks of prosodic emphasis in the utterance.
While the oscillatory motions and emphatic beats may sit atop the
original hold, observers will still perceive the underlying gesture hold.
A careful review of speech and gesture data showed that in fluent
speech contexts, holds appear to be distributed similarly to speech
pauses and overlap with them, suggesting a loose synchronization
between the two entities.
A subsequent study (Esposito et al., 2002) investigated the
robustness of such synchronization looking for it in discourses
elicited by different tasks and spoken in different languages (Italian
and American English in such case).