The Situated Multimodal Facets of Human Communication - Coverbal Synchrony in Human-Machine Interaction - page 175

the overall aim of summarizing and highlighting our work in humans
for the broader multimodal communication community.
2. The Importance of Context
The interaction between our sensory-motor systems and the environment
(which includes people and things) dynamically affects/enhances our
social perception and actions/reactions. In the terminology of cognitive
psychology, this is known as embodiment. Cognitive processes, such as
inference, categorization and memory, are embodied, with individual
choices, perception and actions emerging dynamically from this
interaction. As such, also the processing of human communication
involves embodiment (Esposito et al., 2009). Traditional models of
cognition did not account for such concepts, instead asserting that
cognitive processes are independent of their physical instantiations.
As a consequence, cognition was supposed to be based on amodal
representations and mental operations are performed by a central
processing unit that exploits the sensory (input) and motor (output)
subsystems for collecting and sending representations of the external
world, and executing commands (Block, 1995; Dennett, 1969; Fodor,
1983; Newell and Simon, 1972; Pylyshyn, 1984). Recently, however, new
cognitive models have been proposed, which account for embodied
knowledge acquisition and embodied knowledge use (Barsalou et al.,
2003; Smith and Semin, 2004; Barsalou, 2008), and experimental data
have been provided that support this idea. For example, Schubert
(2004) showed that the visual act of making a fist influenced men's
and women's automatic processing of words related to the concept of
strength. Montgomery et al. (2007) and Kröger et al. (2010) showed that
recognizing and understanding a gesture is wedged in the perceiver's
own motor action inventory.
My own experiments on context aimed to investigate visual context
effects on the perception of musical emotional expressions. The results
of these were partially published in Esposito et al. (2009). A set of
such experiments (unpublished) involved four groups of subjects,
each composed of 38 participants, equally balanced between males
and females and aged from 18 to 30 years. The material consisted of
musical and visual stimuli. There were eight 20-second-long musical
pieces representative of happiness, sadness, fear and anger (two for
each emotion) already assessed as able to arouse the abovementioned
emotional feelings (Esposito and Serio, 2007; Nawrot, 2003). The visual
stimuli consisted of 15 color images, 5 judged to arouse a positive, 5 a
negative, and 5 a neutral feeling according to the emotional dimension