Information Technology Reference
In-Depth Information
•
Facial modality
: analysis of the gaze, head movements, smiles and laughs, facial
expressions, and mouth movements (in our case, the analysis is integrated by
“facial action coding system” (FACS) analysis);
•
Corporeal modality
: analysis of the posture, torso and leg movements, orientation
in relation to interlocutors, and movement of the body in space.
All the productive modalities are used simultaneously and synchronized with
speech, each one offering a precise semantic contribution with different goals. In
fact, when senders deliver the verbal contents of their discourse, conveying the
information at their disposal to the world, they communicate at the same time
through the other modalities the purposes of their communication, the thoughts
and feelings they have about their specific message (Poggi
2007
). The addressee,
furthermore, observing the body of the speaker while they listen to their words, in a
few seconds perceptively acquires or infers specific knowledge about the sex, age,
ethnic and cultural background, and personality of the sender. This information is
often communicated by senders against their own will. This information, deduced
by recipients through observation of the speaker's body, interacts with the effects of
the speaker's strategies of self-presentation (Goffman
1967
), that is, with the image
that the speaker wishes to give of him- or herself, consciously producing signals
or monitoring them while speaking, to induce a type of specific perception of the
speaker in recipients.
Describing speech as a sort of “communicative symphony,” Poggi (
2007
)
proposes the score of multimodal communication as the best tool to transcribe,
analyze, and classify the joint action of several signals transmitted in the different
modalities present in a communicative fragment.
In the score, the five aforementioned modalities are written and analyzed on
parallel lines as in a musical score. Thus, for each signal of each modality, analysis
takes place on five levels:
•
DS
is the description of the signal; it relates to the physical characteristics
of movement, gestures, gaze, posture and vocal elements in terms of length,
intensity, fundamental frequency, and pauses for prosodic-intonative signals.
•
Ts
is the type of signal; each signal may be classified, for instance, batonlike
gesture, deictic gaze, iconic gesture (Ekman et al.
1984
). The gestures can be,
for instance,
deictic
, when they point to an object or person with the index
finger or open hand;
iconic
, when they draw in the air a form or imitate the
typical movements of an object, animal, or person;
symbolic
, when a gesture in a
cultural moment has a meaning easily translatable into words or phrases. Lastly,
the
batonlike
gesture, where the hands go from up to down to emphasize what is
spoken; batonlike gestures are often unconscious.
•
S
is the meaning of each signal, its verbal translation.
•
TS
is the type of meaning; each signal is classified as information about the world
(Imo), information about the identity (IIM) of the sender, or the mind (IMM) of
the sender.
•
F
is the function, the semantic relation between the signal being analyzed and the
concomitant verbal signal or another signal taken as a point of reference and that
Search WWH ::
Custom Search