Mystery Number One - Computational Linguistics and Talking Robots - page 20

The two agents, represented as boxes, are concrete individuals with bodies 8
existing in the real world. The external surface of the word water is shown as
a letter sequence between the two agents (modality of vision).
Such an external surface has neither a meaning attached to it nor any gram-
matical properties. It is simply an external object with a particular shape in a
certain modality, which may be measured and described with the methods of
the natural sciences, e.g., formant structure. The shape is arbitrary in the sense
that it does not matter whether the agents use the surface water or the surface
eau as long as they obey the conventions of their common natural language.
In their minds, in contrast, both agents contain the word water as analyzed in
2.1.1, i.e., the surface, the meaning, and the convention-based connection be-
tween the surface and the meaning. As purely cognitive representations (e.g.,
binary code), the analyzed words are independent of any modality, 9 but include
the meanings and all the grammatical properties.
The speaker realizes the external surface by copying the internal word sur-
face and performing it externally in the modality of choice ( copy-realize ). The
hearer recognizes the external surface by matching it with a learned surface
and by using the recognized surface for lexical lookup ( match-lookup). 10
Even though the internal representations of surfaces and meanings are moda-
lity-free 11 in principle, they may relate indirectly to the modality of the exter-
nal interface by which they have been recognized or realized. During recogni-
tion an internal surface matches a modality-dependent representation such as
sound, and during production it is realized in such a modality.
In this indirect sense, the internal surfaces are usually mono-modal 12 while
the meanings are multi-modal. For example, the surface of the word raspberry
is mono-modal in that the associated modality is purely acoustic, optical, or
tactile (Braille), but does not include taste. The meaning of this word, in con-
trast, may be viewed as relating to a multi-modal conglomerate including taste.
8 The importance of agents with a real body (instead of virtual agents) has been emphasized by Emer-
gentism (MacWhinney 1999, 2008).
9 To illustrate the agent-internal representation of the word water in 2.2.1, we had to resort to a suitable
modality, namely vision. Though unavoidable, this is paradoxical insofar as the words in the cognition
of an agent are inherently modality-free.
10 One may recognize a surface as a real word, yet not know its meaning, for example “reprobate,”
“exigency,” “prevaricate,” or “effloresce” in English. In this case, there is an entry missing in the
hearer's lexicon. This may hold even more when recognizing a surface-like shape as a word form of
a foreign language.
11 Barsalou (1999) calls modality-free modality-general and modality-dependent modality-specific .
12 A multi-modal representation of language surfaces may be found in an opera performance in which
the text sung on stage (acoustic modality) is also displayed more or less simultaneously in writing
above the stage (optical modality). A more mundane example is an English movie with English
subtitles.