Semiotic Foundations of HCI and UX Design - Cross-Cultural Human-Computer Interaction and User Experience Design - page 28

(1966) there are two principal languages involved in HCI: “One is the language of
display by which the machine presents information regarding the state of its data
and the options available for further action by the user. The other is the language of
actions using input devices, by which the man relates his intended transformations of
machine-stored data with references to objects in the displayed picture” (cited from
Foley and Wallace, 1974, p. 465). We shall explain them in the following sections.

2.4.1 V ISIBLE LANGUAGE

The language of display is in most of the cases ruled by a visible grammar, although
different grammars for different perceptions (such as auditory, tactile, oral, olfactory)
can still be used. In our case we focus primarily on the interaction grammar, as well
as the visible grammar, that starts the interaction by providing a visual narrative
that could be followed. The language of display corresponds therefore to a visible
language.
By visible language we mean a systematic language of expression conveying
specific information, that can be translated from one kind of language to another
(Marcus, 2003b; see also Moles, 1966). The visual language, on the other hand,
refers to loose visual means for conveying emotions or general concept, as used by
artists. Currently, several authors analyze visible languages (Roam, 2011; Kress and
van Leeuwen, 2006; Engelhardt, 2002; Horn, 1998; Narayanan and Hubscher, 1998;
McCloud, 1994; Bertin, 2011) which could help us methodologically in the area
of interaction and communication design. The visible language represents numbers,
nouns, and verbs (forming a narration) (see Tufte, 1983, 1991, 1997).

2.4.2 I NTERACTION LANGUAGE

The language of actions is based on user input, and it thus corresponds to the inter-
action language. Based on our decision to act (or refrain from acting) on an object,
we start an interaction that modifies the current screen. The interaction language is
based on a grammar, that connects all of the semiotic dimensions mentioned earlier
in Chapter 2, “Semiotic Foundations .”

Let us remind Peirce's definition of semiotics related to syntax: “Semeiotic gram-
mar is concerned with determining the formal conditions for signs as such (CP 1.444)”
(cited from Liszka, 1996, p. 18). The grammar sets rules on what can be chained in
a cause-result interaction unit, how it can be chained, and on what purpose. Because
of these rules, the grammar of interaction can establish what can be expected based
on the metaphor, mental model, navigation, interaction, and appearance in use on the
working screen. Therefore, the interaction grammar can support the consistency of
all the UI components.
Consistency helps to build expectations, which in turn establishes a set of con-
straints for the interaction and communication design. Constraints should be used, as
Laurel (1993, p. 105) suggests: “without shrinking our perceived range of freedom of
action: Constraints should limit, not what we can do, but what we are likely to think
of doing.” For Norman (2002) these constraints are physical, semantic, cultural, and
logical. The physical constraints “are closely related to real affordances: For example,

...