Semiotic Design and Evaluation Framework - Cross-Cultural Human-Computer Interaction and User Experience Design - page 32

Lexemes and graphemes could be used to capture the user's interaction, but for the
actual analysis we should work with higher-level elements. For example, Richards
(1984) believes that “there seems to be little profit in using such items as an individual
dot or line as a unit of analysis. If we are going to use linguistics as a model, then
what is needed for present purposes is not the pictorial equivalent of a phoneme or
morpheme but something closer to a noun phrase” (cited from Engelhardt, 2007,
p. 27). However, some of these elements are useful to provide additional information,
such as color. In such a case “signifiers—and colours are signifiers, not signs—carry
a set of affordances from which sign-makers and interpreters select according to their
communicative needs and interests in a given context. [ ] First there is association,

or provenance—the question of 'where the colour comes from,' where we have seen
it before.' [

...

] The second type of affordance is that of the 'distinctive features' of
colour” (Kress and Van Leeuwen, 2006, pp. 232-233).
In the interaction syntax one can identify “lexemes,” or interaction “words” present
in an interaction dictionary, (Hjelmslev, 1961, cited from Andersen, 1997, p. 256),
such as mouse-up, mouse-down, mouse-drag. A “word” in tangible UIs, or TUIs, can
be “touch.” One or more lexemes form lexical items, or actions, like “pointing with a
mouse” (Payne, 1991, p. 136), which build up a vocabulary of interaction language.
The standardized vocabulary (lexical items) can be UI widgets (we refer to the
primitives, like links, buttons, icons, pop-up controls, etc., but not to more complex
forms, like Google widgets or Apple gadgets, which are UIs in themselves). These
basic elements can consist of texts, pictures, sounds, animations, etc. To exclude
too much complexity the widgets must fall in the category of noun-phrase (e.g.,
selected radio-button), or verb-phrase (e.g., type of action, which could/should be
performed).
Identifying the discrete elements can help us assess the economy of elements used.
It is important to be able to “form an infinite number of meaningful combinations using
a small number of low-level units (offering economy and power). The infinite use of
finite elements is a feature, which in relation to media in general has been referred to
as 'semiotic economy”' (Chandler, 2001). In interaction and communication design
we should seek to minimize the elements used, thus maximizing the potentional
economical gain in clarity and simplicity of the UI.
The lexical items, in turn, construct an interaction “phrase.” The interaction phrase
consists of noun phrases (involving the subject and object) and/or verb phrases (in-
volving a verb, which provides more information about the subject). The interaction
phrases together form an interaction sentence.

...

3.1.2 I NTERACTION SENTENCES

The interaction sentence is a meaningful unit describing a task in the user's interaction.
For example, in a drill-down menu, the user selects the category he or she needs to
modify, then selects the parameters to set, and then presses the “OK” button. The visual
hierarchy promotes a certain reading (and interpretation) sequence of the UI elements.
The interaction sentence is subject to the visual elements, which, through visible
grammar, can lead the user to act. Each user-driven event in the UI is constituted by (a)
an intention, and (b) an action. The intention is manifested by mouse-over (pointing),