Semiotic Design and Evaluation Framework - Cross-Cultural Human-Computer Interaction and User Experience Design - page 36

3.1.3 N ARRATION

The narrative in UI is made both by the designer's metacommunication, and the
temporal aspect of perceiving UI elements. The perception is guided through UI
languages. This UI narration runs on two scales: the first one on the current screen
(local), the second one through a whole sequence (global), a chained set of screens
(or window states). If the designer's narration doesn't fully correspond to the user's
understanding of the interaction with the UI, the user should at least be able to follow
it along.
By reading the visible language of a screen we follow a certain path on screen ac-
cording to what we want to accomplish, and thus read the seemingly helpful elements
first. By interpreting the signs present on a given screen, we order them in a temporal
manner in a sequence. This sequence takes on the form of a narration/storytelling
layer of the UI. We call it the local narration.
The local narration is supported mostly by the appearance, metaphors, and mental
model component of the UI. The local narration is constituted mainly by noun-phrases
or descriptive sentences corresponding to a film shot (“Here is a revolver”). Because
of this, the local narration is based on a static state of the screen, and on the UI compo-
nents, that does not represent an action (as in the case of the interaction component).
Here, this type of narration is articulated by the visible language and can form an
argument or a statement.
Such narration is supported by the order of perception or appearance of the UI
element and a goal (e.g., a motivation to search for information). The order of per-
ception can be subject to the reading direction, size, shape, or color that would attract
the eye to certain points first and thus creating a sequence (see Sutnar, 1961 for his
psychology-based information design). The relation between elements on a single
screen can be emphasized by distance, same/similar color, or line (for a detailed
explanation see Bertin, 2011).
The sequence of appearance of an UI element can be controlled better by the
designer, because it is not so much subject to the user's perception and interpretation.
Moreover, as Eisenstein shows by the Kuleshov effect (Eisenstein, 1975; see also
Metz, 1974), the viewer inadvertently relates two scenes coming one after the other
and puts them into a narrative sequence. As Kress and Van Leeuwen (2006) shows
us combinations of disconnected comics panels, or film shots that lead to different
interpretation of the narrative in different representational systems: “Should we see
such a [ ] narrative process as one unit of meaning or two? Is it the equivalent of a

sentence like 'The soldier shoots the villagers,' or of a formulation that expresses the
soldier's agency less directly—for instance, 'The soldier fires. The villagers are shot'?
But such an attempt at translating moving images into words cannot fully capture the
difference. Filmic 'disconnection' has no parallel in language. It does have semiotic
potential, however” (Ibid., p. 259).
The narration takes also the form of a progressively disclosing argumentation of
the designer. When we act on an object on the screen, we are presented with a dynamic
argument set forth by the designer (see Bogost, 2007). This argument is not presented
on a single screen, as was the case of the visible language, but unfolds through
multiple screens (globally). We call this type of narration a global narration. The
global narration is supported by the navigation and interaction component running

...