A Formal Framework for Design and Validation of Multimodal Interactive Systems in Transport Domain - Human-Computer Interactions in Transport - page 92

This gave birth to works that are concerned with the formal modeling and verification
of multimodal interactive systems.
In this chapter, we will try and address the formal design of multimodal HCI. In
section 4.2 we will present the main concepts linked with multimodality. In section 4.3
we will succinctly expose the principles of design and formal verification approaches.
Sections 4.4 and 4.5 are dedicated to input and output multimodality, respectively, and
we will present the main modeling and multimodality works, as well as the formal
models of input and output that have been proposed. For each formal model we define
the syntax and the semantics, then illustrate the model via a multimodal HCI referring
to the world of transport.
4.2. Concepts of multimodality
Before addressing the different representations of modeling and multimodal
HCI validation techniques, it is advisable to define the terminology relative to
multimodality that is necessary for understanding the approaches proposed. In the
literature [NIG 96], there are definitions that differ according to the point of view
(user, technology) or the level of abstraction. We will try and clarify these concepts
by presenting the definitions considered below:
Mode: state or manner in which an action is carried out, and that determines its
interpretation when it is made by the user (vocal, direct handling) or system
(visual, auditory and tactile).
Media: physical device in a computing system enabling the transmission of
information (human/machine) that has the role of a sensor (microphone,
camera, keyboard or mouse) or effecter (screen or loudspeaker).
Modality: refers to the structure of information (command) during the interaction,
when it is introduced by the user: speaking, gesture, click; or when it is
perceived by the user: text, audible warning or vibration.
Statement: command that calls (at input) or information that restitutes an elementary
function of the functional core (at output).
Interactive task: complex or elementary informational exchange between user and
machine in input (in view of realizing a goal) or in output (in view of presenting
information).
Input multimodality: type of interaction that consists of fusing information provided
by the user via several modalities. For example, the interaction in which the user
expresses the phrase “ put that there ” by clicking on an object an then on a site of
the interface, constitutes a multimodal input interaction by using the information
respectively introduced via speech modalities via microphone media, and direct
handling via the mouse media.