Touch and Speech: Multimodal Interaction for Elderly Persons - Biomedical Engineering Systems and Technologies - page 367

context-sensitive multimodal interaction. An Input Manager receives and interprets all
incoming messages from GUI Action Recognizer for GUI inputs, Speech Recognizer
for natural language understanding and Sensing Manager for other sensor data. An
Output Manager on the other hand, handles all outgoing commands and distributes
them to View Presenter for visual feedbacks, Speech Synthesizer to generate natural
language responses and Action Actuator to perform necessary motor actions. Know-
ledge Manager uses Database to keep the static data of certain environments and Con-
text to process the dynamic information exchanged with users during the interaction.
Although the essential components of MIGSEP are closely connected with each
other via predefined XML-based communication mechanism, each of them is treated
as an open black box and can be implemented or extended for specific use, without
affecting other MIGSEP components. It provides a general platform for both theoreti-
cal researches and empirical studies on multimodal interaction.

4.2 The Unified Dialogue Model in MIGSEP

The current unified dialogue model (UDM) consists of four extended state transition
diagrams.
Each interaction is initiated with the diagram Dialogue(S, U) (cf. Fig. 5 (left)), by
the initialization of the system's start state and a greeting-like request.

Fig. 5. The Initiate diagram and the transition diagram triggered by user

The dialogue continues with user's instruction to a certain location, request for a
certain information or restart action, leading to the system's further response or dialo-
gue restart, respectively, as well as updating the information state with the attached
update rules (cf. Dialogue(U, S) in Fig. 5 (right)).
After receiving user's input, the system tries to generate an appropriate response
according to its current knowledge base and information state (cf. Response(S, U) in
Fig. 6 (left)). This can be informing the user with requested data, rejecting an unac-
ceptable request with or without certain reasons, providing choices for multiple
options, or asking for further confirmation of taking a critical action, each of which
triggers transitions to different diagrams.
Finally, the user can accept or reject the system's response, or even ignore it by
simply providing new instructions or requests, triggering further state transitions as
well as information state updates (cf. Response(U, S) in Fig. 6 (right)).
Using the FormDia toolkit, the UDM was developed as CSP specifications, and its
functional properties have been validated and verified via FDR, as well as its concep-
tual interaction process using FormDia simulator. The tested specification was then