Multimodal Dialogue System Assessment - Man-Machine Dialogue: Design and Challenges - page 177

- Do users of the same level have the same multimodal behavior and do
they have the same problems using the system?
- Is a user of a higher level more efficient, quicker and more precise in
its use of multimodality? Does he/she cover a wider set of phenomena? Does
he/she exploit more functionalities of the system?
Finally, if we come to managing a level of expertise not for the whole
multimodal system but only in the multimodal aspects of the interaction,
another problem arises: that of the legitimacy of an independent measure of
the system which would thus be valid for any multimodal system. This level
of multimodal expertise could be based on the use of one or more multimodal
systems and would require taking into account an indicator of the complexity
of each system. Taking up the previous analogy, 100 h of flight experience on
a small tourist plane is not the same as 100 h on a fighter plane, even if both
types of planes have similar points when it comes to the way they are piloted.
The weight of the time with a system complexity index is a possible solution.
This complexity index should depend on the following aspects: the number of
words of vocabulary in the system, the number of recognized syntactic
constructions, the number of processed gesture types, the number of speech
acts, the number of devices (terminals and screens), the number of languages
identified, the number of possible exchanges and potentially the number of
primitives to the application task. We find the set of multimodal system
functionalities, and this makes the implementation of an absolute index hard.
A solution thus consists of choosing a reference system and calculating, as a
ratio, the gap between this system and the system considered.
10.3.2. Questionnaires for users
The questionnaire is the main source of subjective information. It helps
broaden the scope of the assessment and carry out a diagnostic, not only of
the system but also of the user training content. We will consider here the
questionnaires offered for oral dialogue and complete them by adding
multimodal preoccupations.
Some general questions can be taken up from the questionnaires by
N.O. Bernsen and L. Dybkjær. They are essentially questions on the user's
impressions, the system's perceived use and the possible ways to improve it:
“was it easy or difficult to use the system? Why?” “could you understand what
he said?” “could he follow what you wanted to talk to him about?” “what do