Multimodal Dialogue System Assessment - Man-Machine Dialogue: Design and Challenges - page 178

you think of his behavior on the screen?” “what was bad about your interaction
with the system?” “what was good about your interaction with the system?”
etc. [BER 04]. We can quickly add more questions to these questions, which
focus on modalities and multimodality:
- Can you understand the gestures generated? Did the system understand
your gestures? What did you think of the force-feedback's operation?
- Were the generated messages correctly created? In a synchronous
manner? With no inconsistency or gap? Did the system correctly link your
gestures to what you said? Did it understand your entire messages?
The general questions of the Paradise paradigm can also quickly be
extended to multimodality with the same principle, which consists of
answering with a satisfaction index between 1 and 5:
- the system was easy to understand (oral and multimodal aspects);
- the system understood what I said (we can replace say by generate);
- I got the information I asked for (always valid);
- the pace of interaction with the system was appropriate (always valid,
but very important in multimodal settings since it includes potential problems
of temporal synchronization between the different modalities - in other words,
the answer is not interpreted in the same way depending on whether the system
is oral or multimodal);
- I knew what I could say at each point (we can replace say by generate);
- the system clearly explained what it had understood (always valid);
- the system's questions or suggestions helped me (always valid).
We can also suggest some examples of questions more or less focused on
multimodal phenomena spread over four categories:
1) Questions focusing on interaction conditions: did you feel free when
you were talking? Did you generate gestures? When did you use the force
feedback device? Did the system correctly manage the devices at its disposal?
Did you feel constrained in their use? And in that case, which of the questions
constrained you?
2) Questions on input processing: did the system understand your way
of communicating with him properly? Did it appear to be sensitive to your