Multimodal Dialogue System Assessment - Man-Machine Dialogue: Design and Challenges - page 179

emotions? In general, did the system take your expressiveness into account?
3) Questions on output management: what did you think of the way in
which the answers were presented (orally, graphically or both)? Did the avatar
seem natural? When the avatar generated a gesture, did it seem relevant?
4) Questions on the relationships between input and output managements:
did you feel there was a coherence between your messages and the messages
generated by the system? Did the expressiveness appear better in input or
output?
As we can see, it is not hard to specify a set of questions covering
multimodality management. They seem to us at least to be more precise than
those by Bernsen and Dybkjær [BER 04] and should avoid overly vague
answers. They cannot make us forget that nothing can replace the spontaneous
writing of a text describing the subject's impressions, especially before any
question can be asked. Indeed, a question directs the subject's attention to the
mentioned topic and can thus introduce a bias in relation to his/her initial
impressions. Even the order of the questions can have an influence on the
answers. The specification of the questionnaire should thus be done under the
direction of a psychologist, which was not the case for Paradise, Peace or this
one, which is one of the challenges of this methodological aspect.
10.3.3. Extending DQR and DCR to multimodal dialogue
J. Zeiliger et al. in [MAR 00] have elected a black box type of methodology
that allows them to carry out a system diagnostic, a methodology that relies
on generic tests to assess the understanding of an isolated utterance. The
contextual aspects have been neglected (we will return to that with Peace), but
it was the price to pay for achieving a simple and well-defined methodology.
The principle is to carry out occasional assessments, each of them focused
on a specific phenomenon. Thus, in the DQR materialization, the occasional
assessment becomes a question Q which the system asks and it allows us
to check that it has understood the original request D properly. One of the
examples given concerns anaphora resolution, with a request, question and
subsequent answer:
- D = “take the first street on the right and follow it for 300 yards” (initial
utterance as it was given to the system to help the task progress);