Multimodal Dialogue System Assessment - Man-Machine Dialogue: Design and Challenges - page 169

introduces the principle of questioning the system about the point to be
assessed, with the advantage of thus displacing the focus of the assessment
from the data to the question, and thus not focusing on the answers or
reactions of the system (black box method, which does not require exploring
the system's internal structures, but lacks precision), nor on the system's
semantic structures (transparent box method, which is precise and easily
leads to a diagnostic but requires to have reference semantic representations).
The system still needs to be able to answer the questions Q of DQR. The
adapted paradigm Demand-Control-Response- Result-Reference (DCR)
[ANT 99], minimizes this issue by replacing the question with a control that
is a simplification or a rephrasing of the initial user query. The Peace
paradigm (automatic understanding assessment paradigm: “Paradigme
d'Evaluation Automatique de ComprEhension”), see the chapter by L.
Devillers et al. in [GAR 02], which contributes to the original idea of
modeling the dialogue history with a paraphrase, allows us to stay within the
black box model while allowing for an assessment of context understanding.
10.1.4. Multimodal dialogue methodologies
In the multimodal MMD context, propositions are far from being as
relevant. The Promise paradigm, Procedure for Multimodal Interactive
System Evaluation [BER 02], is presented as an extension of Paradise to
multimodality, with principles to assign scores to multimodal inputs and
outputs. The proposition in fact remains at a very approximate level, well
below the variety of multimodal phenomena. The interesting aspects of the
article concern the oral dialogue, with considerations on the level of task
completion and the level of user cooperation. The works of N.O. Bernsen and
L. Dybkjær, which are still references in the field of multimodal dialogue, are
rather disappointing when it comes to assessment. Bernsen and Dybkjær,
[BER 04] present a methodology meant for a system, with a focus on the
method of the questionnaire subsequently filled out by users. The reason
given is indeed that the other methods are not yet well established.
Unfortunately, the questions in the questionnaire remain within a very
superficial level when it comes to multimodality: “did you use the mouse or
point to the screen?” “how was making the gestures?” “would you like to be
able to do more with gesture? If yes, what?” The answers provided by the
users also strike us as very poor in content, especially because one of the
authors' conclusions is that the user preferred talking rather than using
the multimodal possibilities. As for Dybkjær et al. [DYB 04], it is more a