Multimodal Dialogue System Assessment - Man-Machine Dialogue: Design and Challenges - page 184

away”, “the left one”, “the one which used to be on the right”, “the last one”.
These referring expressions themselves show that the paraphrase of a
multimodal dialogue history is an impossible task, or it would be achieved
only by simplifications such that the bias introduced would take any
plausibility away from the assessment. Indeed, the only paraphrase process
that can be automatized is the systematic use of referent identifiers, and this
solution seems to be more destructive in multimodal dialogue than it is in oral
dialogue: it pushes away all the multimodal aspects. It thus appears hard to
apply the principles of Peace to a multimodal dialogue.
The assessment of multimodal dialogue systems turns out to be more
complex than that of oral systems (which was already quite difficult),
especially when multimodality is considered to be the complementary
association of natural language and other communication modalities on which
language relies. In this chapter, we have suggested a few methodological
building blocks, especially an extension of the DCR and DQR paradigms to
multimodality. Various aspects still have to be studied to achieve a
methodology covering the field naturally occupied by multimodal dialogue.
An aspect concerns one of the paths currently explored to simplify
multimodal system design, that of model-driven engineering (see section 4.2).
In addition, when there will be enough multimodal dialogue systems, we
believe it is useful to return to the challenging assessment method. Its
principle, whether it is the stage of utterance derivation from a set of initial
utterances or the exchange of roles between different designers, does strike us
as relevant to multimodal dialogue.
10.4. Conclusion
The assessment of man-machine dialogue is not characterized by the
efficiency, objectivity or consensus that can be observed in other natural
language processing fields. The systems are designed for a specific task,
which makes any standardized or comparative assessment difficult. Moreover,
technological progress makes many assessment paradigms obsolete and thus
causes them to be multiplied. This chapter synthesizes the existing methods
and suggests a set of reflections around multimodality assessment in systems
with a strong linguistic component.