Multimodal Dialogue System Assessment - Man-Machine Dialogue: Design and Challenges - page 174

graphical user interface based on windows, menus, icons and buttons.
Accessing the objects displayed on the screen is indeed the basis of both of
them. A comparative assessment procedure should, therefore, include
graphical user interfaces and multimodal systems. The efficiency, speed and
precision aspects become the many measurements that allow us to compare a
multimodal system with a graphical user interface for the same task (not all of
the multimodal systems, however, are made after a first graphical version).
A questionnaire was produced for the people undergoing a user test, an
assessment to diagnose, make a global assessment, segmented assessment,
assessment based on the system's reaction or comparing its internal
representations with reference representations. Most of the methods
suggested for oral dialogue appear to be applicable to multimodal dialogue,
even if a certain amount of care should be taken when implementing them.
Among the approaches that strike us as very hard to apply to the
multimodality are the comparative assessment and the passage on a corpus.
Even if the latter is useful to test or train, it remains hard to imagine how to
reuse a ready-made multimodal corpus with annotations and which could be
adapted to the assessment of a system for which it has not been designed. We
can, however, hope for future development in multimodal corpora, especially
if the synergy we can observe emerging between the corpus-assessment
campaign pair, such as the synergy we can observe in Evalda/Media [DEV
04], reaches multimodality.
Finally, the impressions that come from this are that the MMD domain and
the multimodal dialogue, do not lend themselves well to the other domains of
NLP assessment. Methodology that is precise, trustworthy, objective,
complete, independent of system, modalities and tasks is still an unattainable
Grail. The acronyms of the methodologies proposed echo this impression:
Paradise, Peace, Promise, etc. Luckily, the issues that have arisen and the
questions asked, and the aspects processed, even if they do not lead to a
unanimous assessment methodology, greatly contribute to the improvement of
implementation processes and of dialogue system test processes.
10.3. Methodological elements
Among the methodological aspects we have mentioned, we consider a few
here, which are as many elements for the methodology of the assessment of