Multimodal Dialogue System Assessment - Man-Machine Dialogue: Design and Challenges - page 163

MMD assessment is thus reduced to a very limited area, which does mitigate
its scope somewhat without questioning its use.
Second observation: soon there might be as many assessment
methodologies as systems themselves. This is not an issue per se, but it does
lead to questions. We can, in particular, wonder about the validity of an
assessment method proposed by the designers of a system, which is only
meant to assess this single system, and the method itself is assessed by its
application to the system in question. This description might appear
excessive, but it does reflect a certain reality, or at least get close to it. This
situation cannot be avoided due to the small number of systems and, faced
with each system's progress, to the need to take into account aspects that
existing assessment methodologies do not deal with. Thus, we end up
proposing or extending an assessment methodology to be able to assess the
progress of a new system. The speed of the technological progress only
increases this phenomenon. MMD assessment thus seems to always be a step
behind its goal.
Third observation: the assessment is used not only to improve a particular
system's development (using measurements, diagnostics and satisfaction
questionnaires) but also to compare one system with another. Several
campaigns have been started, and what has emerged is that it is very hard to
compare several dialogue systems, even if they have been designed for
comparable application contexts, for example train time or hotel information,
which are two widely used applications. In this aspect, the MMD domain
seems to give rise to a more delicate issue than the other NLP domains and
contributes to the fragile image that is attached to its assessment.
Faced with these observations, we can wonder how feasible it is to assess
MMD. With this goal in mind, this section goes over the main issues and a
few methodologies that we believe to be promising. The question of
feasibility strikes us as a key issue which has not been discussed enough and
for which we will try and provide a few thoughts. The criticism we have just
carried out with the previous observations does not prevent us, later on, from
suggesting ways to better take multimodality into account in the existing
paradigms. Section 10.2 thus presents a few multimodality extension
possibilities for methodologies meant for the oral, and for the few
methodologies already focusing on multimodality. The example “put that
there” that we used in section 6.2.2 will help us illustrate them.