Multimodal Dialogue System Assessment - Man-Machine Dialogue: Design and Challenges - page 165

dialogue. It is the same when the speech recognition module requires
learning: training this module on reference sentences repeated by the speaker
leads him to get a precise idea of the system's abilities and direct its behavior
as a consequence.
However, our designer can now compare the subjects' utterances with the
utterances imagined at the beginning. It is sometimes a second test for the
designer's morale: by trying to make the system work or simply sticking to
the examples produced, the subjects sometimes limit their utterances to very
simple sentences, which are devoid of any of the phenomena that were
planned for at the beginning and meant to be the point and innovating aspect
of the system. As an example, one of the subjects that was recorded in the
Magnet'Oz corpus (see Chapter 13 in [VAN 02]), only generated almost two
sentences in half an hour, that is “put that here” and “put that there”, which
indeed falls within the scope of the expected multimodality, but proves itself
to be extremely restrained! Our designer, who wishes to assess his/her system
by testing it, for example, on interesting multimodal reference situations, is
thus frustrated to the point that he/she sometimes incites the subject to
generate an utterance close to the utterance that he/she would assess (a
situation that was experienced during the Magnet'Oz recording). At this
point, the experiment conditions are not a worry anymore, and we start taking
up the list of initial phenomena to make them go one by one through the
system.... If this method allows the designer to carry out an experiment on
his system, it is clear that it does not allow him to generate a satisfactory
assessment.
Our designer then attempts another method: obtain the users' impressions
as well as all kinds of information on their experience with the system.
Whether it is a recorded interview or the filling out, in a free or directed
manner, of a questionnaire, the information gathered is often below the
designer's expectations: it is too general (“yes, the system answers correctly,
but not always”, “it's a nice project”, “it was fun”); it lacks precision (“it was
a bit slow”); there is a discrepancy compared with the expectations (“I used
gestures”); it is irrelevant (a long appreciation of the icon appearance and
objects displayed, a criticism of the applicative task or the performance of the
expert system integrated in the system). In the end, the designer is faced with
anecdotes he will not be able to tell anything from that can be properly used
for assessment or to carry out a diagnostic of his system.
For systems that are aimed at specialists, i.e. users who know the task or
users who already use a similar system but in versions built on classic