Digital Signal Processing Reference
In-Depth Information
The results of the first two experiments are shown in Table 4-1. As shown
in Table 4-1, it would be natural for multiple persons to interact together in
order to derive their desired information. Only five dialogues (10%) are
single-speaker dialogues. It can also be observed from Table 4-1 that the three
interaction types happened with almost the same probability. The statistics in
Table 4-1 suggest that the study of MSDS is a necessary one.
As mentioned in Section 3, the determination of interaction type involved
comparing the domain primary feature and secondary features
for speakers. The correct rates of interaction type determination were
98.3%, 95.7% and 94.1% for independent, cooperative, and conflict
interactions, respectively. The wrong determinations occurred in cases in
which the speaker omitted the domain slot and provided just primary or
secondary feature slot information. The system “guessed” the domain slot
based on the identification of the other slots, which may have resulted in
incorrect determinations of the interaction type.
4.3
Experimental Results for the MSDS Evaluation
The evaluation of a spoken dialogue system can be classified as
objective
and
subjective
as indicated by Danieli and Gerbino [17], Hirschman and Pao
[18], and Walker et al. [19]. Objective metrics can be calculated without
human judgment, and in many cases can be logged by the SDS so that they
can be calculated automatically. Subjective metrics require subjects using the
system, and/or human evaluators to categorize the dialogue or utterances with
various qualitative measures. Both subjective and objective evaluations were
used in the experiment. The metrics were:
1.
percentage of different interaction types (i.e., independent, cooperative,
conflict)
