Touch and Speech: Multimodal Interaction for Elderly Persons - Biomedical Engineering Systems and Technologies - page 371

6 Results and Discussion

6.1 Effectiveness of MIGSEP

Regarding the effectiveness of the MIGSEP system, 326 out of 341 tasks (10.5 of 11
for each, 95.6%) were correctly performed by all the participants, where 10.6 (96.6)
and 10.4 (94.5%) tasks were completed by each participant using touch or speech
input modalities respectively. This suggests a generally high effectiveness of the inte-
raction with the MIGSEP system. However, in order to assess the effectiveness at a
more detailed level, the standard statistical method Kappa coefficient was used.
In order to apply the kappa method, we needed to define the attribute value matrix
(AVM), which contains all information that has to be exchanged between MIGSEP
and the participants. E.g. table 1 shows the AVM for the task: ”Drive to a person
named Michael Frieling.” for both touch and speech modalities, where the expected
values of this task are also presented.

Table 1. An example AVM for the task “drive to a person name Michael Frieling”

Touch Speech

Attribute Expected value Attribute Expected value

Reached Level L1, L2, L3, L4 FN Michael

Goal Selection Michael Frieling LN Frieling

Confirm Yes (to the correct goal) G Male

M Person

By combining the actual data recorded during the experiment with the expected
attribute values in the AVMs, we can construct the confusion matrices for all tasks.
Table 2 shows e.g. the confusion matrix for the task ”drive to a person named Michael
Frieling” with the speech input modality, where ”M” and ”N” denote whether the
actual data match with the expected attribute values in the AVMs, and “SNU” for the
system failed-understanding situation. E.g. first name of Michael is wrongly targeted
for 4 times and wrongly understood by the system 14 times. Similar construction is
done with the AVM of touch input.

Table 2. The confusion matrix fort the task „drive to a person named Michael Frieling“

FN LN G M

sum

Data M N SNU M N SNU M N SNU M N SNU

FN 81 4 14 99

LN 82 3 12 97

G 57 4 61

M 3 2 1 6

Given one confusion matrix, the Kappa coefficient can then be calculated with
κ =

, (cf. [4])

In our experiment,

∑ ,

P(A) =