Environmental Engineering Reference
In-Depth Information
Synergy
3
, redundancy, and increased bandwidth of information transfer are proposed benefits
of multimodal presentation (Sarter (2006)). If information is presented in redundant multiple
modalities, then various concurrent forms and different aspects of the same process are
presented. Concurrent information may coordinate activities in response to the ongoing
processes.
Several studies have shown that using auditory and visual feedback together increase
performance compared to using each of these feedbacks separately. For example, under
noisy conditions, observing the movements of the lips and gestures of the speaker can
compensate for the lost speech perception which can be equivalent to increasing the auditory
signal-to-noise ratio (Sumby & Pollack (1954)). It was also demonstrated that reaction times
for detecting a visual signal were slower in unimodal conditions, compared to a condition in
which a sound was heard in close temporal synchrony to the visual signal (Doyle & Snowden
(2001)). Handel and Buffardi (Handel & Buffardi (1969)) found that in a pattern judging task,
performance was better using a combination of auditory and visual cues than either audition
or vision alone.
On the other hand, adding haptic feedback to another modality was also found to improve
performance compared to the use of each modality individually. Pairs of auditory and haptic
signals delivered simultaneously were detected faster than when the same signals presented
unimodally (Murray et al. (2005)). In a texture discrimination task, participants accuracy was
improved (less errors) when they received bi-modal visual and haptic cues as compared
to uni-modal conditions in which only the haptic or the visual cue was presented (Heller
(1982)). Sarlegna et al. (Sarlegna et al. (2007)) demonstrated that feedback that was combined
of different sensory modalities improved the performance in reaching task.
In general, the advantages of the use of multimodal feedback are summarized below:
1. Various forms and different aspects of the same task can be presented simultaneosly when
information is presented using multimodal feedback.
2. The use of a second channel to provide information can compensate the lost of information
of the first one.
3. When one sensorial channel is overloaded, other channel can be used to convey additional
information.
4. The reaction times are slower and the performance is increased with the use of multimodal
feedback.
But, in order to provide all these benefits it is essential to assure the coherence and consistence
among the different stimuli. Conflicts among the information provided by the different
channels can deteriorate the performance of trainees. In addition, it is important to identify
the different components of the task to be learnt in order to select and employ the most suitable
channel to provide useful information for each one.
2.4
State of the art on multimodal systems for procedural tasks
Some studies have tested and analyzed the use of the multimodal systems as a simulation tool
or a training tool to teach and evaluate the knowledge of a trainee in procedural tasks. The
review presented in this section is focused on procedural tasks related to the assembly and
disassembly processes.
3
The merging information from multiple aspects of the same task.
