Visual, Tactile, and Multimodal Information Processing - Fundamentals and Assessment Tools for Occupational Ergonomics - page 510

to a decrease of activity related to processing of stimuli in unattended modalities. Collectively, findings
from these and similar studies suggest that modality expectations lead to a top-down facilitation of per-
ceptual attentional processes, that is, to an enhanced readiness to detect and discriminate information in
specific sensory channels (e.g., Posner and Petersen, 1990).
It is important to note that, in the above laboratory studies, the absolute changes in subjects' response
times due to modality expectations were very small. For example, Spence and Driver (1997) found
average costs and benefits ranging from a 10 m sec decrease to a 76 m sec increase in response time.
However, when viewed as a percentage of the average response times, as much as a 2% decrease and a
15% increase in reaction times to stimuli were found, depending on whether or not the stimuli appeared
in the expected modality. If these effects scale upto more complex environments, they are likely to result
in operationally significant performance costs and benefits and therefore need to be considered in the
design of future multimodal output systems.
23.4.3.2 MSE
MSE (e.g., Ferstl et al., 1994; Zubin, 1975) is another example of a crossmodal constraint on attention. It
describes the strong tendency of people to “respond more slowly to a target in one modality if the pre-
ceding target was presented in a different modality than if the preceding target was presented in the same
modality” (Spence and Driver, 1997). This implies that, in general, responses will be slower to targets in
less frequent modalities. It appears to be particularly difficult (time-consuming) to shift attention to the
visual or auditory channel away from rare events that are presented in the tactile modality (Spence et al.,
2000).
MSE likely plays a role in real-world domains where signals tend to be assigned to modalities based on
appropriateness (rather than experimental control) considerations. For example, auditory signals are
often reserved for warning and alerts; given the rare occurrence of these events, the frequency of auditory
cues is likely lower than that of visual indications, which are considered appropriate for presenting a wide
range of diverse and detailed information.
23.4.3.3 Crossmodal Spatial and Temporal Links
Deliberate shifts of attention to a particular location in either the visual or auditory modality have been
shown to lead to concurrent shifts in the other modality. Even though spatial attention in the secondary
modality appears to be somewhat less focused than in the primary modality, this spatial linkage between
modalities still results in performance benefits such as faster response times to cues in the same location
(e.g., Driver and Grossenbacher, 1996; LaBerge, 1995; Quinlan and Bailey, 1995). Physiological data
confirm that the processing of visual and auditory stimuli at attended spatial locations is facilitated at
an early sensory level. In contrast, spatially disparate stimuli produce either response depression or no
interaction (Calvert et al., 1999; Kennett et al., 2001; Macaluso et al., 2002; McDonald et al., 2000).
While visual -auditory spatial links have been studied extensively, much less is known about the inter-
action between vision or hearing and the sense of touch. Findings from a small number of studies
suggest that auditory-visual interactions may be qualitatively different from visual-haptic links. For
example, Gray et al. (2002) have shown that, while visual and auditory stimuli have to be presented
in close spatial proximity to produce performance benefits, proximal haptic cues can be used to reorient
visual attention to areas in distal space (e.g., to elements of a visual display in front of the person). In
general, tactile cues appear to be less affected by crossmodal spatial links and thus can be decoupled
from other attentional processes more easily and effectively (Eimer, 1999).
Temporal proximity between cues also affects crossmodal information processing. Approximate tem-
poral synchrony of cues in different modalities is a requirement for multisensory integration (e.g., the
ventriloquism 3 effect) and synergy. In other circumstances, however, an unintended close temporal
proximity of cues can lead to a reduced ability to process the second unrelated cue. For example,
3 The production of voice in such a way that the sound seems to come from a source other than the vocal organs of
the speaker (often a puppet).