Crossmodal attention switching: Auditory dominance in temporal discrimination tasks

Visual stimuli are often processed more efficiently than accompanying stimuli in another modality. In line with this “visual dominance”, earlier studies on attentional switching showed a clear benefit for visual stimuli in a bimodal visual–auditory modality-switch paradigm that required spatial stimulus localization in the relevant modality. The present study aimed to examine the generality of this visual dominance effect. The modality appropriateness hypothesis proposes that stimuli in different modalities are differentially effectively processed depending on the task dimension, so that processing of visual stimuli is favored in the dimension of space, whereas processing auditory stimuli is favored in the dimension of time. In the present study, we examined this proposition by using a temporal duration judgment in a bimodal visual–auditory switching paradigm. Two experiments demonstrated that crossmodal interference (i.e., temporal stimulus congruence) was larger for visual stimuli than for auditory stimuli, suggesting auditory dominance when performing temporal judgment tasks. However, attention switch costs were larger for the auditory modality than for visual modality, indicating a dissociation of the mechanisms underlying crossmodal competition in stimulus processing and modality-specific biasing of attentional set.     

Division of attention between simultaneous and successive nonverbal signals varying in discriminability

Impaired performance indexed by accuracy usually accompanies the division of attention between simultaneous difficult signals. Presenting the signals successively reduces the impairment. Presenting easy signals eliminates it. Both variables crucial to these findings, relative time of onset and discriminability, were manipulated in the first experiment using a two-choice recognition paradigm involving bimodal signals. The results showed that, when a difficult signal was followed by two simultaneous easy ones, in a different modality from the first, performance was significantly worse on each than on the same signals presented alone. Simultaneous presentation of all three signals further increased the impairment for the difficult signal, but not for the easy ones. In the second experiment, the discriminability of the following signals was varied. The results showed, that when a difficult signal was followed by either two simultaneous easy signals, in a different modality from the first, or by two difficult ones, performance was significantly worse than on the same difficult signal presented alone. Further, when the preceding difficult signal was auditory, but not when it was visual, performance was better with easy following signals than with difficult ones. It is argued that an adequate account of the results from the two experiments requires the following assumptions concerning the processing of multiple signals: (1) a loss associated both with the acquisition and the storage of information, (2) preferential processing of visual signals over auditory ones. Proposals are made for the incorporation of both assumptions into a single model of divided attention.     

Auditory/visual duration bisection in patients with left or right medial-temporal lobe resection

Patients with unilateral (left or right) medial temporal lobe lesions and normal control (NC) volunteers participated in two experiments, both using a duration bisection procedure. Experiment 1 assessed discrimination of auditory and visual signal durations ranging from 2 to 8 s, in the same test session. Patients and NC participants judged auditory signals as longer than equivalent duration visual signals. The difference between auditory and visual time discrimination was equivalent for the three groups, suggesting that a unilateral temporal lobe resection does not modulate the modality effect. To document interval-timing abilities after temporal lobe resection for different duration ranges, Experiment 2 investigated the discrimination of brief, 50-200 ms, auditory durations in the same patients. Overall, patients with right temporal lobe resection were found to have more variable duration judgments across both signal modality and duration range. These findings suggest the involvement of the right temporal lobe at the level of the decision process in temporal discriminations.     

Auditory-visual conflicts in the perceived duration of lights, tones and gaps

The perceived duration of a short tone (1,000 or 1,500 msec) was longer than that of a separately presented light of equal length. Thus, when light and tone were presented simultaneously, there was a conflict in perceived duration. In that case, the perceived duration of an interval filled with both light and tone was close to that of an interval filled with tone alone. A silent gap in otherwise continuous tone was perceived as longer than a gap in otherwise continuous light, and the perceived duration of a gap occurring simultaneously in both light and tone was close to that of a gap in tone alone. Thus, auditory dominance occurred under the preceding conditions-that is, auditory-visual conflicts in perceived duration, whether occurring between filled intervals or gaps, were resolved in favor of the auditory modality. Visual dominance occurred only under one condition, in which the intensity of tone was reduced, and in which the perceived duration of a 500-msec light was longer than that of a 500-msec tone. The finding of auditory dominance in the perception of time runs counter to the results of studies of sensory conflicts in spatial perception, where vision typically dominates audition and touch.     

On the interplay of visuospatial and audiotemporal dominance: Evidence from a multimodal kappa effect

When participants judge multimodal audiovisual stimuli, the auditory information strongly dominates temporal judgments, whereas the visual information dominates spatial judgments. However, temporal judgments are not independent of spatial features. For example, in the kappa effect, the time interval between two marker stimuli appears longer when they originate from spatially distant sources rather than from the same source. We investigated the kappa effect for auditory markers presented with accompanying irrelevant visual stimuli. The spatial sources of the markers were varied such that they were either congruent or incongruent across modalities. In two experiments, we demonstrated that the spatial layout of the visual stimuli affected perceived auditory interval duration. This effect occurred although the visual stimuli were designated to be task-irrelevant for the duration reproduction task in Experiment 1, and even when the visual stimuli did not contain sufficient temporal information to perform a two-interval comparison task in Experiment 2. We conclude that the visual and auditory marker stimuli were integrated into a combined multisensory percept containing temporal as well as task-irrelevant spatial aspects of the stimulation. Through this multisensory integration process, visuospatial information affected even temporal judgments, which are typically dominated by the auditory modality.     

Spatial variations of visual-auditory fusion areas

The tolerance to spatial disparity between two synchronous visual and auditory components of a bimodal stimulus has been investigated in order to assess their respective contributions to perceptual fusion. The visual and auditory systems each have specific information-processing mechanisms, and provide different cues for scene perception, with the respective dominance of space for vision and of time for hearing. A broadband noise burst and a spot of light, 500 ms in duration, have been simultaneously presented to participants who had to judge whether these cues referred to a single spatial event. We examined the influence of (i) the range and the direction of spatial disparity between the visual and auditory components of a stimulation and (ii) the eccentricity of the bimodal stimulus in the observer's perceptual field. Size and shape properties of visual-auditory fusion areas have been determined in two dimensions. The greater the eccentricity within the perceptual field, the greater the dimension of these areas; however, this increase in size also depends on whether the direction of the disparity is vertical or horizontal. Furthermore, the relative location of visual and auditory signals significantly modifies the perception of unity in the vertical plane. The shape of the fusion areas, their variation in the field, and the perceptual result associated with the relative location of the visual and auditory components of the stimulus, concur towards a strong contribution of audition to visual-auditory fusion. The spatial ambiguity of the localisation capabilities of the auditory system may play a more essential role than accurate visual resolution in determining fusion.     

Differential effects of auditory and visual signals on clock speed and temporal memory

The effects of signal modality on duration classification in college students were studied with the duration bisection task. When auditory and visual signals were presented in the same test session and shared common anchor durations, visual signals were classified as shorter than equivalent duration auditory signals. This occurred when auditory and visual signals were presented sequentially in the same test session and when presented simultaneously but asynchronously. Presentation of a single modality signal within a test session, or both modalities but with different anchor durations did not result in classification differences. The authors posit a model in which auditory and visual signals drive an internal clock at different rates. The clock rate difference is due to an attentional effect on the mode switch and is revealed only when the memories for the short and long anchor durations consist of a mix of contributions from accumulations generated by both the fast auditory and slower visual clock rates. When this occurs auditory signals seem longer than visual signals relative to the composite memory representation.     

视听刺激突显性对视觉主导效应的影响

本研究使用空间任务-转换范式,控制视、听刺激的突显性,探讨自下而上注意对视觉主导效应的影响。结果表明视、听刺激突显性显著地影响视觉主导效应,实验一中当听觉刺激为高突显性时,视觉主导效应显著减弱。实验二中当听觉刺激为高突显性并且视觉刺激为低突显性时,视觉主导效应进一步减弱但依然存在。结果支持偏向竞争理论,在跨通道视听交互过程中视觉刺激更具突显性,在多感觉整合过程中更具加工优势。     

Auditory dominance in temporal processing: new evidence from synchronization with simultaneous visual and auditory sequences

Evidence that audition dominates vision in temporal processing has come from perceptual judgment tasks. This study shows that this auditory dominance extends to the largely subconscious processes involved in sensorimotor coordination. Participants tapped their finger in synchrony with auditory and visual sequences containing an event onset shift (EOS), expected to elicit an involuntary phase correction response (PCR), and also tried to detect the EOS. Sequences were presented in unimodal and bimodal conditions, including one in which auditory and visual EOSs of opposite sign coincided. Unimodal results showed greater variability of taps, smaller PCRs, and poorer EOS detection in vision than in audition. In bimodal conditions, variability of taps was similar to that for unimodal auditory sequences, and PCRs depended more on auditory than on visual information, even though attention was always focused on the visual sequences.     

Effects of temporal asynchrony and stimulus magnitude on competitive audio–visual binding

When making decisions as to whether or not to bind auditory and visual information, temporal and stimulus factors both contribute to the presumption of multimodal unity. In order to study the interaction between these factors, we conducted an experiment in which auditory and visual stimuli were placed in competitive binding scenarios, whereby an auditory stimulus was assigned to either a primary or a secondary anchor in a visual context (VAV) or a visual stimulus was assigned to either a primary or secondary anchor in an auditory context (AVA). Temporal factors were manipulated by varying the onset of the to-be-bound stimulus in relation to the two anchors. Stimulus factors were manipulated by varying the magnitudes of the visual (size) and auditory (intensity) signals. The results supported the dominance of temporal factors in auditory contexts, in that effects of time were stronger in AVA than in VAV contexts, and stimulus factors in visual contexts, in that effects of magnitude were stronger in VAV than in AVA contexts. These findings indicate the precedence for temporal factors, with particular reliance on stimulus factors when the to-be-assigned stimulus was temporally ambiguous. Stimulus factors seem to be driven by high-magnitude presentation rather than cross-modal congruency. The interactions between temporal and stimulus factors, modality weighting, discriminability, and object representation highlight some of the factors that contribute to audio–visual binding.     

Re-evaluating visual and auditory dominance through modality switching costs and congruency analyses

Competition between the senses can lead to modality dominance, where one sense influences multi-modal processing to a greater degree than another. Modality dominance can be influenced by task demands, speeds of processing, contextual influence and practice. To resolve previous discrepancies in these factors, we assessed modality dominance in an audio-visual paradigm controlling for the first three factors while manipulating the fourth. Following a uni-modal task in which auditory and visual processing were equated, participants completed a pre-practice selective attention bimodal task in which the congruency relationship and task-relevant modality changed across trials. Participants were given practice in one modality prior to completing a post-practice selective attention bimodal task similar to the first. The effects of practice were non-specific as participants were speeded post-practice relative to pre-practice. Congruent stimuli relative to incongruent stimuli, also led to increased processing efficiency. RT data tended to reveal symmetric modality switching costs whereas the error rate data tended to reveal asymmetric modality switching costs in which switching from auditory to visual processing was particularly costly. The data suggest that when a number of safeguards are put in place to equate auditory and visual responding as far as possible, evidence for an auditory advantage can arise.     

Crossmodal duration perception involves perceptual grouping, temporal ventriloquism, and variable internal clock rates

Here, we investigate how audiovisual context affects perceived event duration with experiments in which observers reported which of two stimuli they perceived as longer. Target events were visual and/or auditory and could be accompanied by nontargets in the other modality. Our results demonstrate that the temporal information conveyed by irrelevant sounds is automatically used when the brain estimates visual durations but that irrelevant visual information does not affect perceived auditory duration (Experiment 1). We further show that auditory influences on subjective visual durations occur only when the temporal characteristics of the stimuli promote perceptual grouping (Experiments 1 and 2). Placed in the context of scalar expectancy theory of time perception, our third and fourth experiments have the implication that audiovisual context can lead both to changes in the rate of an internal clock and to temporal ventriloquism-like effects on perceived on- and offsets. Finally, intramodal grouping of auditory stimuli diminished any crossmodal effects, suggesting a strong preference for intramodal over crossmodal perceptual grouping (Experiment 5).     

Les effets de la modalité sensorielle sur la perception du temps

The subjective duration of an event can be influenced by the modality of the signal demarcating that event. For example, auditory signals are often judged as of longer duration than equivalent duration visual signals. Within the framework of pacemaker-accumulator models of timing, such modality effects raise the question of whether the internal clock is modality specific or amodal. The answer to this question is quite important for the development of cognitively and neurologically realistic models of timing and time perception. Here, we argue that the internal clock has both modality specific and amodal components. Specifically, we claim there is a common amodal pacemaker, but the switch-accumulator systems are modality specific. Moreover, we also posit that long-term memory representations of duration are stored amodally.     

Influence of auditory and audiovisual stimuli on the right–left prevalence effect

When auditory stimuli are used in two-dimensional spatial compatibility tasks, where the stimulus and response configurations vary along the horizontal and vertical dimensions simultaneously, a right–left prevalence effect occurs in which horizontal compatibility dominates over vertical compatibility. The right–left prevalence effects obtained with auditory stimuli are typically larger than that obtained with visual stimuli even though less attention should be demanded from the horizontal dimension in auditory processing. In the present study, we examined whether auditory or visual dominance occurs when the two-dimensional stimuli are audiovisual, as well as whether there will be cross-modal facilitation of response selection for the horizontal and vertical dimensions. We also examined whether there is an additional benefit of adding a pitch dimension to the auditory stimulus to facilitate vertical coding through use of the spatial-musical association of response codes (SMARC) effect, where pitch is coded in terms of height in space. In Experiment 1, we found a larger right–left prevalence effect for unimodal auditory than visual stimuli. Neutral, non-pitch coded, audiovisual stimuli did not result in cross-modal facilitation, but did show evidence of visual dominance. The right–left prevalence effect was eliminated in the presence of SMARC audiovisual stimuli, but the effect influenced horizontal rather than vertical coding. Experiment 2 showed that the influence of the pitch dimension was not in terms of influencing response selection on a trial-to-trial basis, but in terms of altering the salience of the task environment. Taken together, these findings indicate that in the absence of salient vertical cues, auditory and audiovisual stimuli tend to be coded along the horizontal dimension and vision tends to dominate audition in this two-dimensional spatial stimulus–response task.     

Duration discrimination in the context of age,sex, and cognition

It has been proposed that the perception of very short duration is governed by sensory mechanisms, whereas the perception of longer duration depends on cognitive capacities. Four duration discrimination tasks (modalities: visual, auditory; base duration: 100 ms, 1000 ms) were used to study the relation between time perception, age, sex, and cognitive abilities (alertness, visual and verbal working memory, general fluid reasoning) in 100 subjects aged between 21 and 84 years. Temporal acuity was higher (Weber fractions are lower) for longer stimuli and for the auditory modality. Age was related to the visual 100 ms condition only, with lower temporal acuity in elder participants. Alertness was significantly related to auditory and visual Weber fractions for shorter stimuli only. Additionally, visual working memory was a significant predictor for shorter visual stimuli. These results indicate that alertness, but also working memory, are associated with temporal discrimination of very brief duration.     

A comparison of auditory and visual apparent motion presented individually and with crossmodal moving distractors

Unimodal auditory and visual apparent motion (AM) and bimodal audiovisual AM were investigated to determine the effects of crossmodal integration on motion perception and direction-of-motion discrimination in each modality. To determine the optimal stimulus onset asynchrony (SOA) ranges for motion perception and direction discrimination, we initially measured unimodal visual and auditory AMs using one of four durations (50, 100, 200, or 400 ms) and ten SOAs (40-450 ms). In the bimodal conditions, auditory and visual AM were measured in the presence of temporally synchronous, spatially displaced distractors that were either congruent (moving in the same direction) or conflicting (moving in the opposite direction) with respect to target motion. Participants reported whether continuous motion was perceived and its direction. With unimodal auditory and visual AM, motion perception was affected differently by stimulus duration and SOA in the two modalities, while the opposite was observed for direction of motion. In the bimodal audiovisual AM condition, discriminating the direction of motion was affected only in the case of an auditory target. The perceived direction of auditory but not visual AM was reduced to chance levels when the crossmodal distractor direction was conflicting. Conversely, motion perception was unaffected by the distractor direction and, in some cases, the mere presence of a distractor facilitated movement perception.     

The lifespan of time intervals in reference memory

To further explore how memory influences time judgments, we conducted two experiments on the lifespan of temporal representations in memory. Penney et al (2000, Journal of Experimental Psychology Human Perception and Performance 26 1770-1787) reported that the perceived duration of auditorily and visually marked intervals differs only when both marker-type intervals are compared directly. This finding can be explained by a 'memory-mixing' process, whereby the memory trace of previous intervals influences the perception of upcoming ones, which are then added to the memory content. In the experiments discussed here, we manipulated the mixing mode of auditory/visual signal presentations. In experiment 1, signals from the same modality were either grouped by blocks or randomised within blocks. The results showed that the auditory/visual difference decreased but remained present when modalities were grouped by blocks. In experiment 2, we used a line-segmentation task. The results showed that, after a training block was performed in one modality, the perceived duration of signals from the other modality was distorted for at least 30 trials and that the magnitude of the difference decreased as the block went on. The results of both experiments highlight the influence of memory on time judgments, providing empirical support to, and quantitative portrayal of, the memory-mixing process.     

Hearing what the eyes see: auditory encoding of visual temporal sequences

When the senses deliver conflicting information, vision dominates spatial processing, and audition dominates temporal processing. We asked whether this sensory specialization results in cross-modal encoding of unisensory input into the task-appropriate modality. Specifically, we investigated whether visually portrayed temporal structure receives automatic, obligatory encoding in the auditory domain. In three experiments, observers judged whether the changes in two successive visual sequences followed the same or different rhythms. We assessed temporal representations by measuring the extent to which both task-irrelevant auditory information and task-irrelevant visual information interfered with rhythm discrimination. Incongruent auditory information significantly disrupted task performance, particularly when presented during encoding; by contrast, varying the nature of the rhythm-depicting visual changes had minimal impact on performance. Evidently, the perceptual system automatically and obligatorily abstracts temporal structure from its visual form and represents this structure using an auditory code, resulting in the experience of "hearing visual rhythms."     

选择性注意和分配性注意对多感觉整合的不同影响*

以简单图形为视觉刺激,以短纯音作为听觉刺激,通过指导被试注意不同通道（注意视觉、注意听觉、注意视听）以形成不同注意状态（选择性注意和分配性注意）,考察了注意对多感觉整合的影响,发现只有在分配性注意时被试对双通道目标的反应最快最准确。通过竞争模型分析发现,这种对双通道目标的加工优势源自于视听双通道刺激的整合。上述结果表明,只有在分配性注意状态下才会产生多感觉整合。     

Auditory and visual differences in time perception? An investigation from a developmental perspective with neuropsychological tests

Adults and children (5- and 8-year-olds) performed a temporal bisection task with either auditory or visual signals and either a short (0.5-1.0s) or long (4.0-8.0s) duration range. Their working memory and attentional capacities were assessed by a series of neuropsychological tests administered in both the auditory and visual modalities. Results showed an age-related improvement in the ability to discriminate time regardless of the sensory modality and duration. However, this improvement was seen to occur more quickly for auditory signals than for visual signals and for short durations rather than for long durations. The younger children exhibited the poorest ability to discriminate time for long durations presented in the visual modality. Statistical analyses of the neuropsychological scores revealed that an increase in working memory and attentional capacities in the visuospatial modality was the best predictor of age-related changes in temporal bisection performance for both visual and auditory stimuli. In addition, the poorer time sensitivity for visual stimuli than for auditory stimuli, especially in the younger children, was explained by the fact that the temporal processing of visual stimuli requires more executive attention than that of auditory stimuli.