High-intensity sound increases the size of visually perceived objects

The effect of audiovisual interactions on size perception has yet to be examined, despite its fundamental importance in daily life. Previous studies have reported that object length can be estimated solely on the basis of the sounds produced when an object is dropped. Moreover, it has been shown that people typically and easily perceive the correspondence between object sizes and sound intensities. It is therefore possible that auditory stimuli may act as cues for object size, thereby altering the visual perception of size. Thus, in the present study we examined the effects of auditory stimuli on the visual perception of size. Specifically, we investigated the effects of the sound intensity of auditory stimuli, the temporal window of audiovisual interactions, and the effects of the retinal eccentricity of visual stimuli. The results indicated that high-intensity auditory stimuli increased visually perceived object size, and that this effect was especially strong in the peripheral visual field. Additional consideration indicated that this effect on the visual perception of size is induced when the cue reliability is relatively higher for the auditory than for the visual stimuli. In addition, we further suggest that the cue reliabilities of visual and auditory stimuli relate to retinal eccentricity and sound intensity, respectively.     

Multisensory-mediated auditory localization

Multisensory integration is a powerful mechanism for maximizing sensitivity to sensory events. We examined its effects on auditory localization in healthy human subjects. The specific objective was to test whether the relative intensity and location of a seemingly irrelevant visual stimulus would influence auditory localization in accordance with the inverse effectiveness and spatial rules of multisensory integration that have been developed from neurophysiological studies with animals [Stein and Meredith, 1993 The Merging of the Senses (Cambridge, MA: MIT Press)]. Subjects were asked to localize a sound in one condition in which a neutral visual stimulus was either above threshold (supra-threshold) or at threshold. In both cases the spatial disparity of the visual and auditory stimuli was systematically varied. The results reveal that stimulus salience is a critical factor in determining the effect of a neutral visual cue on auditory localization. Visual bias and, hence, perceptual translocation of the auditory stimulus appeared when the visual stimulus was supra-threshold, regardless of its location. However, this was not the case when the visual stimulus was at threshold. In this case, the influence of the visual cue was apparent only when the two cues were spatially coincident and resulted in an enhancement of stimulus localization. These data suggest that the brain uses multiple strategies to integrate multisensory information.     

Effects of auditory stimulus intensity on response force in simple, go/no-go, and choice RT tasks

In four experiments, increasing the intensities of both relevant and irrelevant auditory stimuli was found to increase response force (RF) in simple, go/no-go, and choice reaction time (RT) tasks. These results raise problems for models that localize the effects of auditory intensity on purely perceptual processes, indicating instead that intensity also affects motor output processes under many circumstances. In Experiment 1, simple RT, go/no-go, and choice RT tasks were compared, using the same stimuli for all tasks. Auditory stimulus intensity affected both RT and RF, and these effects were not modulated by task. In Experiments 2-4, an irrelevant auditory accessory stimulus accompanied a relevant visual stimulus, and the go/no-go and choice tasks were used. The intensity of the irrelevant auditory accessory stimulus was found to affect RT and RF, although the sizes of these effects depended somewhat on the temporal predictability of the accessory stimulus.     

Spatial and temporal factors determine auditory-visual interactions in human saccadic eye movements

In this paper, we show that human saccadic eye movements toward a visual target are generated with a reduced latency when this target is spatially and temporally aligned with an irrelevant auditory nontarget. This effect gradually disappears if the temporal and/or spatial alignment of the visual and auditory stimuli are changed. When subjects are able to accurately localize the auditory stimulus in two dimensions, the spatial dependence of the reduction in latency depends on the actual radial distance between the auditory and the visual stimulus. If, however, only the azimuth of the sound source can be determined by the subjects, the horizontal target separation determines the strength of the interaction. Neither saccade accuracy nor saccade kinematics were affected in these paradigms. We propose that, in addition to an aspecific warning signal, the reduction of saccadic latency is due to interactions that take place at a multimodal stage of saccade programming, where theperceived positions of visual and auditory stimuli are represented in a common frame of reference. This hypothesis is in agreement with our finding that the saccades often are initially directed to the average position of the visual and the auditory target, provided that their spatial separation is not too large. Striking similarities with electrophysiological findings on multisensory interactions in the deep layers of the midbrain superior colliculus are discussed.     

Foreperiod and visual stimulus intensity: A reappraisal

The joint effects of stimulus modality, stimulus intensity, and foreperiod on simple RT were investigated. In experiment 1 an interaction was found between stimulus intensity, both visual and auditory, and a variable FP such that the intensity-effect on RT was largest at the shortest FP. Experiment 2 provided a successful replication with smaller and weaker visual stimuli. No interaction was observed with a constant FP, although the visual stimuli were identical and the auditory ones psychophysically equivalent to the visual stimuli of experiment 1.It is proposed that an additive or interactive relationship between stimulus intensity and FP can be inferred only when the mental processes called for by the various uses of FP are simultaneously considered. Another precondition is an adequate sampling of the intensity-continuum with special reference to the retinal size of visual stimuli.     

Multisensory identification of natural objects in a two-way crossmodal priming paradigm

The question of how vision and audition interact in natural object identification is currently a matter of debate. We developed a large set of auditory and visual stimuli representing natural objects in order to facilitate research in the field of multisensory processing. Normative data was obtained for 270 brief environmental sounds and 320 visual object stimuli. Each stimulus was named, categorized, and rated with regard to familiarity and emotional valence by N=56 participants (Study 1). This multimodal stimulus set was employed in two subsequent crossmodal priming experiments that used semantically congruent and incongruent stimulus pairs in a S1-S2 paradigm. Task-relevant targets were either auditory (Study 2) or visual stimuli (Study 3). The behavioral data of both experiments expressed a crossmodal priming effect with shorter reaction times for congruent as compared to incongruent stimulus pairs. The observed facilitation effect suggests that object identification in one modality is influenced by input from another modality. This result implicates that congruent visual and auditory stimulus pairs were perceived as the same object and demonstrates a first validation of the multimodal stimulus set.     

Locus of the intensity effect in simple reaction time tasks

Evidence is still inconclusive regarding the locus of the stimulus intensity effect on information processing in reaction tasks. Miller, Ulrich, and Rinkenauer (1999) addressed this question by assessing the intensity effect on stimulus- and response-locked lateralized readiness potentials (LRPs) as indices of the sensory and motor parts of reaction time (RT). In the case of visual stimuli, they observed that application of brighter stimuli resulted in a shortening of RT and stimulus-locked LRP (S-LRP), but not of response-locked LRP (R-LRP). The results for auditory stimuli, however, were unclear. In spite of a clear RT reduction due to increased loudness, neither S-LRP nor R-LRP onset was affected. A reason for this failure might have been a relatively small range of intensity variation and the type of task. To check for this possibility, we performed three experiments in which broader ranges of stimulus intensities and simple, rather than choice, response tasks were used. Although the intensity effect on the R-LRP was negligible, S-LRP followed RT changes, irrespective of stimulus modality. These findings support the conclusion that stimulus intensity exerts its effect before the start of motoric processes. Finally, S-LRP and R-LRP findings are discussed within a broader information-processing perspective to check the validity of the claim that S-LRP and R-LRP can, indeed, be considered as pure estimates of the duration of sensory and motor processes.     

The influence of stimulus properties on multisensory processing in the awake primate superior colliculus.

Multisensory integration is a process whereby information converges from different sensory modalities to produce a response that is different from that elicited by the individual modalities presented alone. A neural basis for multisensory integration has been identified within a variety of brain regions, but the most thoroughly examined model has been that of the superior colliculus (SC). Multisensory processing in the SC of anaesthetized animals has been shown to be dependent on the physical parameters of the individual stimuli presented (e.g., intensity, direction, velocity) as well as their spatial relationship. However, it is unknown whether these stimulus features are important, or evident, in the awake behaving animal. To address this question, we evaluated the influence of physical properties of sensory stimuli (visual intensity, direction, and velocity; auditory intensity and location) on sensory activity and multisensory integration of SC neurons in awake, behaving primates. Monkeys were trained to fixate a central visual fixation point while visual and/or auditory stimuli were presented in the periphery. Visual stimuli were always presented within the contralateral receptive field of the neuron whereas auditory stimuli were presented at either ipsi- or contralateral locations. Many of the SC neurons responsive to these sensory stimuli (n = 66/84; 76%) had stronger responses when the visual and auditory stimuli were combined at contralateral locations than when the auditory stimulus was located on the ipsilateral side. This trend was significant across the population of auditory-responsive neurons. In addition, some SC neurons (n = 31) were presented a battery of tests in which the quality of one stimulus of a pair was systematically manipulated. A small proportion of these neurons (n = 8/31; 26%) showed preferential responses to stimuli with specific physical properties, and these preferences were not significantly altered when multisensory stimulus combinations were presented. These data demonstrate that multisensory processing in the awake behaving primate is influenced by the spatial congruency of the stimuli as well as their individual physical properties.     

Visual dominance in the pigeon

In Experiment 1, three pigeons were trained to obtain grain by depressing one foot treadle in the presence of a 746-Hertz tone stimulus and by depressing a second foot treadle in the presence of a red light stimulus. Intertrial stimuli included white light and the absence of tone. The latencies to respond on auditory element trials were as fast, or faster, than on visual element trials, but pigeons always responded on the visual treadle when presented with a compound stimulus composed of the auditory and visual elements. In Experiment 2, pigeons were trained on the auditory-visual discrimination task using as trial stimuli increases in the intensity of auditory or visual intertrial stimuli. Again, pigeons showed visual dominance on subsequent compound stimulus test trials. In Experiment 3, on compound test trials, the onset of the visual stimulus was delayed relative to the onset of the auditory stimulus. Visual treadle responses generally occurred with delay intervals of less than 500 milliseconds, and auditory treadle responses generally occurred with delay intervals of greater than 500 milliseconds. The results are discussed in terms of Posner, Nissen, and Klein's (1976) theory of visual dominance in humans.     

Intermodal perception at birth: Intersensory redundancy guides newborn infants’ learning of arbitrary auditory−visual pairings

In this study the ability of newborn infants to learn arbitrary auditory–visual associations in the absence versus presence of amodal (redundant) and contingent information was investigated. In the auditory-noncontingent condition 2-day-old infants were familiarized to two alternating visual stimuli (differing in colour and orientation), each accompanied by its ‘own’ sound: when the visual stimulus was presented the sound was continuously presented, independently of whether the infant looked at the visual stimulus. In the auditory-contingent condition the auditory stimulus was presented only when the infant looked at the visual stimulus: thus, presentation of the sound was contingent upon infant looking. On the post-familiarization test trials attention recovered strongly to a novel auditory–visual combination in the auditory-contingent condition, but remained low, and indistinguishable from attention to the familiar combination, in the auditory-noncontingent condition. These findings are a clear demonstration that newborn infants’ learning of arbitrary auditory–visual associations is constrained and guided by the presence of redundant (amodal) contingent information. The findings give strong support to Bahrick’s theory of early intermodal perception.     

Reaction time and temporal serial judgment: corroboration or dissociation?]

Dissociations between a motor response and the subject's verbal report have been reported from various experiments that investigated special experimental effects (e.g., metacontrast or induced motion). To examine whether similar dissociations can also be observed under standard experimental conditions, we compared reaction times (RT) and temporal order judgments (TOJ) to visual and auditory stimuli of three intensity levels. Data were collected from six subjects, each of which served for nine sessions. The results showed a strong, highly significant modality dissociation: While RTs to auditory stimuli were shorter than RTs to visual stimuli, the TOJ data indicated longer processing times for auditory than for visual stimuli. This pattern was found over the whole range of intensities investigated. Light intensity had similar effects on RT and TOJ, while there was a marginally significant tendency of tone intensity to affect RT more strongly than TOJ. It is concluded that modality dissociation is an example of "direct parameter specification", where the pathway from stimulus to response in the simple RT experiment is (at least partially) separate from the pathway that leads to a conscious, reportable representation. Two variants of this notion and alternatives to it are discussed.     

Crossmodal attention switching: Auditory dominance in temporal discrimination tasks

Visual stimuli are often processed more efficiently than accompanying stimuli in another modality. In line with this “visual dominance”, earlier studies on attentional switching showed a clear benefit for visual stimuli in a bimodal visual–auditory modality-switch paradigm that required spatial stimulus localization in the relevant modality. The present study aimed to examine the generality of this visual dominance effect. The modality appropriateness hypothesis proposes that stimuli in different modalities are differentially effectively processed depending on the task dimension, so that processing of visual stimuli is favored in the dimension of space, whereas processing auditory stimuli is favored in the dimension of time. In the present study, we examined this proposition by using a temporal duration judgment in a bimodal visual–auditory switching paradigm. Two experiments demonstrated that crossmodal interference (i.e., temporal stimulus congruence) was larger for visual stimuli than for auditory stimuli, suggesting auditory dominance when performing temporal judgment tasks. However, attention switch costs were larger for the auditory modality than for visual modality, indicating a dissociation of the mechanisms underlying crossmodal competition in stimulus processing and modality-specific biasing of attentional set.     

Perceptual load influences auditory space perception in the ventriloquist aftereffect

A period of exposure to trains of simultaneous but spatially offset auditory and visual stimuli can induce a temporary shift in the perception of sound location. This phenomenon, known as the ‘ventriloquist aftereffect’, reflects a realignment of auditory and visual spatial representations such that they approach perceptual alignment despite their physical spatial discordance. Such dynamic changes to sensory representations are likely to underlie the brain’s ability to accommodate inter-sensory discordance produced by sensory errors (particularly in sound localization) and variability in sensory transduction. It is currently unknown, however, whether these plastic changes induced by adaptation to spatially disparate inputs occurs automatically or whether they are dependent on selectively attending to the visual or auditory stimuli. Here, we demonstrate that robust auditory spatial aftereffects can be induced even in the presence of a competing visual stimulus. Importantly, we found that when attention is directed to the competing stimuli, the pattern of aftereffects is altered. These results indicate that attention can modulate the ventriloquist aftereffect.     

Flexibility of working memory encoding in a sentence–picture–sound verification task

Dual-process accounts of working memory have suggested distinct encoding processes for verbal and visual information in working memory, but encoding for nonspeech sounds (e.g., tones) is not well understood. This experiment modified the sentence–picture verification task to include nonspeech sounds with a complete factorial examination of all possible stimulus pairings. Participants studied simple stimuli–pictures, sentences, or sounds–and encoded the stimuli verbally, as visual images, or as auditory images. Participants then compared their encoded representations to verification stimuli–again pictures, sentences, or sounds–in a two-choice reaction time task. With some caveats, the encoding strategy appeared to be as important or more important than the external format of the initial stimulus in determining the speed of verification decisions. Findings suggested that: (1) auditory imagery may be distinct from verbal and visuospatial processing in working memory; (2) visual perception but not visual imagery may automatically activate concurrent verbal codes; and (3) the effects of hearing a sound may linger for some time despite recoding in working memory. We discuss the role of auditory imagery in dual-process theories of working memory.     

Cross-modal training induces changes in spatial representations early in the auditory processing pathway

In the ventriloquism aftereffect, brief exposure to a consistent spatial disparity between auditory and visual stimuli leads to a subsequent shift in subjective sound localization toward the positions of the visual stimuli. Such rapid adaptive changes probably play an important role in maintaining the coherence of spatial representations across the various sensory systems. In the research reported here, we used event-related potentials (ERPs) to identify the stage in the auditory processing stream that is modulated by audiovisual discrepancy training. Both before and after exposure to synchronous audiovisual stimuli that had a constant spatial disparity of 15°, participants reported the perceived location of brief auditory stimuli that were presented from central and lateral locations. In conjunction with a sound localization shift in the direction of the visual stimuli (the behavioral ventriloquism aftereffect), auditory ERPs as early as 100 ms poststimulus (N100) were systematically modulated by the disparity training. These results suggest that cross-modal learning was mediated by a relatively early stage in the auditory cortical processing stream.     

面孔表情和声音情绪信息整合对返回抑制的影响

返回抑制(inhibition of return, IOR)与情绪刺激都具有引导注意偏向、提高搜索效率的特点, 但二者间是否存在一定的交互作用迄今为止尚不明确。研究采用“线索-目标”范式并在视听双通道呈现情绪刺激来考察情绪刺激的加工与IOR的交互作用。实验1中情绪刺激以单通道视觉面孔或一致的视听双通道呈现, 实验2通过在视听通道呈现不一致的情绪刺激进一步考察视听双通道情绪一致刺激对IOR的影响是否是由听觉通道一致的情绪刺激导致的, 即是否对听觉通道的情绪刺激进行了加工。结果发现, 视听双通道情绪一致刺激能够削弱IOR, 但情绪不一致刺激与IOR之间不存在交互作用, 并且单双通道的IOR不存在显著差异。研究结果表明仅在视听双通道呈现情绪一致刺激时, 才会影响同一阶段的IOR, 这进一步支持了IOR的知觉抑制理论。     

Informational and neural adaptation curves are asynchronous

Norwich’s entropy theory of perception (plus a few additional assumptions) suggests the existence of an “informational adaptation curve” (change in stimulus equivocation with stimulus duration) for suprathreshold prothetic stimuli that is synchronous with the “neural adaptation curve” (change in firing rate with stimulus duration) observed for sensory neurons. Five experiments are reported in which informational adaptation curves were measured for auditory and visual stimuli by having subjects make absolute identifications of suprathreshold sound or light intensities of various durations. Information transmissions for the shortest duration stimuli (1 and 5 msec, respectively, for light and sound) were surprisingly large (small equivocations), indicating that intensity information is acquired very rapidly by the whole organism. The equations of entropy theory were fitted to adaptation data for peripheral sensory neurons (spiral ganglion cells and retinal ganglion cells) and were compared to the informational adaptation curves. It was found that informational adaptation occurred more rapidly than neural adaptation. That is, the two types of adaptation process are asynchronous. However, for both audition and vision, the total amount of information mediated by the adaptation process (channel capacity) was-about the same for both types of processes (2.03 bits vs. 2.1 bits per stimulus for sound intensity; 1.3 vs. 2.0 bits per stimulus for light intensity). Faster acquisition could be accomplished in the whole organism through convergent neural circuits that increase sampling, rate-by pooling the samples taken by individual receptor systems     

Informational and neural adaptation curves are asynchronous.

Norwich's entropy theory of perception (plus a few additional assumptions) suggests the existence of an "infromational adaptation curves" (change in stimulus equivocation with stimulus duration) for suprathreshold prothetic stimuli that is synchronous with the "neural adaptation curve" (change in firing rate with stimulus duration) observed for sensory neurons. Five experiments are reported in which informational adaptation curves were measured for auditory and visual stimuli by having subjects make absolute identifications of suprathreshold sound or light intenstiies of various durations. Information transmissions for the shortest duration stimuli (1 and 5 msec, respectively, for light and sound) were surprisingly large (small equivocations), indicating that intensity information is acquired very rapidly by the whole organism. The equations of entropy theory were fitted to adaptation data for peripheral sensory neurons (spiral ganglion cells and retinal ganglion cells) and were compared to the informational adaptation curves. It was found that informational adaptation occurred more rapidly than neural adaptation. That is, the two types of adaptation process are asynchronous. However, for both audition and vision, the total amount of information mediated by the adaptation process (channel capacity) was about the same for both types of processes (2.03 bits vs. 2.1 bits per stimulus for sound intensity; 1.3 vs. 2.0 bits per stimulus for light intensity). Faster acquisition could be accomplished in the whole organism through convergent neural circuits that increase sampling rate by pooling the samples taken by individual receptor systems.     

偏差声音干扰刺激对视觉注意网络系统的影响

结合注意网络测量工具和听视ODDBALL干扰范式,考察偏差声音干扰刺激的出现对注意网络系统不同功能的影响。结果发现,声音刺激序列下,视觉线索唤醒效应均消失; 标准声音刺激下,不存在线索的脱离和转移效应; 偏差声音刺激下,出现线索脱离和转移效应。研究表明,注意网络中的警觉功能具有超通道加工机制,偏差刺激下视觉注意系统中注意定向和线索转移效应的出现表明偏差干扰效应主要由偏差刺激引起的注意脱离和注意转移损耗部分引起。     

Variable criterion analysis of brightness effects in simple reaction time

Using light onset as the stimulus in simple reaction time (SRT), the effect of stimulus intensity was studied in both between-subjects and within-subjects experimental designs. There was a strong intensity effect in both conditions but no significant interaction between the effect of stimulus intensity and the type of design. This differs from previous results with auditory stimuli where such an interaction has been demonstrated. When the criterion parameters of variable criterion theory were evaluated directly, the only significant effect was greater criterion variability in the between-subjects condition. Theoretical functions describing the growth of sensory strength for each intensity had different starting points and were largely parallel, showing only late temporal divergence. This provides an explanation of the rarity, in the SRT literature, of interactions between visual intensity and criterion variables. Correlations illustrating the relations between reaction time (RT) measures and theoretical criterion parameters are presented. Absence of the predicted relation between intensity and RT variability is evidence against theories relating RT to impulse rate treated as a Poisson process.