Cross-modal bias and perceptual fusion with auditory-visual spatial discordance

Investigations of situations involving spatial discordance between auditory and visual data which can otherwise be attributed to a common origin have revealed two main phenomena:cross-modal bias andperceptual fusion (or ventriloquism). The focus of the present study is the relationship between these two. The question asked was whether bias occurred only with fusion, as is predicted by some accounts of reactions to discordance, among them those based on cuesubstitution. The approach consisted of having subjects, on each trial, both point to signals in one modality in the presence of conflicting signals in the other modality and produce same-different origin judgments. To avoid the confounding of immediate effects with cumulative adaptation, which was allowed in most previous studies, the direction and amplitude of discordance was varied randomly from trial to trial. Experiment 1, which was a pilot study, showed that both visual bias of auditory localization and auditory bias of visual localization can be observed under such conditions. Experiment 2, which addressed the main question, used a method which controls for the selection involved in separating fusion from no-fusion trials and showed that the attraction of auditory localization by conflicting visual inputs occurs even when fusion is not reported. This result is inconsistent with purely postperceptual views of cross-modal interactions. The question could not be answered for auditory bias of visual localization, which, although significant, was very small in Experiment 1 and fell below significance under the conditions of Experiment 2.     

Cognitive factors and adaptation to auditory-visual discordance

Subjects pointed blindfolded at auditory targets before and after exposure to spatial conflict between the sound of a percussion instrument and images on a TV screen. Four experimental conditions were obtained by combining two levels of realism, in which sound was paired with either the image of the hands playing the instrument or with synchronously modulated light, and two levels of suggestion, in which a dummy loudspeaker from which the subjects had been told the sound would come was placed either in front of the TV screen or on top of the actual hidden loudspeaker. Adaptation occurred in all four conditions, but no difference between them was detected. These results confirm and extend the previous finding that auditory adaptation, unlike the impression of fusion (ventriloquism), is little influenced by cognitive factors.     

The aftereffects of ventriloquism: patterns of spatial generalization

We examined how visual recalibration of apparent sound location obtained at a particular location generalizes to untrained locations. Participants pointed toward the origin of tone bursts scattered along the azimuth, before and after repeated exposure to bursts in one particular location, synchronized with point flashes of light a constant distance to their left/right. Adapter tones were presented straight ahead in Experiment 1, and in the left or right periphery in Experiment 2. With both arrangements, different generalization patterns were obtained on the visual distractor's side of the auditory adapter and onthe opposite side. On the distractor side, recalibration generalized following a descending gradient; practically no generalization was observed on the other side. This dependence of generalization patterns on the direction of the discordance imposed during adaptation has not been reported before, perhaps because the experimental designs in use did not allow its observation.     

The aftereffects of ventriloquism: generalization across sound-frequencies

Exposure to synchronous but spatially discordant auditory and visual inputs produces, beyond immediate cross-modal biases, adaptive recalibrations of the respective localization processes that manifest themselves in aftereffects. Such recalibrations probably play an important role in maintaining the coherence of spatial representations across the various spatial senses. The present study is part of a research program focused on the way recalibrations generalize to stimulus values different from those used for adaptation. Considering the case of sound frequency, we recently found that, in contradiction with an earlier report, auditory aftereffects generalize nearly entirely across two octaves. In this new experiment, participants were adapted to an 18 degrees auditory-visual discordance with either 400 or 6400 Hz tones, and their subsequent sound localization was tested across this whole four-octave frequency range. Substantial aftereffects, decreasing significantly with increasing difference between test and adapter frequency, were obtained at all combinations of adapter and test frequency. Implications of these results concerning the functional site at which visual recalibration of auditory localization might take place are discussed.     

Visual recalibration of auditory speech identification: a McGurk aftereffect

The kinds of aftereffects, indicative of cross-modal recalibration, that are observed after exposure to spatially incongruent inputs from different sensory modalities have not been demonstrated so far for identity incongruence. We show that exposure to incongruent audiovisual speech (producing the well-known McGurk effect) can recalibrate auditory speech identification. In Experiment 1, exposure to an ambiguous sound intermediate between /aba/ and /ada/ dubbed onto a video of a face articulating either /aba/ or /ada/ increased the proportion of /aba/ or /ada/ responses, respectively, during subsequent sound identification trials. Experiment 2 demonstrated the same recalibration effect or the opposite one, fewer /aba/ or /ada/ responses, revealing selective speech adaptation, depending on whether the ambiguous sound or a congruent nonambiguous one was used during exposure. In separate forced-choice identification trials, bimodal stimulus pairs producing these contrasting effects were identically categorized, which makes a role of postperceptual factors in the generation of the effects unlikely.     

An examination of the relationship between visual capture and prism adaptation

The phenomena of prismatically induced “visual capture” and adaptation of the hand were compared. In Experiment 1, it was demonstrated that when the subject’s hand was transported for him by the experimenter (passive movement) immediately preceding the measure of visual capture, the magnitude of the immediate shift in felt limb position (visual capture) was enhanced relative to when the subject moved the hand himself (active movement). In Experiment 2, where the dependent measure was adaptation of the prism-exposed hand, the opposite effect was produced by the active/passive manipulation. It appears, then, that different processes operate to produce visual capture and adaptation. It was speculated that visual capture represents an immediate weighting of visual over proprioceptive input as a result of the greater precision of vision and/or the subject’s tendency to direct his attention more heavily to this modality. In contrast, prism adaptation is probably a recalibration of felt limb position in the direction of vision, induced by the presence of a registered discordance between visual and proprioceptive inputs.     

Perceptual load influences auditory space perception in the ventriloquist aftereffect

A period of exposure to trains of simultaneous but spatially offset auditory and visual stimuli can induce a temporary shift in the perception of sound location. This phenomenon, known as the ‘ventriloquist aftereffect’, reflects a realignment of auditory and visual spatial representations such that they approach perceptual alignment despite their physical spatial discordance. Such dynamic changes to sensory representations are likely to underlie the brain’s ability to accommodate inter-sensory discordance produced by sensory errors (particularly in sound localization) and variability in sensory transduction. It is currently unknown, however, whether these plastic changes induced by adaptation to spatially disparate inputs occurs automatically or whether they are dependent on selectively attending to the visual or auditory stimuli. Here, we demonstrate that robust auditory spatial aftereffects can be induced even in the presence of a competing visual stimulus. Importantly, we found that when attention is directed to the competing stimuli, the pattern of aftereffects is altered. These results indicate that attention can modulate the ventriloquist aftereffect.     

Children do not recalibrate motor‐sensory temporal order after exposure to delayed sensory feedback

Prolonged adaptation to delayed sensory feedback to a simple motor act (such as pressing a key) causes recalibration of sensory‐motor synchronization, so instantaneous feedback appears to precede the motor act that caused it (Stetson, Cui, Montague & Eagleman, 2006). We investigated whether similar recalibration occurs in school‐age children. Although plasticity may be expected to be even greater in children than in adults, we found no evidence of recalibration in children aged 8–11 years. Subjects adapted to delayed feedback for 100 trials, intermittently pressing a key that caused a tone to sound after a 200 ms delay. During the test phase, subjects responded to a visual cue by pressing a key, which triggered a tone to be played at variable intervals before or after the keypress. Subjects judged whether the tone preceded or followed the keypress, yielding psychometric functions estimating the delay when they perceived the tone to be synchronous with the action. The psychometric functions also gave an estimate of the precision of the temporal order judgment. In agreement with previous studies, adaptation caused a shift in perceived synchrony in adults, so the keypress appeared to trail behind the auditory feedback, implying sensory‐motor recalibration. However, school children of 8 to 11 years showed no measureable adaptation of perceived simultaneity, even after adaptation with 500 ms lags. Importantly, precision in the simultaneity task also improved with age, and this developmental trend correlated strongly with the magnitude of recalibration. This suggests that lack of recalibration of sensory‐motor simultaneity after adaptation in school‐age children is related to their poor precision in temporal order judgments. To test this idea we measured recalibration in adult subjects with auditory noise added to the stimuli (which hampered temporal precision). Under these conditions, recalibration was greatly reduced, with the magnitude of recalibration strongly correlating with temporal precision.     

Effects of visual attentional load on low-level auditory scene analysis

The sharing of processing resources between the senses was investigated by examining the effects of visual task load on auditory event-related brain potentials (ERPs). In Experiment 1, participants completed both a zero-back and a one-back visual task while a tone pattern or a harmonic series was presented. N1 and P2 waves were modulated by visual task difficulty, but neither mismatch negativity (MMN) elicited by deviant stimuli from the tone pattern nor object-related negativity (ORN) elicited by mistuning from the harmonic series was affected. In Experiment 2, participants responded to identity (what) or location (where) in vision, while ignoring sounds alternating in either pitch (what) or location (where). Auditory ERP modulations were consistent with task difficulty, rather than with task specificity. In Experiment 3, we investigated auditory ERP generation under conditions of no visual task. The results are discussed with respect to a distinction between process-general (N1 and P2) and processspecific (MMN and ORN) auditory ERPs.     

Can pupillometry index auditory attentional capture in contexts of active visual processing?

The rare presentation of a sound that deviates from the auditory background tends to capture attention, which is known to impede cognitive functioning. Such disruption is usually measured using performance on a concurrent visual task. Growing evidence recently showed that the pupillary dilation response (PDR) could index the attentional response triggered by a deviant sound. Given that the pupil diameter is sensitive to several vision-related factors, it is unclear whether the PDR could serve to study attentional capture in such contexts. Hence, the present study aimed at verifying whether the PDR can be used as a proxy for auditory attentional capture while a visual serial recall task (Experiment 1) or a reading comprehension task (Experiment 2) – respectively producing changes in luminance and gaze position – is being performed. Results showed that presenting a deviant sound within steady-state standard sounds elicited larger PDRs than a standard sound. Moreover, the magnitude of these PDRs was positively related to the amount of performance disruption produced by deviant sounds in Experiment 1. Performance remained unaffected by the deviants in Experiment 2, thereby implying that the PDR may be a more sensitive attention-capture index than behavioural measures. These results suggest that the PDR can be used to assess attentional capture by a deviant sound in contexts where the pupil diameter can be modulated by the visual environment.     

Effects of varying modality,surface features,and retention interval on priming in word-fragment completion

Subjects in Experiment i studied a list of words under varying presentation conditions (visual or auditory) and in two typographies within the visual condition (typed or hand printed) and then received a word-fragment completion test (e.g., —YS—E—Y formystery) in which the test cues also varied in typography. The main findings were that (1) priming occurred for all study items, relative to nonstudied items, but greater priming occurred for visual than for auditory presentation, and (2) performance in the visual conditions was better when typographies matched between study and test than when the typographies mismatched, but only for words studied in hand-printed form. These findings were generally replicated when the test was delayed 1 week, although priming declined across this retention interval (Experiment 2). In Experiment 3 subjects studied words that were either in focus or blurred and showed greater priming when test fragments were presented in the same manner as at study. Priming in the word-fragment completion task depends on matching surface characteristics of items between study and test and exemplifies the requirement of performing similar mental operations at study and test for maximizing performance (transfer-appropriate processing).     

Experiencing the Cross-Sensory Error Signal During Movement Leads to Proprioceptive Recalibration

Abstract

Reaching to targets in a virtual reality environment with misaligned visual feedback of the hand results in changes in movements (visuomotor adaptation) and sense of felt hand position (proprioceptive recalibration). We asked if proprioceptive recalibration arises even when the misalignment between visual and proprioceptive estimates of hand position is only experienced during movement. Participants performed a “shooting task” through the targets with a cursor that was rotated 30° clockwise relative to hand motion. Results revealed that, following training on the shooting task, participants adapted their reaches to all targets by approximately 16° and recalibrated their sense of felt hand position by 8°. Thus, experiencing a sensory misalignment between visual and proprioceptive estimates of hand position during movement leads to proprioceptive recalibration.     

Investigations of hemispheric specialization of self-voice recognition

Three experiments investigated functional asymmetries related to self-recognition in the domain of voices. In Experiment 1, participants were asked to identify one of three presented voices (self, familiar or unknown) by responding with either the right or the left-hand. In Experiment 2, participants were presented with auditory morphs between the self-voice and a familiar voice and were asked to perform a forced-choice decision on speaker identity with either the left or the right-hand. In Experiment 3, participants were presented with continua of auditory morphs between self- or a familiar voice and a famous voice, and were asked to stop the presentation either when the voice became "more famous" or "more familiar/self". While these experiments did not reveal an overall hand difference for self-recognition, the last study, with improved design and controls, suggested a right-hemisphere advantage for self-compared to other-voice recognition, similar to that observed in the visual domain for self-faces.     

Effects of multimodal presentation and stimulus familiarity on auditory and visual processing

Two experiments examined the effects of multimodal presentation and stimulus familiarity on auditory and visual processing. In Experiment 1, 10-month-olds were habituated to either an auditory stimulus, a visual stimulus, or an auditory-visual multimodal stimulus. Processing time was assessed during the habituation phase, and discrimination of auditory and visual stimuli was assessed during a subsequent testing phase. In Experiment 2, the familiarity of the auditory or visual stimulus was systematically manipulated by prefamiliarizing infants to either the auditory or visual stimulus prior to the experiment proper. With the exception of the prefamiliarized auditory condition in Experiment 2, infants in the multimodal conditions failed to increase looking when the visual component changed at test. This finding is noteworthy given that infants discriminated the same visual stimuli when presented unimodally, and there was no evidence that multimodal presentation attenuated auditory processing. Possible factors underlying these effects are discussed.     

Modification of depth judgment following exposures to magnification of uniocular image: Are changes in perceived absolute distance and registered direction of gaze involved?

Two studies were designed to determine whether the perceptual learning that has been demonstrated to occur during exposure to uniocular image magnification can be explained by either a modification in the perception of egocentric distance or the direction of gaze. Experiment 1 was designed to determine whether exposure to uniocular image magnification produces changes in perceived absolute distance. Experiment 2 tested the possibility that exposure to uniocular image magnification modifies the registration of direction of gaze. The results showed that, despite the occurrence of adaptive shifts in perceived depth, no significant changes in perceived absolute distance or in registered direction of gaze occur. These findings bolster confidence in the hypothesis that adaptation to uniocular image magnification is the result of a recalibration of retinal disparity.     

Adaptation to visual and proprioceptive rearrangement: Origin of the differential effectiveness of active and passive movements

In Experiment 1, subjects exposed to a discordance between the visual and ”proprioceptive” locations of external targets were found to exhibit aftereffects when later pointing without sight of their hands at visual targets. Aftereffects occur both when the discordance is introduced in the traditional fashion by displacing the visual locations of targets and when the proprioceptive locations of targets are displaced. These observations indicate that there is nothing unique about the visual rearrangement paradigm—the crucial factor determining whether adaptation will be elicited is the presence of a discordance in the positional information being conveyed over two different sensory modalities. In a second experiment, the effectiveness of active and passive movements in eliciting adaptation was studied using an experimental paradigm in which subjects were exposed to a systematic discordance between the visual and proprioceptive locations of external targets without ever being permitted sight of their hands; a superiority of active movements was observed, just as is usually found in visual rearrangement experiments in which sight of the hand is permitted. Evidence is presented that the failure of passive movements to elicit adaptation is related to a deterioration in accuracy of position sense information during passive limb movement.     

试次历史对跨通道非空间返回抑制的影响

个体对刺激的反应不仅受刺激本身的影响, 还会受到先前刺激的影响, 表现为对当前试次中刺激的反应会受到前一试次的影响, 即试次历史。本研究采用“线索-中性线索-靶子”范式探讨前一试次有效性对跨通道的非空间返回抑制的影响。实验1通过连续两个试次间的线索有效性考察在跨通道非空间返回抑制中试次历史的影响。为了在跨通道非空间返回抑制中减小试次历史的影响, 实验2通过延长试次间时间间隔考察跨通道非空间返回抑制中试次历史的作用是否减小。结果发现, 前一试次线索无效时, 当前试次中的返回抑制效应量显著小于前一试次有效时, 这种影响会根据试次中线索和靶子通道的不同而不同。并且当延长试次间的时间间隔可以有效地减少前一试次对当前试次的影响。因此本研究表明, 试次历史能够对跨通道非空间返回抑制产生影响, 并且这种影响可以通过增大试次间时间间隔来减小。     

On the role of eye movements and saccade preparation in generating auditory inhibition of return.

In three experiments, listeners were required to either localize or identify the second of two successive sounds. The first sound (the cue) and the second sound (the target) could originate from either the same or different locations, and the interval between the onsets of the two sounds (Stimulus Onset Asynchrony, SOA) was varied. Sounds were presented out of visual range at 135 azimuth left or right. In Experiment 1, localization responses were made more quickly at 100 ms SOA when the target sounded from the same location as the cue (i.e., a facilitative effect), and at 700 ms SOA when the target and cue sounded from different locations (i.e., an inhibitory effect). In Experiments 2 and 3, listeners were required to monitor visual information presented directly in front of them at the same time as the auditory cue and target were presented behind them. These two experiments differed in that in order to perform the visual task accurately in Experiment 3, eye movements to visual stimuli were required. In both experiments, a transition from facilitation at a brief SOA to inhibition at a longer SOA was observed for the auditory task. Taken together these results suggest that location-based auditory IOR is not dependent on either eye movements or saccade programming to sound locations.     

Auditory temporal cues can modulate visual representational momentum

In representational momentum (RM), the final position of a moving target is mislocalized in the direction of motion. Here, the effect of a concurrent sound on visual RM was demonstrated. A visual stimulus moved horizontally and disappeared at unpredictable positions. A complex tone without any motion cues was presented continuously from the beginning of the visual motion. As compared with a silent condition, the RM magnitude increased when the sound lasted longer than and decreased when it did not last as long as the visual motion. However, the RM was unchanged when a brief complex tone was presented before or after the target disappeared (Experiment 2) or when the onset of the long-lasting sound was not synchronized with that of the visual motion (Experiments 3 and 4). These findings suggest that visual motion representation can be modulated by a sound if the visual motion information is firmly associated with the auditory information.     

Stimulus Parameters Underlying Sound-Symbolic Mapping of Auditory Pseudowords to Visual Shapes

Sound symbolism refers to non-arbitrary mappings between the sounds of words and their meanings and is often studied by pairing auditory pseudowords such as “maluma” and “takete” with rounded and pointed visual shapes, respectively. However, it is unclear what auditory properties of pseudowords contribute to their perception as rounded or pointed. Here, we compared perceptual ratings of the roundedness/pointedness of large sets of pseudowords and shapes to their acoustic and visual properties using a novel application of representational similarity analysis (RSA). Representational dissimilarity matrices (RDMs) of the auditory and visual ratings of roundedness/pointedness were significantly correlated crossmodally. The auditory perceptual RDM correlated significantly with RDMs of spectral tilt, the temporal fast Fourier transform (FFT), and the speech envelope. Conventional correlational analyses showed that ratings of pseudowords transitioned from rounded to pointed as vocal roughness (as measured by the harmonics-to-noise ratio, pulse number, fraction of unvoiced frames, mean autocorrelation, shimmer, and jitter) increased. The visual perceptual RDM correlated significantly with RDMs of global indices of visual shape (the simple matching coefficient, image silhouette, image outlines, and Jaccard distance). Crossmodally, the RDMs of the auditory spectral parameters correlated weakly but significantly with those of the global indices of visual shape. Our work establishes the utility of RSA for analysis of large stimulus sets and offers novel insights into the stimulus parameters underlying sound symbolism, showing that sound-to-shape mapping is driven by acoustic properties of pseudowords and suggesting audiovisual cross-modal correspondence as a basis for language users' sensitivity to this type of sound symbolism.