Cognitive factors and adaptation to auditory-visual discordance

Subjects pointed blindfolded at auditory targets before and after exposure to spatial conflict between the sound of a percussion instrument and images on a TV screen. Four experimental conditions were obtained by combining two levels of realism, in which sound was paired with either the image of the hands playing the instrument or with synchronously modulated light, and two levels of suggestion, in which a dummy loudspeaker from which the subjects had been told the sound would come was placed either in front of the TV screen or on top of the actual hidden loudspeaker. Adaptation occurred in all four conditions, but no difference between them was detected. These results confirm and extend the previous finding that auditory adaptation, unlike the impression of fusion (ventriloquism), is little influenced by cognitive factors.     

Adaptation to auditory-visual discordance and ventriloquism in semirealistic situations

While “recalibration by pairing” is now generally held to be the main process responsible for adaptation to intermodal discordance, the conditions under which pairing of heteromodal data occur in spite of a discordance have not been studied systematically. The question has been explored in the case of auditory-visual discordance. Subjects pointed at auditory targets before and after exposure to auditory and visual data from sources 20° apart in azimuth, in conditions varying by (a) the degree of realism of the context, and (b) the synchronization be-tween auditory and visual data. In Experiment 1, the exposure conditions combined the sound of a percussion instrument (bongos) with either the image on a video monitor of the hands of the player (semirealistic situation) or diffuse light modulated by the sound (nonrealistic situation). Experiment 2 featured a voice and either the image of the face of the speaker or light modulated by the voice, and in both situations either sound and image were exactly syn-chronous or the sound was made to lag by 0.35 sec. Desynchronization was found to reduce adaptation significantly, while degree of realism failed to produce an effect. Answers to a question asked at the end of the testing regarding the location of the sound source suggested that the apparent fusion of the auditory and visual data—the phenomenon called “ventriloquism”— was not affected by the conditions in the same way as adaptation. In Experiment 3, subjects were exposed to the experimental conditions of Experiment 2 and were asked to report their impressions of fusion by pressing a key. The results contribute to the suggestion that pairing of registered auditory and visual locations, the hypothetical process at the basis of recalibration, may be a different phenomenon from conscious fusion.     

Rapid adaptation to auditory-visual spatial disparity

The so-called ventriloquism aftereffect is a remarkable example of rapid adaptative changes in spatial localization caused by visual stimuli. After exposure to a consistent spatial disparity of auditory and visual stimuli, localization of sound sources is systematically shifted to correct for the deviation of the sound from visual positions during the previous adaptation period. In the present study, this aftereffect was induced by presenting, within 17 min, 1800 repetitive noise or pure-tone bursts in combination with synchronized, and 20° disparate flashing light spots, in total darkness. Post-adaptive sound localization, measured by a method of manual pointing, was significantly shifted 2.4° (noise), 3.1° (1 kHz tones), or 5.8° (4 kHz tones) compared with the pre-adaptation condition. There was no transfer across frequencies; that is, shifts in localization were insignificant when the frequencies used for adaptation and the post-adaptation localization test were different. It is hypothesized that these aftereffects may rely on shifts in neural representations of auditory space with respect to those of visual space, induced by intersensory spatial disparity, and may thus reflect a phenomenon of neural short-term plasticity.     

Cross-modal, auditory-visual Stroop interference: a reply to Cowan and Barron (1987)

This series of experiments was performed to verify Cowan and Barron's (1987) reported effect of auditory color-word interference on a visual Stroop task. Extant theory predicts effects of irrelevant speech on visual memory tasks involving immediate, ordered recall of the items. Interference between the two sources of information (visual and auditory) is assumed to be located at the phonological store component of the articulatory loop. Perceptual tasks such as the Stroop task, which do not require use of the articulatory loop for rehearsal purposes, should not be similarly susceptible. The present data fail to replicate Cowan and Barron's findings, and are thus consistent with contemporary theory.     

The kinesthetic fusion effect: Perceptual elimination of spatial discordance in the kinesthetic modality

Three experiments in intrasensory kinesthetic discordance are reported. The treatment required both arms to be outstretched in the midsagittal plane. In the absence of vision, a finger of one hand depressed a button that caused a probe to touch the other arm 12.7 cm closer to the body than the position of the finger. This difference was at first clearly sensed, but very rapidly the disparate positions were perceived to be identical: We have called this “the kinesthetic fusion effect.” It was also shown that a delay of up to 4 sec in probe onset after the finger depressed the button did not diminish the extent of fusion. The phenomenon gave rise to a predictable aftereffect, in which a probe directly opposite to the active finger was sensed to be farther away than the finger. The results were not due either to joint adaptation or to postural aftereffects.     

Cross-modal detection using various temporal and spatial configurations

To better understand temporal and spatial cross-modal interactions, two signal detection experiments were conducted in which an auditory target was sometimes accompanied by an irrelevant flash of light. In the first, a psychometric function for detecting a unisensory auditory target in varying signal-to-noise ratios (SNRs) was derived. Then auditory target detection was measured while an irrelevant light was presented with light/sound stimulus onset asynchronies (SOAs) between 0 and ±700 ms. When the light preceded the sound by 100 ms or was coincident, target detection (d') improved for low SNR conditions. In contrast, for larger SOAs (350 and 700 ms), the behavioral gain resulted from a change in both d' and response criterion (β). However, when the light followed the sound, performance changed little. In the second experiment, observers detected multimodal target sounds at eccentricities of ±8°, and ±24°. Sensitivity benefits occurred at both locations, with a larger change at the more peripheral location. Thus, both temporal and spatial factors affect signal detection measures, effectively parsing sensory and decision-making processes.     

Shape constancy and a perceptual bias towards symmetry

In attempting to judge whether a briefly presented rotated figure was symmetrical (circle or square) or asymmetrical (ellipse or rectangle), a bias towards perceiving symmetry was evident. This bias appears to be perceptual rather than judgmental in origin in that it was affected neither by variations in the probability of occurrence of symmetrical figures nor by familiarization with the stimuli. A characterization of the stimulus similarity space based on the shape-slant invariance hypothesis is presented, and a model which incorporates both the similarity space and the symmetry bias is proposed. The model can account for our paradoxical finding that the more asymmetrical some projected images, the more likely they will be perceived as symmetrical. This model can also account for Massaro’s (1973, 1975) results on shape constancy. However, Massaro’s theory was not supported in that extension in depth did not account for increased difficulty of shape judgments.     

Cross-modal enhancement of perceived brightness: sensory interaction versus response bias

Stein, London, Wilkinson, and Price (1996) reported the presence of cross-modal enhancement of perceived visual intensity: Participants tended to rate weak lights as brighter when accompanied by a concurrent pulse of white noise than when presented alone. In the present study, two methods were used to determine whether the enhancement reflects an early-stage sensory process or a later-stage decisional process, such as a response bias. First, enhancement was eliminated when the noise accompanied the light on only 25% versus 50% of the trials. Second, enhancement was absent when tested with a paired-comparison method. These findings are consistent with the hypothesis that the sound-induced enhancement in judgments of brightness reflects a response bias, rather than an early sensory process--that is, enhancement is the result of a relatively late decisional process.     

A bias interpretation of facilitation in perceptual identification

In a typical perceptual identification task, a word is presented for a few milliseconds and masked; then subjects are asked to report the word. It has been found that an earlier presentation of the test word will improve identification of the test word by as much as 30%. In addition, this facilitation has been shown to be preserved under amnesia. In this article we examine a fundamental question: Is the facilitation the result of bias toward the earlier presented item, an improvement in perceptual sensitivity, or both? The experiments presented here use a forced choice procedure to show that prior presentation of an item biases the subject to choose that item but does not improve discriminability. This result is obtained when the distractor items are visually similar to the target items. When distractors are dissimilar, earlier presentation affects neither bias nor discriminability. Two models of word identification are examined in light of the bias effects, and implications for understanding savings in amnesia are also examined.     

Unconscious memory bias in depression: perceptual and conceptual processes

Mood-congruent memory (MCM) bias in depression was investigated using 4 different implicit memory tests. Two of the implicit tests were perceptually driven, and 2 were conceptually driven. Depressed participants and nondepressed controls were assigned to 1 of 4 implicit memory tests after studying positive and negative adjectives. Results showed no MCM bias in the perceptually driven tests. MCM was demonstrated in 1 of the conceptually driven tests, but only for adjectives that were conceptually encoded. Results support the theory that mood-congruent processes in depression are limited to conceptual processing. However, activation of conceptual processes may not be sufficient for demonstrating mood congruency.     

Cross-modal links in exogenous covert spatial orienting between touch, audition, and vision

Three experiments investigated cross-modal links between touch, audition, and vision in the control of covert exogenous orienting. In the first two experiments, participants made speeded discrimination responses (continuous vs. pulsed) for tactile targets presented randomly to the index finger of either hand. Targets were preceded at a variable stimulus onset asynchrony (150,200, or 300 msec) by a spatially uninformative cue that was either auditory (Experiment 1) or visual (Experiment 2) on the same or opposite side as the tactile target. Tactile discriminations were more rapid and accurate when cue and target occurred on the same side, revealing cross-modal covert orienting. In Experiment 3, spatially uninformative tactile cues were presented prior to randomly intermingled auditory and visual targets requiring an elevation discrimination response (up vs. down). Responses were significantly faster for targets in both modalities when presented ipsilateral to the tactile cue. These findings demonstrate that the peripheral presentation of spatially uninforrnative auditory and visual cues produces cross-modal orienting that affects touch, and that tactile cues can also produce cross-modal covert orienting that affects audition and vision.     

Disentangling perceptual and response bias in unilateral neglect: a methodological proposal

Bisiach and his colleagues have developed a new version of Milner's "landmark task" for the purpose of separating "perceptual" and "response" biases in neglect patients. Subjects are required to decide which is the longer (or the shorter) of the two portions of a pre-bisected horizontal line. The authors proposed two indices to measure perceptual and response bias respectively. However, these indices are not mathematically independent of one another. Moreover, they do not exploit all of the information available in the data, since they do not consider the effect of the different transection locations across trials. We now propose an alternative means of analyzing data from the revised Landmark task, which generates independent estimates of perceptual and response biases. The method and its theoretical foundation are summarized, and illustrative data obtained from brain damaged patients and control subjects are presented.     

Cross-modal training induces changes in spatial representations early in the auditory processing pathway

In the ventriloquism aftereffect, brief exposure to a consistent spatial disparity between auditory and visual stimuli leads to a subsequent shift in subjective sound localization toward the positions of the visual stimuli. Such rapid adaptive changes probably play an important role in maintaining the coherence of spatial representations across the various sensory systems. In the research reported here, we used event-related potentials (ERPs) to identify the stage in the auditory processing stream that is modulated by audiovisual discrepancy training. Both before and after exposure to synchronous audiovisual stimuli that had a constant spatial disparity of 15°, participants reported the perceived location of brief auditory stimuli that were presented from central and lateral locations. In conjunction with a sound localization shift in the direction of the visual stimuli (the behavioral ventriloquism aftereffect), auditory ERPs as early as 100 ms poststimulus (N100) were systematically modulated by the disparity training. These results suggest that cross-modal learning was mediated by a relatively early stage in the auditory cortical processing stream.     

Shifts of the psychometric function: Distinguishing bias from perceptual effects

Morgan, Dillenburger, Raphael, and Solomon have shown that observers can use different response strategies when unsure of their answer, and, thus, they can voluntarily shift the location of the psychometric function estimated with the method of single stimuli (MSS; sometimes also referred to as the single-interval, two-alternative method). They wondered whether MSS could distinguish response bias from a true perceptual effect that would also shift the location of the psychometric function. We demonstrate theoretically that the inability to distinguish response bias from perceptual effects is an inherent shortcoming of MSS, although a three-response format including also an “undecided” response option may solve the problem under restrictive assumptions whose validity cannot be tested with MSS data. We also show that a proper two-alternative forced-choice (2AFC) task with the three-response format is free of all these problems so that bias and perceptual effects can easily be separated out. The use of a three-response 2AFC format is essential to eliminate a confound (response bias) in studies of perceptual effects and, hence, to eliminate a threat to the internal validity of research in this area.     

Cognitive load eliminates the global perceptual bias for unlimited exposure durations

In the present study, we used Navon-type Cognitive Psychology 9: 353–383 (1977) hierarchical patterns to demonstrate that cognitive load eliminates a global perceptual bias and enhances the representation of local elements at unlimited exposure durations. We added a cognitive-load manipulation to Kimchi and Palmer’s Journal of Experimental Psychology: Human Perception and Performance 8:521–535 (1982) similarity-matching experiment with hierarchical patterns, and presented the stimuli for either unlimited or limited exposure durations. When exposures were unlimited, we demonstrated that observers exhibited a global bias under low, but not under high, cognitive load (Exp. 2). When exposures were limited, however, cognitive load exerted no effect, and the global bias remained (Exp. 1). We suggest that (1) these findings are best reconciled by proposing two stages in the representation of global structure, namely construction and maintenance; (2) the construction and maintenance stages are isolated, respectively, by limited-duration and unlimited-duration paradigms; and (3) cognitive processes play an integral role only in the maintenance stage. Given that real-world vision is not driven by a series of brief stimulus exposures, and is therefore likely to reflect maintenance processes, we argue that unlimited-exposure paradigms are more suitable for addressing real-world perceptual biases. When unlimited-exposure paradigms are used, cognitive load eliminates the commonly reported global bias.     

Overcoming default categorical bias in spatial memory

In the present study, we investigated whether a strong default categorical bias can be overcome in spatial memory by using alternative membership information. In three experiments, we tested location memory in a circular space while providing participants with an alternative categorization. We found that visual presentation of the boundaries of the alternative categories (Experiment 1) did not induce the use of the alternative categories in estimation. In contrast, visual cuing of the alternative category membership of a target (Experiment 2) and unique target feature information associated with each alternative category (Experiment 3) successfully led to the use of the alternative categories in estimation. Taken together, the results indicate that default categorical bias in spatial memory can be overcome when appropriate cues are provided. We discuss how these findings expand the category adjustment model (Huttenlocher, Hedges, & Duncan, 1991) in spatial memory by proposing a retrieval-based category adjustment (RCA) model.