Multisensory temporal order judgments: when two locations are better than one

In Experiment 1, participants were presented with pairs of stimuli (one visual and the other tactile) from the left and/or right of fixation at varying stimulus onset asynchronies and were required to make unspeeded temporal order judgments (TOJs) regarding which modality was presented first. When the participants adopted an uncrossed-hands posture, just noticeable differences (JNDs) were lower (i.e., multisensory TOJs were more precise) when stimuli were presented from different positions, rather than from the same position. This spatial redundancy benefit was reduced when the participants adopted a crossed-hands posture, suggesting a failure to remap visuotactile space appropriately. In Experiment 2, JNDs were also lower when pairs of auditory and visual stimuli were presented from different positions, rather than from the same position. Taken together, these results demonstrate that people can use redundant spatial cues to facilitate their performance on multisensory TOJ tasks and suggest that previous studies may have systematically overestimated the precision with which people can make such judgments. These results highlight the intimate link between spatial and temporal factors in determining our perception of the multimodal objects and events in the world around us.     

The auditory redundant signals effect: An influence of number of stimuli or number of percepts?

In two experiments, we examined simple reaction times (RTs) for detection of the onsets and offsets of auditory stimuli. Both experiments assessed the redundant signals effect (RSE), which is traditionally defined as the reduction in RT associated with the presentation of two redundant stimuli, rather than a single stimulus. In Experiment 1, with two identical tones presented via headphones to the left ear, right ear, or both, no RSE was found in responding to tone onsets, but a large RSE was found in responding to their offsets. In Experiment 2, with a pure tone and white noise as the two stimulus alternatives, RSEs were found for responding to both onsets and offsets. The results support the notion that the occurrence of an RSE depends on the number of percepts, rather than the number of stimuli, and on the requirement to respond to stimulus onsets versus offsets. The parallel grains model (Miller & Ulrich, 2003) provides one possible account of this pattern of results.     

The influence of synchronous audiovisual distractors on audiovisual temporal order judgments

Participants made unspeeded temporal order judgments (TOJs) regarding which occurred first, anauditory or a visual target stimulus, when they were presented at a variety of different stimulus onset asynchronies. The target stimuli were presented either in isolation or positioned randomly among a stream of three synchronous audiovisual distractors. The largest just noticeable differences were reported when the targets were presented in the middle of the distractor stream. When the targets were presented at the beginning of the stream, performance was no worse than when the audiovisual targets were presented in isolation. Subsequent experiments revealed that performance improved somewhat when the position of the target was fixed or when the target was made physically distinctive from the distractors. These results show that audiovisual TOJs are impaired by the presence of audiovisual distractors and that this cost can be ameliorated by directing attention to the appropriate temporal position within the stimulus stream.     

On the relationship between the redundant signals effect and temporal order judgments: parametric data and a new model

Paradigms used to study the time course of the redundant signals effect (RSE; J. O. Miller, 1986) and temporal order judgments (TOJs) share many important similarities and address related questions concerning the time course of sensory processing. The author of this article proposes and tests a new aggregate diffusion-based model to quantitatively explain both the RSE and TOJs and the relationship between them. Parametric data (13 stimulus onset asynchronies) from an experiment with pairs of visual stimuli (626-nm LEDs) confirm that, relative to central signals (3 degrees), peripheral signals (35 degrees) yield slower reaction times, more strongly modulated RSE time-course functions, and flatter TOJ psychometric functions. All of these qualitative features are well captured, even in quantitative detail, by the aggregate diffusion model.     

Fusion prevents the redundant signals effect: evidence from stereoscopically presented stimuli

In a simple reaction time (RT) experiment, visual stimuli were stereoscopically presented either to one eye (single stimulation) or to both eyes (redundant stimulation), with brightness matched for single and redundant stimulations. Redundant stimulation resulted in two separate percepts when noncorresponding retinal areas were stimulated, whereas it resulted in a single fused percept when corresponding areas were stimulated. With stimulation of noncorresponding areas, mean RT was shorter to redundant than to single stimulation, replicating the redundant signals effect (RSE) commonly found with visual stimuli. With stimulation of corresponding areas, however, no RSE was observed. This suggests that the RSE is driven by the number of percepts rather than by the number of stimulated receptors or sensory organs. These results are consistent with previous findings in the auditory modality and have implications for models of the RSE.     

Multisensory prior entry.

Despite 2 centuries of research, the question of whether attending to a sensory modality speeds the perception of stimuli in that modality has yet to be resolved. The authors highlight weaknesses inherent in this previous research and report the results of 4 experiments in which a novel methodology was used to investigate the effects on temporal order judgments (TOJs) of attending to a particular sensory modality or spatial location. Participants were presented with pairs of visual and tactile stimuli from the left and/or right at varying stimulus onset asynchronies and were required to make unspeeded TOJs regarding which stimulus appeared first. The results provide the strongest evidence to date for the existence of multisensory prior entry and support previous claims for attentional biases toward the visual modality and toward the right side of space. These findings have important implications for studies in many areas of human and animal cognition.     

Temporal order and tactile patterns

Temporal order judgments (TOJs) were obtained for tactile stimuli presented to subjects' fingerpads. In one set of measurements, pairs of spatial patterns were presented successively to a single fingerpad (same-site condition), to two fingers on the same hand (ipsilateral condition), or to two fingers on opposite hands (bilateral condition). The subjects were instructed to report which one of the two patterns was presented first. TOJs were more accurate in the same-site condition than in either the ipsilateral or the bilateral conditions. In the ipsilateral and bilateral conditions, performance improved when judging which one of two locations received a stimulus first, although performance levels were still lower than in the same-site condition. Increasing the size of the pattern set from which the two patterns to be judged were drawn had only a slight effect on same-site performance and no effect on ipsilateral/bilateral performance; however, changing the nature of the patterns had a considerable effect on same-site performance and a smaller effect on ipsilateral/bilateral performance. Introducing an intensity imbalance between members of the pair of stimuli also had a large effect on same-site TOJs: a less intense stimulus tended to be judged as being presented first. In the bilateral condition, however, there was a small effect in the reverse direction: more intense stimuli tended to be judged as being presented first. The intensity imbalance had no effect in the ipsilateral condition. The results suggest that different mechanisms are responsible for TOJs for patterns presented to the same-site and to separate sites and, furthermore, that separate sites may constitute separate channels for spatial information.     

Effects of redundant auditory stimuli on reaction time

Auditory redundancy gains were assessed in two experiments in which a simple reaction time task was used. In each trial, an auditory stimulus was presented to the left ear, to the right ear, or simultaneously to both ears. The physical difference between auditory stimuli presented to the two ears was systematically increased across experiments. No redundancy gains were observed when the stimuli were identical pure tones or pure tones of different frequencies (Experiment 1). A clear redundancy gain and evidence of coactivation were obtained, however, when one stimulus was a pure tone and the other was white noise (Experiment 2). Experiment 3 employed a two-alternative forced choice localization task and provided evidence that dichotically presented pure tones of different frequencies are apparently integrated into a single percept, whereas a pure tone and white noise are not fused. The results extend previous findings of redundancy gains and coactivation with visual and bimodal stimuli to the auditory modality. Furthermore, at least within this modality, the results indicate that redundancy gains do not emerge when redundant stimuli are integrated into a single percept.     

Reaction time and temporal serial judgment: corroboration or dissociation?]

Dissociations between a motor response and the subject's verbal report have been reported from various experiments that investigated special experimental effects (e.g., metacontrast or induced motion). To examine whether similar dissociations can also be observed under standard experimental conditions, we compared reaction times (RT) and temporal order judgments (TOJ) to visual and auditory stimuli of three intensity levels. Data were collected from six subjects, each of which served for nine sessions. The results showed a strong, highly significant modality dissociation: While RTs to auditory stimuli were shorter than RTs to visual stimuli, the TOJ data indicated longer processing times for auditory than for visual stimuli. This pattern was found over the whole range of intensities investigated. Light intensity had similar effects on RT and TOJ, while there was a marginally significant tendency of tone intensity to affect RT more strongly than TOJ. It is concluded that modality dissociation is an example of "direct parameter specification", where the pathway from stimulus to response in the simple RT experiment is (at least partially) separate from the pathway that leads to a conscious, reportable representation. Two variants of this notion and alternatives to it are discussed.     

On the locus of visual dominance

A series of six experiments explored the dominance of vision over audition reported by Colavita (1974). We first confirmed the existence of visual dominance in a paradigm somewhat different from Colavita’s: Mean reaction time (RT) to a light was found to be faster than to a simultaneously presented tone, even though the stimuli were equated in subjective intensity and even though RT to the tone presented alone was faster than to the light presented alone. Additional experiments showed that when subjects did not have to respond to light, tone RT was equal or faster (intersensory facilitation) when a light was present than when it was not. These findings suggest that sensory or perceptual processing of the tone is not affected by the light, i.e., that visual dominance is nonsensory in locus and depends on the relevance of the light stimulus. This interpretation was reinforced by other findings which showed that the degree of visual dominance was sensitive to the probability of light, tone, and light-plus-tone trials and to instructions to attend to a specific modality, but was not sensitive to the intensity of the light.     

Multisensory integration of vision and touch in nonspatial feature discrimination tasks

Multisensory integration of nonspatial features between vision and touch was investigated by examining the effects of redundant signals of visual and tactile inputs. In the present experiments, visual letter stimuli and/or tactile letter stimuli were presented, which participants were asked to identify as quickly as possible. The results of Experiment 1 demonstrated faster reaction times for bimodal stimuli than for unimodal stimuli (the redundant signals effect (RSE)). The RSE was due to coactivation of figural representations from the visual and tactile modalities. This coactivation did not occur for a simple stimulus detection task (Experiment 2) or for bimodal stimuli with the same semantic information but different physical stimulus features (Experiment 3). The findings suggest that the integration process might occur at a relatively early stage of object-identification prior to the decision level.     

Audiotactile temporal order judgments

We report a series of three experiments in which participants made unspeeded 'Which modality came first?' temporal order judgments (TOJs) to pairs of auditory and tactile stimuli presented at varying stimulus onset asynchronies (SOAs) using the method of constant stimuli. The stimuli were presented from either the same or different locations in order to explore the potential effect of redundant spatial information on audiotactile temporal perception. In Experiment 1, the auditory and tactile stimuli had to be separated by nearly 80 ms for inexperienced participants to be able to judge their temporal order accurately (i.e., for the just noticeable difference (JND) to be achieved), no matter whether the stimuli were presented from the same or different spatial positions. More experienced psychophysical observers (Experiment 2) also failed to show any effect of relative spatial position on audiotactile TOJ performance, despite having much lower JNDs (40 ms) overall. A similar pattern of results was found in Experiment 3 when silent electrocutaneous stimulation was used rather than vibrotactile stimulation. Thus, relative spatial position seems to be a less important factor in determining performance for audiotactile TOJ than for other modality pairings (e.g., audiovisual and visuotactile).     

Impact of contingency manipulations on accessory stimulus effects

Accessory tone stimuli facilitate response performance despite being irrelevant for the current task. In order to investigate which processes are affected by accessory stimuli, we presented accessory tones in a simple response time (RT) task while varying the contingencies between accessory stimulation and either responses(Experiment 1) or stimulus conditions (Experiment 2). Accessory tones speeded up responding to a larger degree when they were conjointly presented within go compared with no-go trials. In contrast, contingency variation with stimulus conditions did not alter the impact of accessory stimuli. Additionally, accessory tones increased response force. Thus, we conclude that in simple RT tasks accessory tones influence response-related stages such as response selection and response execution rather than perceptual processes.     

Redundancy gain with static versus moving hands: a test of the hemispheric coactivation model

A simple reaction time (RT) experiment was conducted to test the hypothesis that redundancy gain arises partly because of hemispheric coactivation. Stimuli were presented to the left or right of fixation, or redundantly to both, and the participants had to make keypress responses as rapidly as possible to stimulus onset. A "static" condition, in which the participants held their hands at rest, was compared with a "dynamic" condition, in which they moved their hands back and forth in an oscillating motion prior to stimulus onset. As predicted from the hypothesis of hemispheric coactivation, redundancy gain, measured as the decrease in mean RT to redundant stimuli as compared with single stimuli, was smaller in the dynamic condition than in the static one.     

Tactile--visual temporal ventriloquism: no effect of spatial disparity

Participants made visual temporal order judgments (TOJs) about which of two lights appeared first while task-irrelevant vibrotactile stimuli delivered to the index finger were presented before the first and after the second light. Temporally misaligned tactile stimuli captured the onsets of the lights, thereby improving sensitivity on the visual TOJ task, indicative of tactile-visual (TV) temporal ventriloquism (Experiment 1). The size of this effect was comparable to auditory-visual (AV) temporal ventriloquism (Experiment 2). Spatial discordance between the TV stimuli, as in the AV case, did not harm the effect (Experiments 3 and 4). TV stimuli thus behaved like AV stimuli, demonstrating that spatial co-occurrence is not a necessary constraint for intersensory pairing to occur.     

Tactile—visual temporal ventriloquism: No effect of spatial disparity

Participants made visual temporal order judgments (TOJs) about which of two lights appeared first while taskirrelevant vibrotactile stimuli delivered to the index finger were presented before the first and after the second light. Temporally misaligned tactile stimuli captured the onsets of the lights, thereby improving sensitivity on the visual TOJ task, indicative of tactile-visual (TV) temporal ventriloquism (Experiment 1). The size of this effect was comparable to auditory-visual (AV) temporal ventriloquism (Experiment 2). Spatial discordance between the TV stimuli, as in the AV case, did not harm the effect (Experiments 3 and 4). TV stimuli thus behaved like AV stimuli, demonstrating that spatial co-occurrence is not a necessary constraint for intersensory pairing to occur.     

Differential redundancy gain in onset detection versus offset detection

In a visual simple reaction time paradigm with attention divided between the left and right visual fields, redundancy gain refers to the finding of faster responses to stimuli presented in both fields than to single stimuli. The present study investigated whether the effects of low-level perceptual processing affect the redundancy gain by comparing the detection of onsets versus offsets. In different blocks, participants responded to left and right visual field stimuli that either appeared (onset) or disappeared (offset), with a single stimulus change in some trials and two redundant stimulus changes in others. With onset stimuli, the results replicated previous redundancy gain effects. In contrast, there was much less redundancy gain when participants responded to stimulus offsets. This finding suggests that redundancy gain is sensitive to low-level perceptual characteristics, such as onset versus offset presentation of stimuli.     

Redundant visual signals boost saccade execution

The redundant signal effect (RSE) refers to the fact that human beings react more quickly to a pair of stimuli than to only one stimulus. In previous studies of the RSE in the oculomotor system, bimodal signals have been used as the goal of the saccade. In consistency with studies using manual response times (RTs), saccadic RTs have been shown to be shorter for redundant multimodal stimuli than for single unimodal stimuli. In the present experiments, we extended these findings by demonstrating an RSE in the saccadic system elicited only by unimodal visual stimuli. In addition, we found that shorter saccadic RTs were accompanied by an increased saccadic peak velocity. The present results are of relevance for neurophysiological models of saccade execution, since the boost of saccades was elicited by two visual transients (acting as a “go” signal) that were presented not at the goal of the saccade but at various other locations.     

Visual dominance in the pigeon

In Experiment 1, three pigeons were trained to obtain grain by depressing one foot treadle in the presence of a 746-Hertz tone stimulus and by depressing a second foot treadle in the presence of a red light stimulus. Intertrial stimuli included white light and the absence of tone. The latencies to respond on auditory element trials were as fast, or faster, than on visual element trials, but pigeons always responded on the visual treadle when presented with a compound stimulus composed of the auditory and visual elements. In Experiment 2, pigeons were trained on the auditory-visual discrimination task using as trial stimuli increases in the intensity of auditory or visual intertrial stimuli. Again, pigeons showed visual dominance on subsequent compound stimulus test trials. In Experiment 3, on compound test trials, the onset of the visual stimulus was delayed relative to the onset of the auditory stimulus. Visual treadle responses generally occurred with delay intervals of less than 500 milliseconds, and auditory treadle responses generally occurred with delay intervals of greater than 500 milliseconds. The results are discussed in terms of Posner, Nissen, and Klein's (1976) theory of visual dominance in humans.     

Evaluating the influence of the 'unity assumption' on the temporal perception of realistic audiovisual stimuli

Vatakis, A. and Spence, C. (in press) [Crossmodal binding: Evaluating the 'unity assumption' using audiovisual speech stimuli. Perception &Psychophysics] recently demonstrated that when two briefly presented speech signals (one auditory and the other visual) refer to the same audiovisual speech event, people find it harder to judge their temporal order than when they refer to different speech events. Vatakis and Spence argued that the 'unity assumption' facilitated crossmodal binding on the former (matching) trials by means of a process of temporal ventriloquism. In the present study, we investigated whether the 'unity assumption' would also affect the binding of non-speech stimuli (video clips of object action or musical notes). The auditory and visual stimuli were presented at a range of stimulus onset asynchronies (SOAs) using the method of constant stimuli. Participants made unspeeded temporal order judgments (TOJs) regarding which modality stream had been presented first. The auditory and visual musical and object action stimuli were either matched (e.g., the sight of a note being played on a piano together with the corresponding sound) or else mismatched (e.g., the sight of a note being played on a piano together with the sound of a guitar string being plucked). However, in contrast to the results of Vatakis and Spence's recent speech study, no significant difference in the accuracy of temporal discrimination performance for the matched versus mismatched video clips was observed. Reasons for this discrepancy are discussed.