Audiovisual influences on the perception of visual apparent motion: exploring the effect of a single sound

Previous research has shown that irrelevant sounds can facilitate the perception of visual apparent motion. Here the effectiveness of a single sound to facilitate motion perception was investigated in three experiments. Observers were presented with two discrete lights temporally separated by stimulus onset asynchronies from 0 to 350 ms. After each trial, observers classified their impression of the stimuli using a categorisation system. A short sound presented temporally (and spatially) midway between the lights facilitated the impression of motion relative to baseline (lights without sound), whereas a sound presented either before the first or after the second light or simultaneously with the lights did not affect motion impression. The facilitation effect also occurred with sound presented far from the visual display, as well as with continuous-sound that was started with the first light and terminated with the second light. No facilitation of visual motion perception occurred if the sound was part of a tone sequence that allowed for intramodal perceptual grouping of the auditory stimuli prior to the critical audiovisual stimuli. Taken together, the findings are consistent with a low-level audiovisual integration approach in which the perceptual system merges temporally proximate sound and light stimuli, thereby provoking the impression of a single multimodal moving object.     

Crossmodal duration perception involves perceptual grouping, temporal ventriloquism, and variable internal clock rates

Here, we investigate how audiovisual context affects perceived event duration with experiments in which observers reported which of two stimuli they perceived as longer. Target events were visual and/or auditory and could be accompanied by nontargets in the other modality. Our results demonstrate that the temporal information conveyed by irrelevant sounds is automatically used when the brain estimates visual durations but that irrelevant visual information does not affect perceived auditory duration (Experiment 1). We further show that auditory influences on subjective visual durations occur only when the temporal characteristics of the stimuli promote perceptual grouping (Experiments 1 and 2). Placed in the context of scalar expectancy theory of time perception, our third and fourth experiments have the implication that audiovisual context can lead both to changes in the rate of an internal clock and to temporal ventriloquism-like effects on perceived on- and offsets. Finally, intramodal grouping of auditory stimuli diminished any crossmodal effects, suggesting a strong preference for intramodal over crossmodal perceptual grouping (Experiment 5).     

Adaptation to spiral motion in crowding condition

When a single, moving stimulus is presented in the peripheral visual field, its direction of motion can be easily distinguished, but when the same stimulus is flanked by other similar moving stimuli, observers are unable to report its direction of motion. In this condition, known as 'crowding', specific features of visual stimuli do not access conscious perception. The aim of this study was to investigate whether adaptation to spiral motion is preserved in crowding conditions. Logarithmic spirals were used as adapting stimuli. A rotating spiral stimulus (target spiral) was presented, flanked by spirals of the same type, and observers were adapted to its motion. The observers' task was to report the rotational direction of a directionally ambiguous motion (test stimulus) presented afterwards. The directionally ambiguous motion consisted of a pair of spirals flickering in counterphase, which were mirror images of the target spiral. Although observers were not aware of the rotational direction of the target and identified it at chance levels, the direction of rotation reported by the observers during the test phase (motion aftereffect) was contrarotational to the direction of the adapting spiral. Since all contours of the adapting and test stimuli were 90 degrees apart, local motion detectors tuned to the directions of the mirror-image spiral should fail to respond, and therefore not adapt to the adapting spiral. Thus, any motion aftereffect observed should be attributed to adaptation of global motion detectors (ie rotation detectors). Hence, activation of rotation-selective cells is not necessarily correlated with conscious perception.     

Early motion processes and the kinetic depth effect

It has been known for over 30 years that motion information alone is sufficient to yield a vivid impression of three-dimensional object structure. For example, a computer simulation of a transparent sphere, the surface of which is randomly speckled with dots, gives no impression of depth when presented as a stationary pattern on a visual display. As soon as the sphere is made to rotate in a series of discrete steps or frames, its 3-D structure becomes apparent. Three experiments are described which use this stimulus, and find that depth perception in these conditions depends crucially on the spatial and temporal properties of the display:

1. Depth is seen reliably only for between-frame rotations of less than 15°, using two-frame and four-frame sequences.

2. Parametric observations using a wide range of frame durations and inter-frame intervals reveal that depth is seen only for inter-frame intervals below 80 msec and is optimal when the stimulus can be sampled at intervals of about 40-60 msec.

3. Monoptic presentation of two frames of the stimulus is sufficient to yield depth, but the impression is destroyed by dichoptic presentation.

These data are in close agreement with the observed limits of direction perception in experiments using “short-range” stimuli. It is concluded that depth perception in the motion display used in these experiments depends on the outputs of low-level or “short-range” motion detectors.     

A comparison of the effects of spatial separation on apparent motion in the auditory and visual modalities

In the present investigation, the effects of spatial separation on the interstimulus onset intervals (ISOIs) that produce auditory and visual apparent motion were compared. In Experiment 1, subjects were tested on auditory apparent motion. They listened to 50-msec broadband noise pulses that were presented through two speakers separated by one of six different values between 0 degrees and 160 degrees. On each trial, the sounds were temporally separated by 1 of 12 ISOIs from 0 to 500 msec. The subjects were instructed to categorize their perception of the sounds as "single," "simultaneous," "continuous motion," "broken motion," or "succession." They also indicated the proper temporal sequence of each sound pair. In Experiments 2 and 3, subjects were tested on visual apparent motion. Experiment 2 included a range of spatial separations from 6 degrees to 80 degrees; Experiment 3 included separations from .5 degrees to 10 degrees. The same ISOIs were used as in Experiment 1. When the separations were equal, the ISOIs at which auditory apparent motion was perceived were smaller than the values that produced the same experience in vision. Spatial separation affected only visual apparent motion. For separations less than 2 degrees, the ISOIs that produced visual continuous motion were nearly equal to those which produced auditory continuous motion. For larger separations, the ISOIs that produced visual apparent motion increased.     

Seeing Motion and Apparent Motion

In apparent motion experiments, participants are presented with what is in fact a succession of two brief stationary stimuli at two different locations, but they report an impression of movement. Philosophers have recently debated whether apparent motion provides evidence in favour of a particular account of the nature of temporal experience. I argue that the existing discussion in this area is premised on a mistaken view of the phenomenology of apparent motion and, as a result, the space of possible philosophical positions has not yet been fully explored. In particular, I argue that the existence of apparent motion is compatible with an account of the nature of temporal experience that involves a version of direct realism. In doing so, I also argue against two other claims often made about apparent motion, viz. that apparent motion is the psychological phenomenon that underlies motion experience in the cinema, and that apparent motion is subjectively indistinguishable from real motion.     

Second-order reversed phi.

In a first-order reversed-phi motion stimulus (Anstis, 1970), the black-white contrast of successive frames is reversed, and the direction of apparent motion may, under some conditions, appear to be reversed. It is demonstrated here that, for many classes of stimuli, this reversal is a mathematical property of the stimuli themselves, and the real problem is in perceiving forward motion, which involves the second- or third-order motion systems or both. Three classes of novel second-order reversed-phi stimuli (contrast, spatial frequency, and flicker modulation) that are invisible to first-order motion analysis were constructed. In these stimuli, the salient stimulus features move in the forward (feature displacement) direction, but the second-order motion energy model predicts motion in the reversed direction. In peripheral vision, for all stimulus types and all temporal frequencies, all the observers saw only the reversed-phi direction of motion. In central vision, the observers also perceived reversed motion at temporal frequencies above about 4 Hz, but they perceived movement in the forward direction at lower temporal frequencies. Since all of these stimuli are invisible to first-order motion, these results indicate that the second-order reversed-phi stimuli activate two subsequent competing motion mechanisms, both of which involve an initial stage of texture grabbing (spatiotemporal filtering, followed by fullwave rectification). The second-order motion system then applies a Reichardt detector (or equivalently, motion energy analysis) directly to this signal and arrives at the reversed-phi direction. The third-order system marks the location of features that differ from the background (the figure) in a salience map and computes motion in the forward direction from the changes in the spatiotemporal location of these marks. The second-order system's report of reversed movement dominates in peripheral vision and in central vision at higher temporal frequencies, because it has better spatial and temporal resolution than the third-order system, which has a cutoff frequency of 3-4 Hz (Lu & Sperling, 1995b). In central vision, below 3-4 Hz, the third-order system's report of resolvable forward movement of something salient (the figure) dominates the second-order system's report of texture contrast movement.     

Auditory stimulation affects apparent motion1

Abstract: In two experiments, we investigated how the number of auditory stimuli affected the apparent motion induced by visual stimuli. The multiple visual stimuli that induced the apparent motion on the front parallel plane, or in the depth dimension in terms of the binocular disparity cue, were accompanied by multiple auditory stimuli. Observers reported the number of visual stimuli (Experiments 1 and 2) and the displacement of the apparent motion that was defined by the distance between the first and last visual stimuli (Experiment 2). When the number of auditory stimuli was more/less than that of the visual stimuli, observers tended to perceive more/less visual stimuli and a larger/smaller displacement than when the numbers of the auditory and visual stimuli were the same, regardless of the dimension of motion. These results suggest that auditory stimulation may modify the visual processing of motion by modulating the spatiotemporal resolution and extent of the displacement.     

Self-relevance enhances the benefits of attention on perception

ABSTRACT

Considerable efforts have focused on elucidating the influence that self-relevance exerts on perceptual decision-making. To explore this issue further, the current research explored the extent to which stimulus applicability facilitates the benefits of covert attention on early visual processing. In two experiments, we manipulated the personal-relevance of peripheral cues (i.e., geometric shapes) that preceded the appearance of target stimuli (i.e., Gabors) and asked participants to report the orientation of the stimulus with the highest contrast. The results revealed a significant effect of self-relevance on task performance. First, compared to cues associated with a friend or stranger, self-relevant cues enhanced the apparent contrast of a stimulus. Second, the benefits of self-relevance were most pronounced when cues pertained to identities that were significant (vs. trivial or irrelevant) to observers. Together, these findings demonstrate that self-relevance potentiates the benefits of transient attention on stimulus processing.     

Influence of temporal overlap on time course of the Simon effect

Two experiments are reported in which we manipulated relevant and irrelevant stimulus dimensions to assess whether an increase in temporal overlap would influence the time course of a "standard" Simon effect (obtained when visual stimuli are presented on the left/right of the screen and left/right responses are performed with uncrossed hands). This procedure is new in two ways: First, the manipulations were intended to reduce, instead of increase, the distance between conditional and unconditional response-activation processes. Second, we manipulated the relevant and irrelevant stimulus dimensions in a manner that did not vary stimulus onset asynchronies, precues, or go/no go trials, or alter the stimulus quality. Results were consistent with the hypothesis that when the two response processes are shifted closer to each other, the Simon effect would be sustained across time, instead of decreasing as typically found. These findings are discussed in line with the temporal overlap hypothesis and with an automatic activation account.     

Second-order reversed phi

In a first-order reversed-phi motion stimulus (Anstis, 1970), the black-white contrast of successive frames is reversed, and the direction of apparent motion may, under some conditions, appear to be reversed. It is demonstrated here that, for many classes of stimuli, this reversal is a mathematical property of the stimuli themselves, and the real problem is in perceiving forward motion, which involves the second- or third-order motion systems or both. Three classes of novel second-order reversed-phi stimuli (contrast, spatial frequency, and flicker modulation) that are invisible to first-order motion analysis were constructed. In these stimuli, the salient stimulus features move in theforward (feature displacement) direction, but the second-order motion energy model predicts motion in thereversed direction. In peripheral vision, for all stimulus types and all temporal frequencies, all the observers saw only the reversed-phi direction of motion. In central vision, the observers also perceived reversed motion at temporal frequencies above about 4 Hz, but they perceived movement in the forward direction at lower temporal frequencies. Since all of these stimuli are invisible to first-order motion, these results indicate that the second-order reversed-phi stimuli activate two subsequent competing motion mech-anisms, both of which involve an initial stage of texture grabbing (spatiotemporal filtering, followed by fullwave rectification). The second-order motion system then applies a Reichardt detector (or equiva-lently, motion energy analysis) directly to this signal and arrives at the reversed-phi direction. The third-order system marks the location of features that differ from the background (the figure) in a salience map and computes motion in the forward direction from the changes in the spatiotemporal location of these marks. The second-order system’s report of reversed movement dominates in peripheral vision and in central vision at higher temporal frequencies, because it has better spatial and temporal resolu-tion than the third-order system, which has a cutoff frequency of 3–4 Hz (Lu & Sperling, 1995b). In cen-tral vision, below 3–4 Hz, the third-order system’s report of resolvable forward movement of something salient (the figure) dominates the second-order system’s report of texture contrast movement.     

Discrimination of spatial and temporal intervals defined by three light flashes: Effects of spacing on temporal judgments and of timing on spatial judgments

Observers were presented stimulus patterns consisting of a sequence of three laterally displaced light flashes, which defined two spatial intervals and two temporal intervals. The position and time of the second flash were varied factorially, and observers were asked to make relative judgments of either the two spatial intervals or the two temporal intervals. “Induction” effects of stimulus timing on spatial judgments and of stimulus spacing on temporal judgments were both found; however, the directionality of these effects differed between subjects. The results are inconsistent with the hypothesis, derived from previous findings, that such effects are determined primarily by a tendency toward perceiving constant velocity of apparent motion; it is proposed that the directionality of the induction effects is determined largely by the strategy adopted by the observer for combining spatial and temporal stimulus information.     

Flexible and abstract resolutions to crossmodal conflicts

Participants judged whether two sequential visual events were presented for the same length of time or for different lengths of time, while ignoring two irrelevant sequential sounds. Sounds could be either the same or different in terms of their duration or their pitch. When the visual stimuli were in conflict with the sound stimuli (e.g., visual events were the same, but the sounds were different) performance declined. This was true whether sounds varied in duration or in pitch. The influence of sounds was eliminated when visual duration discriminations were made easier. Together these results demonstrate that resolutions to crossmodal conflicts are flexible across the neural and cognitive architectures. More importantly, they suggest that interactions between modalities can span to abstract levels of same/different representations.     

0 + 1 > 1: How adding noninformative sound improves performance on a visual task

It is well known that the nervous system combines information from different cues within and across sensory modalities to improve performance on perceptual tasks. In this article, we present results showing that in a visual motion-detection task, concurrent auditory motion stimuli improve accuracy even when they do not provide any useful information for the task. When participants judged which of two stimulus intervals contained visual coherent motion, the addition of identical moving sounds to both intervals improved accuracy. However, this enhancement occurred only with sounds that moved in the same direction as the visual motion. Therefore, it appears that the observed benefit of auditory stimulation is due to auditory-visual interactions at a sensory level. Thus, auditory and visual motion-processing pathways interact at a sensory-representation level in addition to the level at which perceptual estimates are combined.     

Illusory temporal order for stimuli at different depth positions

We used four experiments to examine how the perceived temporal order of two visual stimuli depends on the depth position of the stimuli specified by a binocular disparity cue. When two stimuli were presented simultaneously at different depth positions in front of or around a fixation point, the observer perceived the more distant stimulus before the nearer stimulus (Experiments 1 and 2). This illusory temporal order was found only for sudden stimulus presentation (Experiment 3). These results suggest that a common processing, which is triggered by sudden luminance change, underlies this illusion. The strength of the illusion increased with the disparity gradient and the disparity size (Experiment 4). We propose that this illusion has a basis in the processing of motion in depth, which would alert the observer to a potential collision with an object that suddenly emerges in front of the observer.     

The phenomenology of spatial integration: data and models

A briefly presented visual stimulus followed by darkness seems to persist beyond its physical offset. We are concerned here with the relation between two characteristics of this visible persistence: first, its phenomenological resemblance to the stimulus that spawned it and second, its usefulness as a basis for integrating visual stimuli that are separated in time. We describe two experiments using a task in which two halves of a visual stimulus were presented successively and observers reported how complete the stimulus appeared to be. Stimuli appeared less complete with increases in both the duration of the interval intervening between presentation of the two halves and the duration of the initially presented stimulus half. This data pattern is similar to that obtained in tasks in which spatial integration of two temporally disparate stimuli is necessary for correct responding. On the basis of this similarity, we argue that phenomenological appearance and ability to integrate stimuli over time are two facets of the same perceptual events. We describe a formal model to account for these and other data.     

Stimulus–response correspondence effect as a function of temporal overlap between relevant and irrelevant information processing

The stimulus–response correspondence (SRC) effect refers to advantages in performance when stimulus and response correspond in dimensions or features, even if the common features are irrelevant to the task. Previous research indicated that the SRC effect depends on the temporal course of stimulus information processing. The current study investigated how the temporal overlap between relevant and irrelevant stimulus processing influences the SRC effect. In this experiment, the irrelevant stimulus (a previously associated tone) preceded the relevant stimulus (a coloured rectangle). The irrelevant and relevant stimuli onset asynchrony was varied to manipulate the temporal overlap between the irrelevant and relevant stimuli processing. Results indicated that the SRC effect size varied as a quadratic function of the temporal overlap between the relevant stimulus and irrelevant stimulus. This finding extends previous experimental observations that the SRC effect size varies in an increasing or decreasing function with reaction time. The current study demonstrated a quadratic function between effect size and the temporal overlap.     

The effect of motion on tactile and visual temporal order judgments

Four experiments were conducted, three with tactile stimuli and one with visual stimuli, in which subjects made temporal order judgments (TOJs). The tactile stimuli were patterns that moved laterally across the fingerpads. The subject's task was to judge which finger received the pattern first. Even though the movement was irrelevant to the task, the subjects' TOJs were greatly affected by the direction of movement of the patterns. Accuracy in judging temporal order was enhanced when the patterns moved in a direction that was consistent with the temporal order of presentation--for example, when the movement on each fingerpad was from right to left and the temporally leading site of stimulation was to the right of the temporally trailing site of stimulation. When movement was inconsistent with the temporal order of presentation, accuracy was considerably reduced, often well below chance.The bias in TOJs was unaffected by training or by presenting the stimuli to fingers on opposite hands. In a fourth experiment, subjects judged the temporal order of visual stimuli that, like the tactile stimuli, moved in a direction that was either consistent or inconsistent with the TOJ. The results were similar to those obtained with tactile stimuli. It is suggested that the bias may be affected by attentional mechanisms and by apparent motion generated between the two sites on the skin.     

Position uncertainty and the perception of apparent movement

Two experiments compared the perception of apparent movement when the second of two successive stimuli always appeared in the same position and when it varied randomly between two spatial positions. The results of both experiments showed that foreknowledge of the position of the second stimulus does not facilitate the perception of apparent movement. Experiment 2 also clearly showed that the space-time relationships of Korte’s third law of apparent movement does not depend on foreknowledge of the position of the second stimulus. These findings imply that apparent movement in real time occurs after the second stimulus has been registered by the visual system. It suggests that apparent movement involves a delayed decision mechanism that stores the first stimulus, the interstimulus temporal interval, and the second stimulus, and then impletes a motion compatible with the stimulus information.     

Perception of sequentially presented tactile point stimuli

Either two or three brief (10 msec) airjet stimuli were sequentially presented to any of the 24 interjoint regions of the fingers (thumbs excluded). The stimulus onset interval (SOI) ranged from zero (simultaneous presentation) through 200 msec. The S’s task in one part of the experiment was to report the positions stimulated in the order that the stimuli were presented; in a second part it was to rate the apparent motion produced by the stimulus sequence. While the ability of Ss to spatially localize the stimuli was a constant independent of SOI, their ability to temporally order the stimuli depended strongly on SOI. With two stimuli, these sequential errors decayed exponentially with SOI with a time constant of 26 msec. With three stimuli, however, both the sequential errors and equivalent temporal Urnen were more than twice as large as with two stimuli, indicating that the three-stimulus task is considerably more difficult than the two, and that the same simple temporal resolution model does not explain both cases. A model with a constant rate of information uptake, however, can explain both of these cases.