Assessing automaticity in the audiovisual integration of motion

This study examined the multisensory integration of visual and auditory motion information using a methodology designed to single out perceptual integration processes from post-perceptual influences. We assessed the threshold stimulus onset asynchrony (SOA) at which the relative directions (same vs. different) of simultaneously presented visual and auditory apparent motion streams could no longer be discriminated (Experiment 1). This threshold was higher than the upper threshold for direction discrimination (left vs. right) of each individual modality when presented in isolation (Experiment 2). The poorer performance observed in bimodal displays was interpreted as a consequence of automatic multisensory integration of motion information. Experiment 3 supported this interpretation by ruling out task differences as the explanation for the higher threshold in Experiment 1. Together these data provide empirical support for the view that multisensory integration of motion signals can occur at a perceptual level.     

Twice upon a time: multiple concurrent temporal recalibrations of audiovisual speech

Audiovisual timing perception can recalibrate following prolonged exposure to asynchronous auditory and visual inputs. It has been suggested that this might contribute to achieving perceptual synchrony for auditory and visual signals despite differences in physical and neural signal times for sight and sound. However, given that people can be concurrently exposed to multiple audiovisual stimuli with variable neural signal times, a mechanism that recalibrates all audiovisual timing percepts to a single timing relationship could be dysfunctional. In the experiments reported here, we showed that audiovisual temporal recalibration can be specific for particular audiovisual pairings. Participants were shown alternating movies of male and female actors containing positive and negative temporal asynchronies between the auditory and visual streams. We found that audiovisual synchrony estimates for each actor were shifted toward the preceding audiovisual timing relationship for that actor and that such temporal recalibrations occurred in positive and negative directions concurrently. Our results show that humans can form multiple concurrent estimates of appropriate timing for audiovisual synchrony.     

Serial dependence in the estimation of a car's arrival time

Serial dependence of a car's arrival time. We have studied the sequential structure of data in the arrival- time estimations. Forty participants estimated the arrival-time of a vehicle under two experimental conditions: real car and video-image. Various time series regression models were fit to our data, and residuals autocorrelations were computed. For each serial-dependence model, data were fit to three functions, namely, power, logarithmic, and linear. In both experimental conditions, the response magnitude (R) on a given trial in t was a function of the stimulus intensity (S) in such a trial (t) and of the S and R on t-1 and t-2. Assimilation effect to the previous responses and contrast effect to the previous stimuli has been found.     

Properties of the recombination of one-dimensional motion signals into a pattern motion signal

We have examined the human ability to determine the direction of movement of a variety of plaid patterns. The plaids were composed of two orthogonal sine-wave gratings. When the plaid components are of unequal spatial frequency or sometimes of unequal contrast, observers judge the direction of movement incorrectly. In terms of the two-stage model of Adelson and Movshon (1982), these errors may result from either a misjudgment in the perceived speeds of each of the components or a failure in the combination of one-dimensional-component movements into a coherent direction of motion of the two-dimensional plaid pattern, or both. A comparison of the perceived direction of motion of plaids with the relative perceived-speeds of the plaid component gratings suggests that both failures occur, but in different circumstances The relative perceived speed of the plaid components was measured with a spatial and a temporal forced-choice technique, the former leading to larger differences. Our results support the notion that the visual system decomposes a moving plaid into oriented components and subsequently recombines the component motions.     

Perceptual and decisional contributions to audiovisual interactions in the perception of apparent motion: a signal detection study

Motion information available to different sensory modalities can interact at both perceptual and post-perceptual (i.e., decisional) stages of processing. However, to date, researchers have only been able to demonstrate the influence of one of these components at any given time, hence the relationship between them remains uncertain. We addressed the interplay between the perceptual and post-perceptual components of information processing by assessing their influence on performance within the same experimental paradigm. We used signal detection theory to discriminate changes in perceptual sensitivity (d') from shifts in response criterion (c) in performance on a detection (Experiment 1) and a classification (Experiment 2) task regarding the direction of auditory apparent motion streams presented in noise. In the critical conditions, a visual motion distractor moving either leftward or rightward was presented together with the auditory motion. The results demonstrated a significant decrease in sensitivity to the direction of the auditory targets in the crossmodal conditions as compared to the unimodal baseline conditions that was independent of the relative direction of the visual distractor. In addition, we also observed significant shifts in response criterion, which were dependent on the relative direction of the distractor apparent motion. These results support the view that the perceptual and decisional components involved in audiovisual interactions in motion processing can coexist but are largely independent of one another.     

Integration of speed signals in the direction of motion

Speed discrimination tasks were used to examine the spatial and temporal characteristics of the integration mechanism involved when signals are extended in the direction of motion. We varied the aspect ratio of a signal patch whose speed differed from the background, while holding the area of the signal patch constant, so that the signal patch could be either extended in the direction of motion or extended orthogonal to the direction of motion. Speed discrimination thresholds decreased dramatically as the signal patch was extended in the direction of motion. The spatial and temporal integration regions were larger than would be expected if the integration mechanism were a low-level motion detector. The mechanism was tuned for direction of motion. The data are discussed with reference to two alternative integration mechanisms: a low-level detector that is elongated in the direction of motion and a higher level integration mechanism characterized by cooperative or facilitatory interactions between low-level detectors tuned to the same direction of motion. Our data are consistent with a second-level, direction-specific process that integrates the responses of low-level motion detectors.     

Spectral information for detection of acoustic time to arrival

The exponential increase of intensity for an approaching sound source provides salient information for a listener to make judgments of time to arrival (TTA). Specifically, a listener will experience a greater rate of increasing intensity for higher than for lower frequencies during a sound source’s approach. To examine the relative importance of this spectral information, listeners were asked to make judgments about the arrival times of nine 1-octave-band sound sources (the bands were consecutive, nonoverlapping single octaves, ranging from 40–80 Hz to ~10–20 kHz). As is typical in TTA tasks, listeners tended to underestimate the arrival time of the approaching sound source. In naturally occurring and independently manipulated amplification curves, bands with center frequencies between 120 and 250 Hz caused the least underestimation, and bands with center frequencies between 2000 and 7500 Hz caused the most underestimation. This spectral influence appears to be related to the greater perceived urgency of higher-frequency sounds.     

Properties of the recombination of one-dimensional motion signals into a pattern motion signal.

We have examined the human ability to determine the direction of movement of a variety of plaid patterns. The plaids were composed of two orthogonal sine-wave gratings. When the plaid components are of unequal spatial frequency or sometimes of unequal contrast, observers judge the direction of movement incorrectly. In terms of the two-stage model of Adelson and Movshon (1982), these errors may result from either a misjudgment in the perceived speeds of each of the components or a failure in the combination of one-dimensional component movements into a coherent direction of motion of the two-dimensional plaid pattern, or both. A comparison of the perceived direction of motion of plaids with the relative perceived speeds of the plaid component gratings suggest that both failures occur, but in different circumstances. The relative perceived speed of the plaid components was measured with a spatial and a temporal forced-choice technique, the former leading to larger differences. Our results support the notion that the visual system decomposes a moving plaid into oriented components and subsequently recombines the component motions.     

Catching the mind in flight: using behavioral indices to detect mindless reading in real time

Although mind wandering during reading is extremely common, researchers have only recently begun to study how it relates to reading behavior. In the present study, we used a word-by-word reading paradigm to investigate whether it could be possible to predict in real time whether a participant would report mind wandering when probed. By taking advantage of the finding that reaction times to individual words vary based on reports of mind wandering (with participants being less affected by length, number of syllables, and familiarity, and also showing an overall speed-up, during mindless reading), we were able to develop an algorithm that could successfully predict in real time whether a participant would report being on versus off task. In addition, for participants run without thought probes, there was a significant negative correlation between the number of predicted mind-wandering episodes and reading comprehension. Together, these findings offer a key advance toward the development of pedagogical tools for minimizing the negative impact of mindless reading, and they provide a new covert measure that could be used to study mind wandering without requiring participants to report on their mental states.     

In the footsteps of biological motion and multisensory perception: judgments of audiovisual temporal relations are enhanced for upright walkers

Observers judged whether a periodically moving visual display (point-light walker) had the same temporal frequency as a series of auditory beeps that in some cases coincided with the apparent footsteps of the walker. Performance in this multisensory judgment was consistently better for upright point-light walkers than for inverted point-light walkers or scrambled control stimuli, even though the temporal information was the same in the three types of stimuli. The advantage with upright walkers disappeared when the visual "footsteps" were not phase-locked with the auditory events (and instead offset by 50% of the gait cycle). This finding indicates there was some specificity to the naturally experienced multisensory relation, and that temporal perception was not simply better for upright walkers per se. These experiments indicate that the gestalt of visual stimuli can substantially affect multisensory judgments, even in the context of a temporal task (for which audition is often considered dominant). This effect appears to be constrained by the ecological validity of the particular pairings.     

Nondirected light signals and the structure of time

Temporal betweenness in space-time is defined solely in terms of light signals, using a signalling relation that does not distinguish between the sender and the receiver of a light signal. Special relativity and general relativity are considered separately, because the latter can be treated only locally. We conclude that the (local) coherence of time can be described if we know only which pairs of space-time points are light-connected. Other consequences in the case of special relativity: (1) a categorical axiom system exists in terms of nondirected light connection alone, with neither particle nor time order as a primitive concept, though we do not actually present the axioms; (2) any concept definable by coordinates is also definable in terms of nondirected light signals if and only if it is invariant under Lorentz transformations, translations, dilations, space reflections, and time reflections; and (3) any transformation of space-time (not necessarily continuous) which preserves nondirected light connection is a product of transformations just listed above. The bulk of the paper is devoted to proving that the definitions we give correspond to their intended interpretations in the usual space-time continua.     

Accuracy of judging time to arrival: effects of modality, trajectory, and gender

Observers' accuracy in using time-to-arrival (Ta) information was examined in 4 experiments. The issues included use of visual vs. acoustic Ta information, use of acoustic Ta information by blind Ss, use of Ta information controlling for velocity, and effects of angle of approach and arrival time on judgment accuracy. Visual information was used more efficiently than audiovisual and auditory information. Blind Ss used acoustical approach information as accurately as sighted Ss used visual information. Radial, oblique, and transverse orientations were used to examine effects of approach trajectory. Radial events were underestimated, whereas the more accurate transverse approach was likely to be overestimated. Oblique angle events yielded intermediate accuracies implying a spatiotemporal anisotropy. Women underestimated Ta more than did men. Possible reasons for Ss' judgment accuracy, including linear vs. nonlinear optical changes and relation to spatial skills and experience, were discussed.     

On audiovisual spatial synergy: the fragility of the phenomenon

Recent literature has highlighted the importance and ubiquity of cross-modal links in spatial attention, whereby shifts in attention in one modality often induce corresponding shifts in other modalities. We attempted to provide further evidence for the case of audiovisual links during sustained endogenous attention by addressing several potential methodological confounds in previous demonstrations. However, we failed repeatedly to reproduce the phenomenon of spatial synergies between auditory and visual attention, found by Driver and Spence (1994) and frequently cited to support the automatic nature of cross-modal attention links. We discuss the results in light of recent evidence about cross-modal spatial links during sustained attention and support the idea that such links can weaken or even disappear under certain circumstances, such as during periods of sustained attention. The implication is that individuals can select inputs from different modalities from different locations more easily than previously had been thought.     

Spatial factors in visual attention: some compensatory effects of location and time of arrival of nontargets

It is well established that the identity of nontarget events may affect reaction to a target event, but that spatial separation between the two will reduce such an influence. Two experiments are reported in which an attempt was made to distinguish between two accounts of this effect. On one, some of the information about events spatially distant from the target is shut out from analysis altogether. On the other, such events are fully analysed, but either the analysis proceeds more slowly or else it starts only after a delay. In the experiments the time of arrival of, and the distance between, the target and nontarget events were systematically varied. The conventional effects of the distance of nontargets from target were greatly reduced when the target and nontarget events were asynchronous. If the nontargets arrived first, they had an effect on reaction to the target whether they were near to or far from it. If they arrived second, their identity had no effect at either separation. These results appear to rule out any simple view of attention according to which information outside the target region is denied analysis. Rather, distant nontarget events are analysed, but produce their effects at a later time than less peripheral events.     

The integration of auditory and visual motion signals at threshold

To interpret our environment, we integrate information from all our senses. For moving objects, auditory and visual motion signals are correlated and provide information about the speed and the direction of the moving object. We investigated at what level the auditory and the visual modalities interact and whether the human brain integrates only motion signals that are ecologically valid. We found that the sensitivity for identifying motion was improved when motion signals were provided in both modalities. This improvement in sensitivity can be explained by probability summation. That is, auditory and visual stimuli are combined at a decision level, after the stimuli have been processed independently in the auditory and the visual pathways. Furthermore, this integration is direction blind and is not restricted to ecologically valid motion signals.