Temporal coding in rhythm tasks revealed by modality effects

Temporal coding has been studied by examining the perception and reproduction of rhythms and by examining memory for the order of events in a list. We attempt to link these research programs both empirically and theoretically. Glenberg and Swanson (1986) proposed that the superior recall of auditory material, compared with visual material, reflects more accurate temporal coding for the auditory material. In this paper, we demonstrate that a similar modality effect can be produced in a rhythm task. Auditory rhythms composed of stimuli of two durations are reproduced more accurately than are visual rhythms. Furthermore, it appears that the auditory superiority reflects enhanced chunking of the auditory material rather than better identification of durations.     

A unique visual rhythm does not pop out

We investigated attentional demands in visual rhythm perception of periodically moving stimuli using a visual search paradigm. A dynamic search display consisted of vertically “bouncing dots” with regular rhythms. The search target was defined by a unique visual rhythm (i.e., a shorter or longer period) among rhythmic distractors with identical periods. We found that search efficiency for a faster or a slower periodically moving target decreased as the number of distractors increased, although searching for a faster target was about one second faster than searching for a slower target. We conclude that perception of a visual rhythm defined by a unique period is not a “pop-out” process, but a serial one that demands considerable attention.     

The role of rhythm in perceiving speech in noise: a comparison of percussionists,vocalists and non-musicians

The natural rhythms of speech help a listener follow what is being said, especially in noisy conditions. There is increasing evidence for links between rhythm abilities and language skills; however, the role of rhythm-related expertise in perceiving speech in noise is unknown. The present study assesses musical competence (rhythmic and melodic discrimination), speech-in-noise perception and auditory working memory in young adult percussionists, vocalists and non-musicians. Outcomes reveal that better ability to discriminate rhythms is associated with better sentence-in-noise (but not words-in-noise) perception across all participants. These outcomes suggest that sensitivity to rhythm helps a listener understand unfolding speech patterns in degraded listening conditions, and that observations of a “musician advantage” for speech-in-noise perception may be mediated in part by superior rhythm skills.     

On the relation between time and space in the visual discrimination of velocity.

Is the perception of velocity determined by the prior discrimination of spatial and temporal distances? Two experiments sought to answer this question by comparing the discriminabilities of moving stimuli varied in spatial extent, temporal duration, or in redundant combinations of both variables. The subject's task was to identify which of two alternative stimuli was presented on each trial. A set of four stimuli was constructed from two values of spatial extent and two values of temporal duration. Separate conditions required discrimination of each of the six possible pairs of these stimuli. Experiment 1 examined continuous motion and Experiment 2 examined apparent motion for stimuli with short (50 versus 65 msec) and with long (500 versus 650 msec) interstimulus intervals. With continuous motion and with good apparent motion (short intervals), the discrimination between the different-velocity bivariate pairs was too accurate to be attributed only to discriminations of the spatial and temporal extents of the motion. This did not occur with poor apparent motion. Evidently, time and space are perceptually related.     

A comparison of auditory and visual apparent motion presented individually and with crossmodal moving distractors

Unimodal auditory and visual apparent motion (AM) and bimodal audiovisual AM were investigated to determine the effects of crossmodal integration on motion perception and direction-of-motion discrimination in each modality. To determine the optimal stimulus onset asynchrony (SOA) ranges for motion perception and direction discrimination, we initially measured unimodal visual and auditory AMs using one of four durations (50, 100, 200, or 400 ms) and ten SOAs (40-450 ms). In the bimodal conditions, auditory and visual AM were measured in the presence of temporally synchronous, spatially displaced distractors that were either congruent (moving in the same direction) or conflicting (moving in the opposite direction) with respect to target motion. Participants reported whether continuous motion was perceived and its direction. With unimodal auditory and visual AM, motion perception was affected differently by stimulus duration and SOA in the two modalities, while the opposite was observed for direction of motion. In the bimodal audiovisual AM condition, discriminating the direction of motion was affected only in the case of an auditory target. The perceived direction of auditory but not visual AM was reduced to chance levels when the crossmodal distractor direction was conflicting. Conversely, motion perception was unaffected by the distractor direction and, in some cases, the mere presence of a distractor facilitated movement perception.     

Intersensory redundancy guides attentional selectivity and perceptual learning in infancy

This study assessed an intersensory redundancy hypothesis, which holds that in early infancy information presented redundantly and in temporal synchrony across two sense modalities selectively recruits attention and facilitates perceptual differentiation more effectively than does the same information presented unimodally. Five-month-old infants' sensitivity to the amodal property of rhythm was examined in 3 experiments. Results revealed that habituation to a bimodal (auditory and visual) rhythm resulted in discrimination of a novel rhythm, whereas habituation to the same rhythm presented unimodally (auditory or visual) resulted in no evidence of discrimination. Also, temporal synchrony between the bimodal auditory and visual information was necessary for rhythm discrimination. These findings support an intersensory redundancy hypothesis and provide further evidence for the importance of redundancy for guiding and constraining early perceptual learning.     

Lateralization of auditory rhythm length in temporal lobe lesions

In the visual modality, short rhythmic stimuli have been proven to be better processed (sequentially) by the left hemisphere, while longer rhythms appear to be better (holistically) processed by the right hemisphere. This study was set up to see if the same holds in the auditory modality. The rhythm task as originally designed by Seashore was computerized and is part of the Fepsy Neuropsychological battery. This task was performed by 85 patients with intractable temporal lobe epilepsy (left TLE = 32; right TLE = 53) enrolled in the Dutch Collaborative Epilepsy Surgery Program. They performed the task before and 6 months after surgery. The task consists of 30 pairs of rhythmic patterns in 3 series of 10 items. The series contains patterns of 5, 6, or 7 notes. The purpose is to indicate whether the two patterns are the same or different. Reaction times are also measured. If the hypothesis is true, the short-item sequence will be better processed by patients with right temporal lobe epilepsy (nonimpaired left temporal lobe), the longer sequence will be better processed by the left temporal epilepsy group (nonimpaired right temporal lobe). No overall laterality effect on rhythm perception could be found and no difference was found between both test moments. IQ did not correlate with rhythm performance. However, there was an interaction effect of laterality and rhythm length on performance and reaction time. This effect can be explained by the increase after the operation of the score of the left focus group and a decrease in the right focus group on the longer rhythms. This effect was somewhat less strong in the reaction times: a clear tendency for faster reaction times after surgery in the left and longer reaction times in the right focus group. The effect could not be explained for by the difference in extent of resection in either temporal lobe. This study showed that memory for and discrimination of auditory rhythm is dependent on which hemisphere is used in processing. The effect could be demonstrated for the right hemisphere, which uses a holistic processing of stimuli, which outperforms the left in rhythms consisting of a long sequence. In left temporal resections an improvement occurs on the longer rhythms and in right temporal resections the performance on the longest rhythms decreases.     

Metrical perception of trisyllabic speech rhythms

The perception of duration-based syllabic rhythm was examined within a metrical framework. Participants assessed the duration patterns of four-syllable phrases set within the stress structure XxxX (an Abercrombian trisyllabic foot). Using on-screen sliders, participants created percussive sequences that imitated speech rhythms and analogous non-speech monotone rhythms. There was a tendency to equalize the interval durations for speech stimuli but not for non-speech. Despite the perceptual regularization of syllable durations, different speech phrases were conceived in various rhythmic configurations, pointing to a diversity of perceived meters in speech. In addition, imitations of speech stimuli showed more variability than those of non-speech. Rhythmically skilled listeners exhibited lower variability and were more consistent with vowel-centric estimates when assessing speech stimuli. These findings enable new connections between meter- and duration-based models of speech rhythm perception.     

Infants' perception of subjective contours from apparent motion

We examined infants' perception of subjective contours in Subjective-Contour-from-Apparent-Motion (SCAM) stimuli [e.g., Cicerone, C. M., Hoffman, D. D., Gowdy, P. D., & Kim, J. S. (1995). The perception of color from motion. Perception & Psychophysics, 57, 761-777] using the preferential looking technique. The SCAM stimulus is composed of random dots which are assigned two different colors. Circular region assigned one color moved apparently, keeping all dots' location unchanged. In the SCAM stimulus, adults can perceive subjective color spreading and subjective contours in apparent motion (http://c-faculty.chuo-u.ac.jp/ approximately ymasa/okamura/ibd_demo.html). In the present study, we conducted two experiments by using this type of SCAM stimulus. A total of thirty-six 3-8-month-olds participated. In experiment 1, we presented two stimuli to the infants side by side: a SCAM stimulus consisting of different luminance, and a non-SCAM stimulus consisting of isoluminance dots. The results indicated that the 5-8-month-olds showed preference for the SCAM stimuli. In experiments 2 and 3, we confirmed that the infants' preference for the SCAM stimulus was not generated by the local difference and local change made by luminance of dots but by the subjective contours. These results suggest that 5-8-month-olds were able to perceive subjective contours in the SCAM stimuli.     

Audiovisual beat induction in complex auditory rhythms: Point-light figure movement as an effective visual beat

This study investigated whether explicit beat induction in the auditory, visual, and audiovisual (bimodal) modalities aided the perception of weakly metrical auditory rhythms, and whether it reinforced attentional entrainment to the beat of these rhythms. The visual beat-inducer was a periodically bouncing point-light figure, which aimed to examine whether an observed rhythmic human movement could induce a beat that would influence auditory rhythm perception. In two tasks, participants listened to three repetitions of an auditory rhythm that were preceded and accompanied by (1) an auditory beat, (2) a bouncing point-light figure, (3) a combination of (1) and (2) synchronously, or (4) a combination of (1) and (2), with the figure moving in anti-phase to the auditory beat. Participants reproduced the auditory rhythm subsequently (Experiment 1), or detected a possible temporal change in the third repetition (Experiment 2). While an explicit beat did not improve rhythm reproduction, possibly due to the syncopated rhythms when a beat was imposed, bimodal beat induction yielded greater sensitivity to a temporal deviant in on-beat than in off-beat positions. Moreover, the beat phase of the figure movement determined where on-beat accents were perceived during bimodal induction. Results are discussed with regard to constrained beat induction in complex auditory rhythms, visual modulation of auditory beat perception, and possible mechanisms underlying the preferred visual beat consisting of rhythmic human motions.     

Motion influences gaze direction discrimination and disambiguates contradictory luminance cues

In two experiments, we investigated the role of apparent motion in discriminating left/right gaze deviation judgments. We demonstrated that discrimination accuracy and response confidence was significantly higher when the eyes were moved to the left or right, compared to when the eyes were presented in their final shifted position (static images). To dissociate the role of motion signals from luminance signals, gaze stimuli were also presented in reverse contrast. Replicating past studies polarity reversal had a profound and detrimental effect on gaze discrimination in static images, although, intriguingly, while response confidence remained low, participant performance improved as gaze angle increased. In striking contrast to these data, polarity reversal had no negative effect on performance when the eyes were moved. We discuss these findings in the context of a multiple-cue account of gaze perception.     

Anticipation across modalities in children and adults: Relating anticipatory alpha rhythm lateralization,reaction time,and executive function

The development of the ability to anticipate—as manifested by preparatory actions and neural activation related to the expectation of an upcoming stimulus—may play a key role in the ontogeny of cognitive skills more broadly. This preregistered study examined anticipatory brain potentials and behavioral responses (reaction time; RT) to anticipated target stimuli in relation to individual differences in the ability to use goals to direct action (as indexed by measures of executive function; EF). A cross-sectional investigation was conducted in 40 adults (aged 18–25 years) and 40 children (aged 6–8 years) to examine the association of changes in the amplitude of modality-specific alpha-range rhythms in the electroencephalogram (EEG) during anticipation of lateralized visual, tactile, or auditory stimuli with inter- and intraindividual variation in RT and EF. Children and adults exhibited contralateral anticipatory reductions in the mu rhythm and the visual alpha rhythm for tactile and visual anticipation, respectively, indicating modality and spatially specific attention allocation. Variability in within-subject anticipatory alpha lateralization (the difference between contralateral and ipsilateral alpha power) was related to single-trial RT. This relation was more prominent in adults than in children, and was not apparent for auditory stimuli. Multilevel models indicated that interindividual differences in anticipatory mu rhythm lateralization contributed to the significant association with variability in EF, but this was not the case for visual or auditory alpha rhythms. Exploratory microstate analyses were undertaken to cluster global field power (GFP) into a distribution-free temporal analysis examining developmental differences across samples and in relation to RT and EF. Anticipation is suggested as a developmental bridge construct connecting neuroscience, behavior, and cognition, with anticipatory EEG oscillations being discussed as quantifiable and potentially malleable indicators of stimulus prediction.     

Audiovisual influences on the perception of visual apparent motion: exploring the effect of a single sound

Previous research has shown that irrelevant sounds can facilitate the perception of visual apparent motion. Here the effectiveness of a single sound to facilitate motion perception was investigated in three experiments. Observers were presented with two discrete lights temporally separated by stimulus onset asynchronies from 0 to 350 ms. After each trial, observers classified their impression of the stimuli using a categorisation system. A short sound presented temporally (and spatially) midway between the lights facilitated the impression of motion relative to baseline (lights without sound), whereas a sound presented either before the first or after the second light or simultaneously with the lights did not affect motion impression. The facilitation effect also occurred with sound presented far from the visual display, as well as with continuous-sound that was started with the first light and terminated with the second light. No facilitation of visual motion perception occurred if the sound was part of a tone sequence that allowed for intramodal perceptual grouping of the auditory stimuli prior to the critical audiovisual stimuli. Taken together, the findings are consistent with a low-level audiovisual integration approach in which the perceptual system merges temporally proximate sound and light stimuli, thereby provoking the impression of a single multimodal moving object.     

Topological structure in the perception of apparent motion

Not only translations and rotations, but also intriguing 'plastic deformations' are observed in apparent motion. What kinds of invariants does the visual system depend on during these transformations to determine that two figures of different shapes nevertheless represent the same object? Experiments are reported in which seven pairs of stimuli with topological differences were used. The evidence suggests that topological invariants may be used in the perception of apparent motion. In spite of variations in other factors, such as brightness, spatial frequency, terminators, etc, subjects displayed a strong preference for motion from a central figure to a figure with the same topological invariants. The results emphasize the importance of topological structure in figure perception.     

Correlates of linguistic rhythm in the speech signal

Spoken languages have been classified by linguists according to their rhythmic properties, and psycholinguists have relied on this classification to account for infants' capacity to discriminate languages. Although researchers have measured many speech signal properties, they have failed to identify reliable acoustic characteristics for language classes. This paper presents instrumental measurements based on a consonant/vowel segmentation for eight languages. The measurements suggest that intuitive rhythm types reflect specific phonological properties, which in turn are signaled by the acoustic/phonetic properties of speech. The data support the notion of rhythm classes and also allow the simulation of infant language discrimination, consistent with the hypothesis that newborns rely on a coarse segmentation of speech. A hypothesis is proposed regarding the role of rhythm perception in language acquisition.     

Correlates of linguistic rhythm in the speech signal

Spoken languages have been classified by linguists according to their rhythmic properties, and psycholinguists have relied on this classification to account for infants' capacity to discriminate languages. Although researchers have measured many speech signal properties, they have failed to identify reliable acoustic characteristics for language classes. This paper presents instrumental measurements based on a consonant/vowel segmentation for eight languages. The measurements suggest that intuitive rhythm types reflect specific phonological properties, which in turn are signaled by the acoustic/phonetic properties of speech. The data support the notion of rhythm classes and also allow the simulation of infant language discrimination, consistent with the hypothesis that newborns rely on a coarse segmentation of speech. A hypothesis is proposed regarding the role of rhythm perception in language acquisition.     

Metrical congruency and kinematic familiarity facilitate temporal binding between musical and dance rhythms

Although music and dance are often experienced simultaneously, it is unclear what modulates their perceptual integration. This study investigated how two factors related to music–dance correspondences influenced audiovisual binding of their rhythms: the metrical match between the music and dance, and the kinematic familiarity of the dance movement. Participants watched a point-light figure dancing synchronously to a triple-meter rhythm that they heard in parallel, whereby the dance communicated a triple (congruent) or a duple (incongruent) visual meter. The movement was either the participant’s own or that of another participant. Participants attended to both streams while detecting a temporal perturbation in the auditory beat. The results showed lower sensitivity to the auditory deviant when the visual dance was metrically congruent to the auditory rhythm and when the movement was the participant’s own. This indicated stronger audiovisual binding and a more coherent bimodal rhythm in these conditions, thus making a slight auditory deviant less noticeable. Moreover, binding in the metrically incongruent condition involving self-generated visual stimuli was correlated with self-recognition of the movement, suggesting that action simulation mediates the perceived coherence between one’s own movement and a mismatching auditory rhythm. Overall, the mechanisms of rhythm perception and action simulation could inform the perceived compatibility between music and dance, thus modulating the temporal integration of these audiovisual stimuli.     

Modality effects in rhythm processing: Auditory encoding of visual rhythms is neither obligatory nor automatic

Modality effects in rhythm processing were examined using a tempo judgment paradigm, in which participants made speeding-up or slowing-down judgments for auditory and visual sequences. A key element of stimulus construction was that the expected pattern of tempo judgments for critical test stimuli depended on a beat-based encoding of the sequence. A model-based measure of degree of beat-based encoding computed from the pattern of tempo judgments revealed greater beat sensitivity for auditory rhythms than for visual rhythms. Visual rhythms with prior auditory exposure were more likely to show a pattern of tempo judgments similar to that for auditory rhythms than were visual rhythms without prior auditory exposure, but only for a beat period of 600 msec. Slowing down the rhythms eliminated the effect of prior auditory exposure on visual rhythm processing. Taken together, the findings in this study support the view that auditory rhythms demonstrate an advantage over visual rhythms in beat-based encoding and that the auditory encoding of visual rhythms can be facilitated with prior auditory exposure, but only within a limited temporal range. The broad conclusion from this research is that “hearing visual rhythms” is neither obligatory nor automatic, as was previously claimed by Guttman, Gilroy, and Blake (2005).     

Auditory stimulation affects apparent motion1

Abstract: In two experiments, we investigated how the number of auditory stimuli affected the apparent motion induced by visual stimuli. The multiple visual stimuli that induced the apparent motion on the front parallel plane, or in the depth dimension in terms of the binocular disparity cue, were accompanied by multiple auditory stimuli. Observers reported the number of visual stimuli (Experiments 1 and 2) and the displacement of the apparent motion that was defined by the distance between the first and last visual stimuli (Experiment 2). When the number of auditory stimuli was more/less than that of the visual stimuli, observers tended to perceive more/less visual stimuli and a larger/smaller displacement than when the numbers of the auditory and visual stimuli were the same, regardless of the dimension of motion. These results suggest that auditory stimulation may modify the visual processing of motion by modulating the spatiotemporal resolution and extent of the displacement.     

11-month-olds' knowledge of how familiar words sound

During the first year of life, infants' perception of speech becomes tuned to the phonology of the native language, as revealed in laboratory discrimination and categorization tasks using syllable stimuli. However, the implications of these results for the development of the early vocabulary remain controversial, with some results suggesting that infants retain only vague, sketchy phonological representations of words. Five experiments using a preferential listening procedure tested Dutch 11-month-olds' responses to word, nonword and mispronounced-word stimuli. Infants listened longer to words than nonwords, but did not exhibit this response when words were mispronounced at onset or at offset. In addition, infants preferred correct pronunciations to onset mispronunciations. The results suggest that infants' encoding of familiar words includes substantial phonological detail.