Learning and discrimination of audiovisual events in human infants: the hierarchical relation between intersensory temporal synchrony and rhythmic pattern cues

This study examined 4- to 10-month-old infants' perception of audio-visual (A-V) temporal synchrony cues in the presence or absence of rhythmic pattern cues. Experiment 1 established that infants of all ages could successfully discriminate between two different audiovisual rhythmic events. Experiment 2 showed that only 10-month-old infants detected a desynchronization of the auditory and visual components of a rhythmical event. Experiment 3 showed that 4- to 8-month-old infants could detect A-V desynchronization but only when the audiovisual event was nonrhythmic. These results show that initially in development infants attend to the overall temporal structure of rhythmic audiovisual events but that later in development they become capable of perceiving the embedded intersensory temporal synchrony relations as well.     

Intersensory redundancy guides attentional selectivity and perceptual learning in infancy

This study assessed an intersensory redundancy hypothesis, which holds that in early infancy information presented redundantly and in temporal synchrony across two sense modalities selectively recruits attention and facilitates perceptual differentiation more effectively than does the same information presented unimodally. Five-month-old infants' sensitivity to the amodal property of rhythm was examined in 3 experiments. Results revealed that habituation to a bimodal (auditory and visual) rhythm resulted in discrimination of a novel rhythm, whereas habituation to the same rhythm presented unimodally (auditory or visual) resulted in no evidence of discrimination. Also, temporal synchrony between the bimodal auditory and visual information was necessary for rhythm discrimination. These findings support an intersensory redundancy hypothesis and provide further evidence for the importance of redundancy for guiding and constraining early perceptual learning.     

Perception of intersensory synchrony: A tutorial review

For most multisensory events, observers perceive synchrony among the various senses (vision, audition, touch), despite the naturally occurring lags in arrival and processing times of the different information streams. A substantial amount of research has examined how the brain accomplishes this. In the present article, we review several key issues about intersensory timing, and we identify four mechanisms of how intersensory lags might be dealt with: by ignoring lags up to some point (a wide window of temporal integration), by compensating for predictable variability, by adjusting the point of perceived synchrony on the longer term, and by shifting one stream directly toward the other.     

A test battery to assess intrasensory and intersensory development of young children.

The purpose of the study was to develop a battery of tests for use in evaluation of intra- and intersensory development of young children. A battery of 15 tests (4 visual, 4 auditory, 4 tactile-kinesthetic, and 3 intersensory) was administered to 109 normally developing 6- and 8-year-old and 32 slowly developing or learning disabled children. Interdependence of test items within each intrasensory and the intersensory category was determined; intercorrelations ranged from .00 to .78. Reliability estimates were also determined. Face validity was claimed for each item. The effects of age or developmental level on test performance were established. Based upon the interdependence of the tests, reliability estimates, and the capacity of the tests to discriminate among groups classified according to age or developmental level, a battery of 10 intra- and intersensory tests was proposed. The battery has 3 tests of visual perception-visual memory, dynamic depth perception, and size discrimination; 3 tests of auditory perception-auditory discrimination, auditory memory of related syllables, and auditory sequential memory of numbers; 2 tests of tactile-kinesthetic perception-tactile integration and movement awareness; and 2 tests of intersensory integration-auditory-tactile intergration and auditory-visual integration.     

Infants' learning,memory, and generalization of learning for bimodal events

Study 1 investigated whether infants 3 and 7 months of age show differential learning of and memory for sight-sound pairs depending on whether or not temporal synchrony was present; memory was assessed after a 10-min and 1-week interval. Study 2 examined whether 7-month-olds show generalization of learning when they encounter novel bimodal events that are similar (changes in size, orientation, or color, and spectral sound properties) to the sight-sound pairs learned 1 week earlier based on temporal synchrony. For Study 1, infants received a familiarization phase followed by a paired-comparison preference procedure to assess for learning of the sight-sound pairs. One week later a memory test was given. Results confirmed that 7-month-olds had no difficulty learning auditory-visual pairings regardless of whether or not events were temporally synchronous, and they remembered these 10 min and 1 week later. In contrast, 3-month-olds showed poorer learning of sight-sound associations in the no-synchrony than synchrony conditions, and memory for sight-sound pairs 1 week later was shown only for the synchrony conditions. Results for Study 2 revealed generalization of learning of bimodal pairings under all stimulus conditions after a 1-week interval at 7 months of age. Implications of these findings for development of intersensory knowledge are discussed.     

The development of infant intersensory perception: advantages of a comparative convergent-operations approach

Despite impressive demonstrations of human infants' intersensory capabilities over the past several decades, there has been little focus on the contributions of prenatal and postnatal experience or the specific developmental processes underlying the emergence of intersensory functioning. Research with nonhuman animals has, however, provided a number of advances in understanding early intersensory perception. The authors explore the value of a comparative, convergent-operations approach to the study of early intersensory perception and examine how this approach has highlighted the study of (a) prenatal factors, (b) brain-behavior relations, and (c) context and experience variables contributing to infants' intersensory responsiveness. Examples of how human and animal research programs can cross-fertilize one another in their attempts to understand developmental processes underlying intersensory perception are considered.     

The effects of socioeconomic status on working memory in childhood are partially mediated by intersensory processing of audiovisual events in infancy

Socioeconomic status (SES) is a well-established predictor of individual differences in childhood language and cognitive functioning, including executive functions such as working memory. In infancy, intersensory processing—selectively attending to properties of events that are redundantly specified across the senses at the expense of non-redundant, irrelevant properties—also predicts language development. Our recent research demonstrates that individual differences in intersensory processing in infancy predict a variety of language outcomes in childhood, even after controlling for SES. However, relations among intersensory processing and cognitive outcomes such as working memory have not yet been investigated. Thus, the present study examines relations between intersensory processing in infancy and working memory in early childhood, and the role of SES in this relation. Children (N = 101) received the Multisensory Attention Assessment Protocol at 12-months to assess intersensory processing (face-voice and object-sound matching) and received the WPPSI at 36-months to assess working memory. SES was indexed by maternal education, paternal education, and income. A variety of novel findings emerged. 1) Individual differences in intersensory processing at 12-months predicted working memory at 36-months of age even after controlling for SES. 2) Individual differences in SES predicted intersensory processing at 12-months of age. 3) The well-established relation between SES and working memory was partially mediated by intersensory processing. Children from families of higher-SES have better intersensory processing skills at 12-months and this combination of factors predicts greater working memory two years later at 36-months. Together these findings reveal the role of intersensory processing in cognitive functioning.     

Intersensory redundancy facilitates prenatal perceptual learning in bobwhite quail (Colinus virginianus) embryos.

Information presented redundantly and in temporal synchrony across sensory modalities (intersensory redundancy) selectively recruits attention and facilitates perceptual learning in human infants. This comparative study examined whether intersensory redundancy also facilitates perceptual learning prenatally. The authors assessed quail (Colinus virginianus) embryos' ability to learn a maternal call when it was (a) unimodal, (b) concurrent but asynchronous with patterned light, or (c) redundant and synchronous with patterned light. Chicks' preference for the familiar over a novel maternal call was assessed 24 hr following hatching. Chicks receiving redundant, synchronous stimulation as embryos learned the call 4 times faster than those who received unimodal exposure. Chicks who received asynchronous bimodal stimulation showed no evidence of learning. These results provide the first evidence that embryos are sensitive to redundant, bimodal information and that it can facilitate learning during the prenatal period.     

Increasing specificity in perceptual development: infants' detection of nested levels of multimodal stimulation

This research assessed the development of infants' sensitivity to two nested amodal temporal relations in audible and visible events. Their detection of global temporal synchrony between visible and audible impacts and internal temporal structure nested within each impact specifying object composition (single versus compound objects) was assessed. Infants of 4, 7, and 11 weeks of age were habituated to a single and a compound object striking a surface and then received test trials depicting a change in synchrony or object composition. Results indicated an interaction between age and condition where sensitivity to synchrony was present by 4 weeks and remained stable across age, whereas sensitivity to composition emerged later, by 7 weeks, and increased dramatically with age. These findings converge with other recent findings to illustrate a pattern of increasing specificity in the development of perception, where infants first detect global and later detect embedded relations. The early sensitivity to global relations may provide an organizational framework for development by focusing infant attention on unitary events, guiding and constraining further exploration, and buffering infants from learning incongruent relations.     

Infant distractibility from social events mediates the relation between maternal responsiveness and infant language outcomes

Research demonstrates that contingent and appropriate maternal responsiveness to infant requests and bids for attention leads to better language outcomes. Research also indicates that infants who are less distracted by irrelevant competing stimulation and attend efficiently to audiovisual social events (e.g., faces and voices) show better language outcomes. However, few studies have assessed relations between maternal responsiveness, infant attention to faces and voices, and distractibility, and how together these factors lead to early language outcomes. A newly developed audiovisual protocol, the Multisensory Attention Assessment Protocol (MAAP; Bahrick et al., 2018), allows researchers to examine individual differences in attention to faces and voices and distractibility, and to assess relations with other variables. At 12 months, infants (n = 79) in an ongoing longitudinal study participated in the MAAP to assess intersensory matching of synchronous faces and voices and attention to an irrelevant competing visual distractor event. They also were observed in a brief play interaction to assess infant bids for attention and maternal responsiveness (accept, redirect, or ignore). At 18 months, receptive and expressive language were assessed using the Mullen Scales of Early Learning. Several noteworthy findings emerged: 1) mothers were generally responsive, accepting 74% and redirecting 14% of infant bids, 2) infants who had a greater number of their bids redirected by mothers, and who had better intersensory matching of synchronous faces and voices, showed less attention to the distractor, and 3) infants who showed less attention to the distractor had better receptive language. Findings demonstrate that maternal redirecting of infant attention by mothers who are generally responsive may promote better infant attentional control (lower distractibility) which in turn predicts better receptive language in toddlers.     

Infants' intermodal perception of two levels of temporal structure in natural events

Infants' intermodal perception of two levels of temporal structure uniting the visual and acoustic stimulation from natural, complex events was investigated in four experiments. Films depicting a single object (single, large marble) and a compound object (group of smaller marbles) colliding against a surface in an erratic pattern were presented to infants between 3 and months of age using an intermodal preference and search method. These stimulus events portrayed two levels of invariant temporal structure: (a) temporal synchrony united the sights and sounds of object impact, and (b) temporal microstructure, the internal temporal structure of each impact sound and motion, specified the composition of the object (single vs. compound). Experiment 1 demonstrated that by 6 months infants detected a relation between the audible and visible stimulation from these events when both levels of invariant temporal structure guided their intermodal exploration. Experiment 2 revealed that by 6 months infants detected the bimodal temporal microstructure specifying object composition. They looked predominantly to the film whose natural soundtrack was played even though the motions of objects in both films were synchronized with the soundtrack. Experiment 3 assessed infants' sensitivity to temporal synchrony relations. Two films depicting objects of the same composition were presented while the motions of only one of them was synchronized with the appropriate soundtrack. Both 6-month-olds showed evidence of detecting temporal synchrony relations under some conditions. Experiment 4 examined how temporal synchrony and temporal microstructure interact in directing intermodal exploration. The natural soundtrack to one of the objects was played out-of-synchrony with the motions of both. In contrast with the results of Experiment 2, infants at 6 months showed no evidence of detecting a relationship between the film and its appropriate soundtrack. This suggests that the temporal asynchrony disrupted their detection of the temporal microstructure specifying object composition. Results of these studies support on invariant-detection view of the development of intermodal perception.     

The intersensory origins of word‐comprehension: an ecological–dynamic systems view

How do infants begin to understand spoken words? Recent research suggests that word comprehension develops from the early detection of intersensory relations between conventionally paired auditory speech patterns (words) and visible objects or actions. More importantly, in keeping with dynamic systems principles, the findings suggest that word comprehension develops from a dynamic and complementary relationship between the organism (the infant) and the environment (language addressed to the infant). In addition, parallel findings from speech and non‐speech studies of intersensory perception provide evidence for domain general processes in the development of word comprehension. These research findings contrast with the view that a lexical acquisition device with specific lexical principles and innate constraints is required for early word comprehension. Furthermore, they suggest that learning of word–object relations is not merely an associative process. The data support an alternative view of the developmental process that emphasizes the dynamic and reciprocal interactions between general intersensory perception, selective attention and learning in infants, and the specific characteristics of maternal communication.     

Developmental changes in associations between auditory-visual events

The goal of the present study was twofold: to examine the influence of two amodal properties, co-location and temporal synchrony, on infants' associating a sight with a sound, and to determine if the relative influence of these properties on crossmodal learning changes with age. During familiarization 2-, 4-, 6- and 8-month-olds were presented two toys and a sound, with sights and sounds varying with respect to co-location and temporal synchrony. Following each familiarization phase infants were given a paired preference test to assess their learning of sight-sound associations. Measures of preferential looking revealed age-related changes in the influence of co-location and temporal synchrony on infants' learning sight-sound associations. At all ages, infants could use temporal synchrony and co-location as a basis for associating an auditory with a visual event and, in the absence of temporal synchrony, co-location was sufficient to support crossmodal learning. However, when these cues conflicted there were developmental changes in the influence of these cues on infants' learning auditory-visual associations. At 2 and 4 months infants associated the sounds with the toy that moved in synchrony with the sound's rhythm despite extreme violation of co-location of this sight and sound. In contrast, 6- and 8-month-olds did not associate a specific toy with the sound when co-location and synchrony information conflicted. The findings highlight the unique and interactive effects of distinct amodal properties on infants' learning arbitrary crossmodal relations. Possible explanations for the age shift in performance are discussed.     

Infants' perceptual differentiation of amodal and modality-specific audio-visual relations.

Ninety-six infants of 3 1/2 months were tested in an infant-control habituation procedure to determine whether they could detect three types of audio-visual relations in the same events. The events portrayed two amodal invariant relations, temporal synchrony and temporal microstructure specifying the composition of the objects, and one modality-specific relation, that between the pitch of the sound and the color/shape of the objects. Subjects were habituated to two events accompanied by their natural, synchronous, and appropriate sounds and then received test trials in which the relation between the visual and the acoustic information was changed. Consistent with Gibson's increasing specificity hypothesis, it was expected that infants would differentiate amodal invariant relations prior to detecting arbitrary, modality-specific relations. Results were consistent with this prediction, demonstrating significant visual recovery to a change in temporal synchrony and temporal microstructure, but not to a change in the pitch-color/shape relations. Two subsequent discrimination studies demonstrated that infants' failure to detect the changes in pitch-color/shape relations could not be attributed to an inability to discriminate the pitch or the color/shape changes used in Experiment 1. Infants showed robust discrimination of the contrasts used.     

The role of audiovisual asynchrony in person recognition

Audiovisual integration (AVI) has been demonstrated to play a major role in speech comprehension. Previous research suggests that AVI in speech comprehension tolerates a temporal window of audiovisual asynchrony. However, few studies have employed audiovisual presentation to investigate AVI in person recognition. Here, participants completed an audiovisual voice familiarity task in which the synchrony of the auditory and visual stimuli was manipulated, and in which visual speaker identity could be corresponding or noncorresponding to the voice. Recognition of personally familiar voices systematically improved when corresponding visual speakers were presented near synchrony or with slight auditory lag. Moreover, when faces of different familiarity were presented with a voice, recognition accuracy suffered at near synchrony to slight auditory lag only. These results provide the first evidence for a temporal window for AVI in person recognition between approximately 100 ms auditory lead and 300 ms auditory lag.     

The development of infant discrimination of affect in multimodal and unimodal stimulation: The role of intersensory redundancy

This research examined the developmental course of infants' ability to perceive affect in bimodal (audiovisual) and unimodal (auditory and visual) displays of a woman speaking. According to the intersensory redundancy hypothesis (L. E. Bahrick, R. Lickliter, & R. Flom, 2004), detection of amodal properties is facilitated in multimodal stimulation and attenuated in unimodal stimulation. Later in development, however, attention becomes more flexible, and amodal properties can be perceived in both multimodal and unimodal stimulation. The authors tested these predictions by assessing 3-, 4-, 5-, and 7-month-olds' discrimination of affect. Results demonstrated that in bimodal stimulation, discrimination of affect emerged by 4 months and remained stable across age. However, in unimodal stimulation, detection of affect emerged gradually, with sensitivity to auditory stimulation emerging at 5 months and visual stimulation at 7 months. Further temporal synchrony between faces and voices was necessary for younger infants' discrimination of affect. Across development, infants first perceive affect in multimodal stimulation through detecting amodal properties, and later their perception of affect is extended to unimodal auditory and visual stimulation. Implications for social development, including joint attention and social referencing, are considered.     

Synchrony in mother-infant vocal interactions revealed through timed event sequences

This report extends a previous cross-cultural study of synchrony in mother-infant vocal interactions (Bornstein et al., 2015) to immigrant samples. Immigrant dyads from three cultures of origin (Japan, South Korea, South America) living in the same culture of destination (the United States) were compared to nonmigrant dyads in those same cultures of origin and to nonmigrant European American dyads living in the same culture of destination (the United States). This article highlights an underutilized analysis to assess synchrony in mother-infant interaction and extends cross-cultural research on mother-infant vocal interaction. Timing of onsets and offsets of maternal speech to infants and infant nondistress vocalizations were coded separately from 50-min recorded naturalistic observations of mothers and infants. Odds ratios were computed to analyze synchrony in mother-infant vocal interactions. Synchrony was analyzed in three ways -- contingency of timed event sequences, mean differences in contingency by acculturation level and within dyads, and coordination of responsiveness within dyads. Immigrant mothers were contingently responsive to their infants’ vocalizations, but only Korean immigrant infants were contingently responsive to their mothers’ vocalizations. For the Japanese and South American comparisons, immigrant mothers were more contingently responsive than their infants (but not robustly so for South American immigrants). For the Korean comparison, mean differences in contingent responsiveness were found among acculturation groups (culture of origin, immigrant, culture of destination), but not between mothers and infants. Immigrant dyads’ mean levels of responsiveness did not differ. Immigrant mothers’ and infants’ levels of responsiveness were coordinated. Strengths and flexibility of the timed event sequential analytic approach to assessing synchrony in mother-infant interactions are discussed, particularly for culturally diverse samples.     

多感觉整合的时间再校准

跨通道刺激的时间同步性是多感觉整合的必要条件, 但是由于刺激物理传导与神经传导的差异, 它们在时间上并非完全匹配。时间再校准指大脑能够适应跨通道刺激间很短的时间延迟的现象, 反映了多感觉整合在时间维度上的可塑性, 表现为适应相继呈现的跨通道刺激后, 主观同时点向时间延迟方向的偏移。本文主要介绍了时间再校准的通道效应与潜在机制, 其初始加工阶段, 它与刺激内容加工的关系及主要的影响因素。今后的研究应当进一步探索时间再校准能否发生于早期加工阶段, 检验其认知过程是否具有双向性, 探讨空间选择性注意的作用, 并结合神经机制的研究, 从综合的视角进行更完善的理论建构。     

Infant perception of audio-visual speech synchrony in familiar and unfamiliar fluent speech

We investigated the effects of linguistic experience and language familiarity on the perception of audio-visual (A-V) synchrony in fluent speech. In Experiment 1, we tested a group of monolingual Spanish- and Catalan-learning 8-month-old infants to a video clip of a person speaking Spanish. Following habituation to the audiovisually synchronous video, infants saw and heard desynchronized clips of the same video where the audio stream now preceded the video stream by 366, 500, or 666 ms. In Experiment 2, monolingual Catalan and Spanish infants were tested with a video clip of a person speaking English. Results indicated that in both experiments, infants detected a 666 and a 500 ms asynchrony. That is, their responsiveness to A-V synchrony was the same regardless of their specific linguistic experience or familiarity with the tested language. Compared to previous results from infant studies with isolated audiovisual syllables, these results show that infants are more sensitive to A-V temporal relations inherent in fluent speech. Furthermore, the absence of a language familiarity effect on the detection of A-V speech asynchrony at eight months of age is consistent with the broad perceptual tuning usually observed in infant response to linguistic input at this age.     

Temporal binding and the neural correlates of sensory awareness

Theories of binding have recently come into the focus of the consciousness debate. In this review, we discuss the potential relevance of temporal binding mechanisms for sensory awareness. Specifically, we suggest that neural synchrony with a precision in the millisecond range may be crucial for conscious processing, and may be involved in arousal, perceptual integration, attentional selection and working memory. Recent evidence from both animal and human studies demonstrates that specific changes in neuronal synchrony occur during all of these processes and that they are distinguished by the emergence of fast oscillations with frequencies in the gamma-range.