Infant perception of audio-visual speech synchrony in familiar and unfamiliar fluent speech

We investigated the effects of linguistic experience and language familiarity on the perception of audio-visual (A-V) synchrony in fluent speech. In Experiment 1, we tested a group of monolingual Spanish- and Catalan-learning 8-month-old infants to a video clip of a person speaking Spanish. Following habituation to the audiovisually synchronous video, infants saw and heard desynchronized clips of the same video where the audio stream now preceded the video stream by 366, 500, or 666 ms. In Experiment 2, monolingual Catalan and Spanish infants were tested with a video clip of a person speaking English. Results indicated that in both experiments, infants detected a 666 and a 500 ms asynchrony. That is, their responsiveness to A-V synchrony was the same regardless of their specific linguistic experience or familiarity with the tested language. Compared to previous results from infant studies with isolated audiovisual syllables, these results show that infants are more sensitive to A-V temporal relations inherent in fluent speech. Furthermore, the absence of a language familiarity effect on the detection of A-V speech asynchrony at eight months of age is consistent with the broad perceptual tuning usually observed in infant response to linguistic input at this age.     

No two cues are alike: Depth of learning during infancy is dependent on what orients attention

Human infants develop a variety of attentional mechanisms that allow them to extract relevant information from a cluttered multimodal world. We know that both social and nonsocial cues shift infants’ attention, but not how these cues differentially affect learning of multimodal events. Experiment 1 used social cues to direct 8- and 4-month-olds’ attention to two audiovisual events (i.e., animations of a cat or dog accompanied by particular sounds) while identical distractor events played in another location. Experiment 2 directed 8-month-olds’ attention with colorful flashes to the same events. Experiment 3 measured baseline learning without attention cues both with the familiarization and test trials (no cue condition) and with only the test trials (test control condition). The 8-month-olds exposed to social cues showed specific learning of audiovisual events. The 4-month-olds displayed only general spatial learning from social cues, suggesting that specific learning of audiovisual events from social cues may be a function of experience. Infants cued with the colorful flashes looked indiscriminately to both cued locations during test (similar to the 4-month-olds learning from social cues) despite attending for equal duration to the training trials as the 8-month-olds with the social cues. Results from Experiment 3 indicated that the learning effects in Experiments 1 and 2 resulted from exposure to the different cues and multimodal events. We discuss these findings in terms of the perceptual differences and relevance of the cues.     

Infants use meter to categorize rhythms and melodies: implications for musical structure learning

Little is known about whether infants perceive meter, the underlying temporal structure of music. We employed a habituation paradigm to examine whether 7-month-old infants could categorize rhythmic and melodic patterns on the basis of the underlying meter, which was implied from event and accent frequency of occurrence. In Experiment 1, infants discriminated duple and triple classes of rhythm on the basis of implied meter. Experiment 2 replicated this result while controlling for rhythmic grouping structure, confirming that infants perceived metrical structure despite occasional ambiguities and conflicting group structure. In Experiment 3, infants categorized melodies on the basis of contingencies between metrical position and pitch. Infants presented with metrical melodies detected reversals of pitch/meter contingencies, while infants presented with non-metrical melodies showed no preference. Results indicate that infants can infer meter from rhythmic patterns, and that they may use this metrical structure to bootstrap their knowledge acquisition in music learning.     

That’s the way the ball bounces: infants’ and adults’ perception of spatial and temporal contiguity in collisions involving bouncing balls

Three experiments investigated the perception of collisions involving bouncing balls by 7- and 10-month-old infants and adults. In previous research, 10-month-old infants perceived the causality of launching collisions (events in which one object moves along a smooth horizontal trajectory toward a second object, apparently launching it into motion) in relatively simple event contexts. In more complex event contexts, infants failed to discriminate among the events or respond to changes in individual features. Experiments 1 and 2 of the present investigation revealed that 7- and 10-month-old infants attended to spatial and temporal contiguity, but not causality, in collisions involving the movement of bouncing balls. In Experiment 3, both spatiotemporal contiguity and general knowledge about movement trajectories influenced adults’ judgments of causality for these collisions. The present results add to a growing understanding of infants’ event perception as constructive and a function, in part, of the complexity of the event context.     

Infants' Perception of Causal Chains

Three experiments examined infants' and adults' perception of causal sequences of events. In a causal-chain sequence, the first action causes a second action that then causes a final outcome; in a temporal-chain sequence, the first two actions are independent and the second action causes a final outcome. Infants and adults were shown the same event sequences; infants were tested using a visual habituation paradigm, whereas adults were given a questionnaire. Experiment 1 indicated that 15-month-old infants perceive the primary cause of the final outcome to be the first action in a causal chain but the second action in a temporal chain. Experiment 2 showed that adults interpret the causal sequences in a manner similar to that of 15-month-olds. Finally, Experiment 3 showed that 10-month-old infants do not yet perceive causal sequences in the same manner as 15-month-olds and adults. These results are interpreted in terms of both infants' developing knowledge of causal events and adults' attributions of causality in complex events.     

Infants' intermodal perception of two levels of temporal structure in natural events

Infants' intermodal perception of two levels of temporal structure uniting the visual and acoustic stimulation from natural, complex events was investigated in four experiments. Films depicting a single object (single, large marble) and a compound object (group of smaller marbles) colliding against a surface in an erratic pattern were presented to infants between 3 and months of age using an intermodal preference and search method. These stimulus events portrayed two levels of invariant temporal structure: (a) temporal synchrony united the sights and sounds of object impact, and (b) temporal microstructure, the internal temporal structure of each impact sound and motion, specified the composition of the object (single vs. compound). Experiment 1 demonstrated that by 6 months infants detected a relation between the audible and visible stimulation from these events when both levels of invariant temporal structure guided their intermodal exploration. Experiment 2 revealed that by 6 months infants detected the bimodal temporal microstructure specifying object composition. They looked predominantly to the film whose natural soundtrack was played even though the motions of objects in both films were synchronized with the soundtrack. Experiment 3 assessed infants' sensitivity to temporal synchrony relations. Two films depicting objects of the same composition were presented while the motions of only one of them was synchronized with the appropriate soundtrack. Both 6-month-olds showed evidence of detecting temporal synchrony relations under some conditions. Experiment 4 examined how temporal synchrony and temporal microstructure interact in directing intermodal exploration. The natural soundtrack to one of the objects was played out-of-synchrony with the motions of both. In contrast with the results of Experiment 2, infants at 6 months showed no evidence of detecting a relationship between the film and its appropriate soundtrack. This suggests that the temporal asynchrony disrupted their detection of the temporal microstructure specifying object composition. Results of these studies support on invariant-detection view of the development of intermodal perception.     

Developmental changes in associations between auditory-visual events

The goal of the present study was twofold: to examine the influence of two amodal properties, co-location and temporal synchrony, on infants' associating a sight with a sound, and to determine if the relative influence of these properties on crossmodal learning changes with age. During familiarization 2-, 4-, 6- and 8-month-olds were presented two toys and a sound, with sights and sounds varying with respect to co-location and temporal synchrony. Following each familiarization phase infants were given a paired preference test to assess their learning of sight-sound associations. Measures of preferential looking revealed age-related changes in the influence of co-location and temporal synchrony on infants' learning sight-sound associations. At all ages, infants could use temporal synchrony and co-location as a basis for associating an auditory with a visual event and, in the absence of temporal synchrony, co-location was sufficient to support crossmodal learning. However, when these cues conflicted there were developmental changes in the influence of these cues on infants' learning auditory-visual associations. At 2 and 4 months infants associated the sounds with the toy that moved in synchrony with the sound's rhythm despite extreme violation of co-location of this sight and sound. In contrast, 6- and 8-month-olds did not associate a specific toy with the sound when co-location and synchrony information conflicted. The findings highlight the unique and interactive effects of distinct amodal properties on infants' learning arbitrary crossmodal relations. Possible explanations for the age shift in performance are discussed.     

Reasoning about containment events in very young infants

The present research examined very young infants' expectations about containment events. In Experiment 1, 3.5-month-old infants saw a test event in which an object was lowered inside a container with either a wide opening (open-container condition) or no opening (closed-container condition) in its top surface. The infants looked reliably longer at the closed- than at the open-container test event. These and baseline data suggested that the infants recognized that the object could be lowered inside the container with the open but not the closed top. In Experiment 2, 3.5-month-old infants saw a test event in which an object was lowered either behind (behind-container condition) or inside (inside-container condition) a container; next, the container was moved forward and to the side, revealing the object behind it. The infants looked reliably longer at the inside- than at the behind-container test event. These and baseline results suggested that the infants in the inside-container condition realized that the object could not pass through the back wall of the container and hence should have moved with it to its new location. Experiments 3 and 4 extended the results of Experiments 1 and 2 to 2.5-month-old infants. Together, the present results indicate that even very young infants possess expectations about containment events. The possible origins and development of these expectations are discussed in the context of Baillargeon's model (Advances in infancy research 9 (1995) 305. Norwood, NJ: Ablex) of infants' acquisition of physical knowledge, and of Spelke's proposal (Cognition 50 (1994) 431) that, from birth, infants interpret physical events in accord with a solidity principle.     

Development of multisensory spatial integration and perception in humans

Previous studies have shown that adults respond faster and more reliably to bimodal compared to unimodal localization cues. The current study investigated for the first time the development of audiovisual (A-V) integration in spatial localization behavior in infants between 1 and 10 months of age. We observed infants' head and eye movements in response to auditory, visual, or both kinds of stimuli presented either 25 degrees or 45 degrees to the right or left of midline. Infants under 8 months of age intermittently showed response latencies significantly faster toward audiovisual targets than toward either auditory or visual targets alone They did so, however, without exhibiting a reliable violation of the Race Model, suggesting that probability summation alone could explain the faster bimodal response. In contrast, infants between 8 and 10 months of age exhibited bimodal response latencies significantly faster than unimodal latencies for both eccentricity conditions and their latencies violated the Race Model at 25 degrees eccentricity. In addition to this main finding, we found age-dependent eccentricity and modality effects on response latencies. Together, these findings suggest that audiovisual integration emerges late in the first year of life and are consistent with neurophysiological findings from multisensory sites in the superior colliculus of infant monkeys showing that multisensory enhancement of responsiveness is not present at birth but emerges later in life.     

Precursors to infants'' perception of the causality of a simple event

Three habituation experiments examined the perception of causal and non-causal events by infants at 6 1/4, 5 1/2, and 4 months of age. Experiment 1 replicated previous studies showing that by 6 1/4 months, infants begin to respond to the interaction of simple objects in Michottian type launching events on the basis of causality. In Experiments 2 and 3, however, younger infants exposed to the identical event sequences responded on the basis of simpler perceptual features of the launching events, and not on the basis of causality. 4-month-old infants responded to continuous versus non-continuous movement. 5 1/2-month-old infants also responded somewhat on the basis of continuous movement. However, in addition they responded on the basis of the spatial and temporal perceptual features of the events. This change in the pattern of results over age suggests a multi-step developmental progression.     

Infants' perceptual differentiation of amodal and modality-specific audio-visual relations.

Ninety-six infants of 3 1/2 months were tested in an infant-control habituation procedure to determine whether they could detect three types of audio-visual relations in the same events. The events portrayed two amodal invariant relations, temporal synchrony and temporal microstructure specifying the composition of the objects, and one modality-specific relation, that between the pitch of the sound and the color/shape of the objects. Subjects were habituated to two events accompanied by their natural, synchronous, and appropriate sounds and then received test trials in which the relation between the visual and the acoustic information was changed. Consistent with Gibson's increasing specificity hypothesis, it was expected that infants would differentiate amodal invariant relations prior to detecting arbitrary, modality-specific relations. Results were consistent with this prediction, demonstrating significant visual recovery to a change in temporal synchrony and temporal microstructure, but not to a change in the pitch-color/shape relations. Two subsequent discrimination studies demonstrated that infants' failure to detect the changes in pitch-color/shape relations could not be attributed to an inability to discriminate the pitch or the color/shape changes used in Experiment 1. Infants showed robust discrimination of the contrasts used.     

Perception of serial order in infants

Serial order is fundamental to perception, cognition and behavioral action. Three experiments investigated infants' perception, learning and discrimination of serial order. Four- and 8-month-old infants were habituated to three sequentially moving objects making visible and audible impacts and then were tested on separate test trials for their ability to detect auditory, visual or auditory-visual changes in their ordering. The 4-month-old infants did not respond to any order changes and instead appeared to attend to the 'local' audio-visual synchrony part of the event. When this local part of the event was blocked from view, the 4-month-olds did perceive the serial order feature of the event but only when it was specified multimodally. In contrast, the 8-month-old infants perceived all three kinds of order changes regardless of whether the synchrony part of the event was visible or not. The findings show that perception of spatiotemporal serial order emerges early in infancy and that its perception is initially facilitated by multimodal specification.     

Infants' concept of animacy

Nine- and 12-month-old infants' concept of animacy was investigated by exposing them to autonomous motion by an animate and by an inanimate object in a series of three experiments. In the first experiment, increases in negative affect in comparison to a baseline condition were taken to indicate that children considered an event to be anomalous. Results showed that 12-month-old infants consider self-propulsion by a small robot to be anomalous, but not self-propulsion by a human stranger. Experiment 2 indicated that 9- and 12-month-old infants expressed similar affective reactions when the robot's motion was contingent on verbal commands given by the mother, suggesting that these children are aware that it is not appropriate for an inanimate object's movements to be contingent on events occuring at a distance. The third experiment was designed to rule out the possibility that the infants' reactions in Experiment 2 were a function of the incongruity of the mother's behavior rather than due to the violation of the infant's concept of animacy. In this experiment, 12-month-olds' levels of attentiveness are increased when the robot obeyed verbal commands but not when a human stranger did so. These results suggest that infants discriminate animate from inanimate objects on the basis of motion cues by the age of 9 months.     

Brief pitch-priming facilitates infants’ discrimination of pitch-evoking noise: Evidence from event-related potentials

Pitch is derived by the auditory system through complex spectrotemporal processing. Pitch extraction is thought to depend on both spectral cues arising from lower harmonics that are resolved by cochlear filters in the inner ear, and on temporal cues arising from the pattern of action potentials contained in the cochlear output. Adults are capable of extracting pitch in the absence of robust spectral cues, taking advantage of the temporal cues that remain. However, recent behavioral evidence suggests that infants have difficulty discriminating between stimuli with different pitches when resolvable spectral cues are absent. In the current experiments, we used the mismatch negativity (MMN) component of the event related potential derived from electroencephalographic (EEG) recordings to examine a cortical representation of pitch discrimination for iterated rippled noise (IRN) stimuli in 4- and 8-month-old infants. IRN stimuli are pitch-evoking sounds generated by repeatedly adding a segment of white noise to itself at a constant delay. We created IRN stimuli (delays of 5 and 6 ms creating pitch percepts of 200 and 167 Hz) and high-pass filtered them to remove all resolvable spectral pitch cues. In experiment 1, we did not find EEG evidence that infants could detect the change in the pitch of these IRN stimuli. However, in Experiment 2, after a brief period of pitch-priming during which we added a sine wave component to the IRN stimulus at its perceived pitch, infants did show significant MMN in response to pitch changes in the IRN stimuli with sine waves removed. This suggests that (1) infants can use temporal cues to process pitch, although such processing is not mature and (2) that a short amount of pitch-priming experience can alter pitch representations in auditory cortex during infancy.     

The development of young infants' intuitions about support

Previous research has shown that 3-month-old infants, like adults, expect a box to be stable when it is in full contact with a platform, and to fall when it loses all contact with the platform. Do young infants also have expectations about what should happen when the box is only in partial contact with the platform? The present research was designed to address this question. In Experiment 1, 6.5-month-old infants saw two test events: a full-contact and a partial-contact test event. In both events, the infants watched the extended finger of a gloved hand push a box along the top of a platform. In the full-contact event, the box was pushed until its leading edge reached the end of the platform. In the partial-contact event, the box was pushed until only 15% or 70% of its bottom surface remained on the platform. The infants looked reliably longer at the partial-than at the full-contact event when 15%, but not 70%, of the box rested on the platform. These results suggested that the infants were able to judge how much contact was needed between the box and the platform for the box to be stable. A control condition provided evidence for this interpretation. In Experiment 2, 5.5- to 6-month-old infants were found to look equally at the full- and the partial-contact events, even when only 15% of the box's bottom surface remained on the platform. This result suggested that prior to 6.5 months of age infants perceive any amount of contact between the box and the platform to be sufficient to ensure the box's stability. Interpretations of this developmental sequence are considered in the Conclusion.     

Who is crossing where? Infants' discrimination of figures and grounds in events

To learn relational terms such as verbs and prepositions, children must first dissect and process dynamic event components. This paper investigates the way in which 8- to 14-month-old English-reared infants notice the event components, figure (i.e., the moving entity) and ground (i.e., stationary setting), in both dynamic (Experiment 1) and static representations of events (Experiment 2) for categorical ground distinctions expressed in Japanese, but not in English. We then compare both 14- and 19-month-old English- and Japanese-reared infants’ processing of grounds to understand how language learning interacts with the conceptualization of these constructs (Experiment 3). Results suggest that (1) infants distinguish between figures and grounds in events; (2) they do so differently for static vs. dynamic displays; (3) early in the second year, children from diverse language environments form nonnative – perhaps universal – event categories; and (4) these event categories shift over time as children have more exposure to their native tongue.     

Crossmodal duration perception involves perceptual grouping, temporal ventriloquism, and variable internal clock rates

Here, we investigate how audiovisual context affects perceived event duration with experiments in which observers reported which of two stimuli they perceived as longer. Target events were visual and/or auditory and could be accompanied by nontargets in the other modality. Our results demonstrate that the temporal information conveyed by irrelevant sounds is automatically used when the brain estimates visual durations but that irrelevant visual information does not affect perceived auditory duration (Experiment 1). We further show that auditory influences on subjective visual durations occur only when the temporal characteristics of the stimuli promote perceptual grouping (Experiments 1 and 2). Placed in the context of scalar expectancy theory of time perception, our third and fourth experiments have the implication that audiovisual context can lead both to changes in the rate of an internal clock and to temporal ventriloquism-like effects on perceived on- and offsets. Finally, intramodal grouping of auditory stimuli diminished any crossmodal effects, suggesting a strong preference for intramodal over crossmodal perceptual grouping (Experiment 5).     

Brief pitch-priming facilitates infants’ discrimination of pitch-evoking noise: Evidence from event-related potentials

Pitch is derived by the auditory system through complex spectrotemporal processing. Pitch extraction is thought to depend on both spectral cues arising from lower harmonics that are resolved by cochlear filters in the inner ear, and on temporal cues arising from the pattern of action potentials contained in the cochlear output. Adults are capable of extracting pitch in the absence of robust spectral cues, taking advantage of the temporal cues that remain. However, recent behavioral evidence suggests that infants have difficulty discriminating between stimuli with different pitches when resolvable spectral cues are absent. In the current experiments, we used the mismatch negativity (MMN) component of the event related potential derived from electroencephalographic (EEG) recordings to examine a cortical representation of pitch discrimination for iterated rippled noise (IRN) stimuli in 4- and 8-month-old infants. IRN stimuli are pitch-evoking sounds generated by repeatedly adding a segment of white noise to itself at a constant delay. We created IRN stimuli (delays of 5 and 6 ms creating pitch percepts of 200 and 167 Hz) and high-pass filtered them to remove all resolvable spectral pitch cues. In experiment 1, we did not find EEG evidence that infants could detect the change in the pitch of these IRN stimuli. However, in Experiment 2, after a brief period of pitch-priming during which we added a sine wave component to the IRN stimulus at its perceived pitch, infants did show significant MMN in response to pitch changes in the IRN stimuli with sine waves removed. This suggests that (1) infants can use temporal cues to process pitch, although such processing is not mature and (2) that a short amount of pitch-priming experience can alter pitch representations in auditory cortex during infancy.     

Exploring early developmental changes in face scanning patterns during the perception of audiovisual mismatch of speech cues

Young infants are capable of integrating auditory and visual information and their speech perception can be influenced by visual cues, while 5-month-olds detect mismatch between mouth articulations and speech sounds. From 6 months of age, infants gradually shift their attention away from eyes and towards the mouth in articulating faces, potentially to benefit from intersensory redundancy of audiovisual (AV) cues. Using eye tracking, we investigated whether 6- to 9-month-olds showed a similar age-related increase of looking to the mouth, while observing congruent and/or redundant versus mismatched and non-redundant speech cues. Participants distinguished between congruent and incongruent AV cues as reflected by the amount of looking to the mouth. They showed an age-related increase in attention to the mouth, but only for non-redundant, mismatched AV speech cues. Our results highlight the role of intersensory redundancy and audiovisual mismatch mechanisms in facilitating the development of speech processing in infants under 12 months of age.