Effects of multimodal presentation and stimulus familiarity on auditory and visual processing

Two experiments examined the effects of multimodal presentation and stimulus familiarity on auditory and visual processing. In Experiment 1, 10-month-olds were habituated to either an auditory stimulus, a visual stimulus, or an auditory-visual multimodal stimulus. Processing time was assessed during the habituation phase, and discrimination of auditory and visual stimuli was assessed during a subsequent testing phase. In Experiment 2, the familiarity of the auditory or visual stimulus was systematically manipulated by prefamiliarizing infants to either the auditory or visual stimulus prior to the experiment proper. With the exception of the prefamiliarized auditory condition in Experiment 2, infants in the multimodal conditions failed to increase looking when the visual component changed at test. This finding is noteworthy given that infants discriminated the same visual stimuli when presented unimodally, and there was no evidence that multimodal presentation attenuated auditory processing. Possible factors underlying these effects are discussed.     

Hearing what the body feels: auditory encoding of rhythmic movement

Phillips-Silver and Trainor (Phillips-Silver, J., Trainor, L.J., (2005). Feeling the beat: movement influences infants' rhythm perception. Science, 308, 1430) demonstrated an early cross-modal interaction between body movement and auditory encoding of musical rhythm in infants. Here we show that the way adults move their bodies to music influences their auditory perception of the rhythm structure. We trained adults, while listening to an ambiguous rhythm with no accented beats, to bounce by bending their knees to interpret the rhythm either as a march or as a waltz. At test, adults identified as similar an auditory version of the rhythm pattern with accented strong beats that matched their previous bouncing experience in comparison with a version whose accents did not match. In subsequent experiments we showed that this effect does not depend on visual information, but that movement of the body is critical. Parallel results from adults and infants suggest that the movement-sound interaction develops early and is fundamental to music processing throughout life.     

10‐month‐old infants’ inference of invisible agent: Distinction in causality between object motion and human action1

Abstract: Two habituation experiments investigated 10‐month‐old infants’ interpretation of events where a stationary object began to move without any visible causes. During habituation, infants saw that an object partly hidden by an occluder began to move away from the occluder. Then, they were tested with three test events without the occluder: the ?rst event showed a hand pushing the object, the second event showed a hand failing to touch the object, and the last event had no agent. The objects were a ball in Experiment 1, and a person in Experiment 2. The test event that the infants looked at for the shortest duration in Experiment 1 was where the hand pushed the ball, whereas they looked at the three test events almost equal amounts of time in Experiment 2. These results indicate that 10‐month‐old infants responded to the events in terms of causality and could infer the presence of the agent behind the occluder only when they saw the habituation event featuring the ball.     

Infants’ auditory enumeration: Evidence for analog magnitudes in the small number range

Vigorous debate surrounds the issue of whether infants use different representational mechanisms to discriminate small and large numbers. We report evidence for ratio-dependent performance in infants’ discrimination of small numbers of auditory events, suggesting that infants can use analog magnitudes to represent small values, at least in the auditory domain. Seven-month-old infants in the present study reliably discriminated two from four tones (a 1:2 ratio) in Experiment 1, when melodic and continuous temporal properties of the sequences were controlled, but failed to discriminate two from three tones (a 2:3 ratio) under the same conditions in Experiment 2. A third experiment ruled out the possibility that infants in Experiment 1 were responding to greater melodic variety in the four-tone sequences. The discrimination function obtained here is the same as that found for infants’ discrimination of large numbers of visual and auditory items at a similar age, as well as for that obtained for similar-aged infants’ duration discriminations, and thus adds to a growing body of evidence suggesting that human infants may share with adults and nonhuman animals a mechanism for representing quantities as “noisy” mental magnitudes.     

The auditory motionaftereffect: Its tuning and specificity in the spatial and frequency domains

In this paper, the auditory motion aftereffect (aMAE) was studied, using real moving sound as both the adapting and the test stimulus. The sound was generated by a loudspeaker mounted on a robot arm that was able to move quietly in three-dimensional space. A total of 7 subjects with normal hearing were tested in three experiments. The results from Experiment 1 showed a robust and reliable negative aMAE in all the subjects. After listening to a sound source moving repeatedly to the right, a stationary sound source was perceived to move to the left. The magnitude of the aMAE tended to increase with adapting velocity up to the highest velocity tested (20°/sec). The aftereffect was largest when the adapting and the test stimuli had similar spatial location and frequency content. Offsetting the locations of the adapting and the test stimuli by 20° reduced the size of the effect by about 50%. A similar decline occurred when the frequency of the adapting and the test stimuli differed by one octave. Our results suggest that the human auditory system possesses specialized mechanisms for detecting auditory motion in the spatial domain.     

The auditory motion aftereffect: its tuning and specificity in the spatial and frequency domains

In this paper, the auditory motion aftereffect (aMAE) was studied, using real moving sound as both the adapting and the test stimulus. The sound was generated by a loudspeaker mounted on a robot arm that was able to move quietly in three-dimensional space. A total of 7 subjects with normal hearing were tested in three experiments. The results from Experiment 1 showed a robust and reliable negative aMAE in all the subjects. After listening to a sound source moving repeatedly to the right, a stationary sound source was perceived to move to the left. The magnitude of the aMAE tended to increase with adapting velocity up to the highest velocity tested (20 degrees/sec). The aftereffect was largest when the adapting and the test stimuli had similar spatial location and frequency content. Offsetting the locations of the adapting and the test stimuli by 20 degrees reduced the size of the effect by about 50%. A similar decline occurred when the frequency of the adapting and the test stimuli differed by one octave. Our results suggest that the human auditory system possesses specialized mechanisms for detecting auditory motion in the spatial domain.     

Auditory-visual intermodal matching of small numerosities in 6-month-old infants

Recent studies have reported that preverbal infants are able to discriminate between numerosities of sets presented within a particular modality. There is still debate, however, over whether they are able to perform intermodal numerosity matching, i.e. to relate numerosities of sets presented with different sensory modalities. The present study investigated auditory-visual intermodal matching of small numerosities in infancy by using a violation-of-expectation paradigm. After being familiarized with events of a few objects impacting a surface successively, 6-month-old infants were alternatively presented with two and three tones while the movement of each object remained hidden behind an opaque screen. The screen was then removed to reveal either two or three objects. Results showed that the infants looked significantly longer at the numerically nonequivalent events (the three-tone/two-object and the two-tone/three-object events) than at the numerically equivalent events (the two-tone/two-object and the three-tone/three-object events) irrespective of the rate or duration of auditory tones presented. These findings suggest that infants are capable of performing intermodal matching of small numerosities and that they might possess abstract representations of numerosity beyond sensory modalities.     

The McGurk effect in infants

In the McGurk effect, perceptual identification of auditory speech syllables is influenced by simultaneous presentation of discrepant visible speech syllables. This effect has been found in subjects of different ages and with various native language backgrounds. But no McGurk tests have been conducted with prelinguistic infants. In the present series of experiments, 5-month-old English-exposed infants were tested for the McGurk effect. Infants were first gaze-habituated to an audiovisual /va/. Two different dishabituation stimuli were then presented: audio /ba/-visual /va/ (perceived by adults as /va/), and audio /da/-visual /va/ (perceived by adults as /da/). The infants showed generalization from the audiovisual /va/ to the audio /ba/-visual /va/ stimulus but not to the audio /da/-visual /va/ stimulus. Follow-up experiments revealed that these generalization differences were not due to a general preference for the audio /da/-visual /va/ stimulus or to the auditory similarity of /ba/ to /va/ relative to /da/. These results suggest that the infants were visually influenced in the same way as Englishspeaking adults are visually influenced.     

Learning and discrimination of audiovisual events in human infants: the hierarchical relation between intersensory temporal synchrony and rhythmic pattern cues

This study examined 4- to 10-month-old infants' perception of audio-visual (A-V) temporal synchrony cues in the presence or absence of rhythmic pattern cues. Experiment 1 established that infants of all ages could successfully discriminate between two different audiovisual rhythmic events. Experiment 2 showed that only 10-month-old infants detected a desynchronization of the auditory and visual components of a rhythmical event. Experiment 3 showed that 4- to 8-month-old infants could detect A-V desynchronization but only when the audiovisual event was nonrhythmic. These results show that initially in development infants attend to the overall temporal structure of rhythmic audiovisual events but that later in development they become capable of perceiving the embedded intersensory temporal synchrony relations as well.     

No two cues are alike: Depth of learning during infancy is dependent on what orients attention

Human infants develop a variety of attentional mechanisms that allow them to extract relevant information from a cluttered multimodal world. We know that both social and nonsocial cues shift infants’ attention, but not how these cues differentially affect learning of multimodal events. Experiment 1 used social cues to direct 8- and 4-month-olds’ attention to two audiovisual events (i.e., animations of a cat or dog accompanied by particular sounds) while identical distractor events played in another location. Experiment 2 directed 8-month-olds’ attention with colorful flashes to the same events. Experiment 3 measured baseline learning without attention cues both with the familiarization and test trials (no cue condition) and with only the test trials (test control condition). The 8-month-olds exposed to social cues showed specific learning of audiovisual events. The 4-month-olds displayed only general spatial learning from social cues, suggesting that specific learning of audiovisual events from social cues may be a function of experience. Infants cued with the colorful flashes looked indiscriminately to both cued locations during test (similar to the 4-month-olds learning from social cues) despite attending for equal duration to the training trials as the 8-month-olds with the social cues. Results from Experiment 3 indicated that the learning effects in Experiments 1 and 2 resulted from exposure to the different cues and multimodal events. We discuss these findings in terms of the perceptual differences and relevance of the cues.     

Should all stationary objects move when hit? Developments in infants’ causal and statistical expectations about collision events

Four experiments examined 8- and 9-month-old infants’ expectations about collision events. The infants saw test events in which a small cylinder rolled down a ramp and hit one of several different boxes. These boxes varied in width and height and always remained stationary when hit. The results revealed two separate developments. The first involved infants’ knowledge of the variables relevant to collision events. At 8 months, the infants expected all of the boxes to move when hit, regardless of their sizes; at 9 months, the infants began to take into account the size of the boxes to predict whether they should move when hit. The second development concerned infants’ ability to generate explanations for outcomes that violated their collision knowledge. At both ages, upon observing that a box with a salient vertical dimension did not move when hit, the infants apparently concluded that the box must be one of those objects we term pillars—vertical objects that are attached at one or both ends to adjacent surfaces. At 8 months, the infants considered any vertical box as a potential pillar; at 9 months, the infants considered only boxes that were both vertical and narrow as potential pillars. The development of infants’ knowledge about collision events is thus one that is complex and protracted and weaves together many separate developments.     

The sound of darkness: why do auditory cues aid infants' search for objects hidden by darkness but not by visible occluders?

In manual search tasks designed to assess infants' knowledge of the object concept, why does search for objects hidden by darkness precede search for objects hidden by visible occluders by several months? A graded representations account explains this décalage by proposing that the conflicting visual input from occluders directly competes with object representations, whereas darkness merely weakens representations. This study tests the prediction that representations of objects hidden by darkness are strong enough for infants to bind auditory cues to them and support search, whereas representations of objects hidden by occluders are not. Six-and-half-month-olds were presented with audible or silent objects that remained visible, became hidden by darkness, or became hidden by a visible occluder. Search required engaging in the same means-end action in all conditions. As predicted, auditory cues increased search when objects were hidden by darkness but not when they were hidden by a visible occluder. Results are discussed in the context of different facets of object concept development highlighted by graded representations perspectives and core knowledge perspectives and in relation to other work on multimodal object representations.     

Infants' and adults' use of duration and intensity cues in the segmentation of tone patterns

Adults and 8-month-olds were presented with sequences in which every third complex tone was either longer or more intense. Segmentation was measured by comparing the detection of silent gaps inserted into three possible locations in each pattern: Silent gaps inserted at perceived segmentation boundaries are harder to detect than gaps within perceived phrases or groups. A go/no-go conditioned head-turn (hand-raising for adults) procedure was used. In Experiment 1, detection was worse for the gaps following the longer complex tones than for the gaps at the other locations, suggesting that the longer tones marked the ends of perceived groups for both infants and adults. Experiment 2 showed that an increase in intensity did not result in any systematic grouping at either age.     

Sensorimotor foundations of speech perception in infancy

The perceptual system for speech is highly organized from early infancy. This organization bootstraps young human learners’ ability to acquire their native speech and language from speech input. Here, we review behavioral and neuroimaging evidence that perceptual systems beyond the auditory modality are also specialized for speech in infancy, and that motor and sensorimotor systems can influence speech perception even in infants too young to produce speech-like vocalizations. These investigations complement existing literature on infant vocal development and on the interplay between speech perception and production systems in adults. We conclude that a multimodal speech and language network is present before speech-like vocalizations emerge.     

Examining inhibition of return with multiple sequential cues in younger and older adults

Studies with younger adults have shown that when multiple peripheral cues are presented sequentially, inhibition of return (IOR) occurs at several locations with the greatest IOR at the most recently cued location and the least at the earliest cued location. The inhibitory ability needed to tag multiple locations requires visuospatial working memory, and it is thought that this type of memory may be vulnerable to the effects of aging. The present experiments examined whether older adults would show less IOR at multiple cued locations than younger adults when placeholders were present (Experiment 1) and absent (Experiment 2). Of interest, in both experiments older adults showed an almost identical pattern of IOR, in both magnitude and number of inhibited locations, to that of younger adults. This finding, in conjunction with research on memory-guided saccades, suggests that there may be a form of visuospatial working memory, specific to oculomotor and visual attention processes, that is relatively resistant to the effects of aging.     

Comparison of infant and adult P300 from auditory stimuli.

Event-related brain potentials (ERPs) using auditory stimuli were recorded from 5- to 10-month-old infants and normal young adults with a passive tone sequence paradigm. A P3 (P300) component was obtained which demonstrated the same central-parietal maximum scalp distribution for both subject groups. P3 amplitude was smaller and its peak latency longer for infants compared to those of the adults across all electrode sites. P3 measures remained stable across stimulus trials indicating that ERP habituation was not occurring. Evaluation of individual subjects suggests that the P3 can be elicited from infants with auditory stimuli in a manner similar to that from adults and may serve as a useful index for the assessment of cognitive function in infants.     

Infants' responsiveness to the auditory and visual attributes of a sounding/moving stimulus.

Responses to unimodal and multimodal attributes of a compound auditory/visual stimulus were investigated in 4-, 6-, 8-, and 10-month-old infants. First, infants were habituated to a compound stimulus consisting of a visual stimulus that moved up and down on a video monitor and a sound that occurred each time the visual stimulus reversed direction at the bottom. Once each infant met a habituation criterion, a series of test trials was administered to assess responsiveness to the components of the compound stimulus. Response was defined as the total duration of visual fixation in each trial. In the two unimodal test trials, the rate at which the component was presented was changed while the rate of the other component remained the same, whereas in the bimodal test trial the rate of both components was changed simultaneously. Results indicated that infants at each age successfully discriminated the bimodal and the two unimodal changes and that regression to the mean did not account for the results. Results also showed that disruption of the temporal relationship that accompanied the change in rate in the two unimodal test trials was also discriminable, but rate changes appeared to play a greater role in responsiveness than did synchrony changes. Considered together with results from similar prior studies, the current results are consistent with the modality appropriateness hypothesis in showing that discrimination of temporal changes in the auditory and visual modalities is dependent on the specialization of the sensory modalities.     

Electromyographic responses in infants after auditory stimulation.

49 of 50 newborn infants, when presented with a strong auditory stimulus, showed significant increases in EMG potential. The infant who failed initially responded when rechecked in 1 mo. The results suggest it may be possible to develop an EMG-Audiometric process which could be employed in screening hearing of newborn infants.     

Hemodynamic responses to speech and music in preverbal infants

Numerous studies have provided clues about the ontogeny of lateralization of auditory processing in humans, but most have employed specific subtypes of stimuli and/or have assessed responses in discrete temporal windows. The present study used near-infrared spectroscopy (NIRS) to establish changes in hemodynamic activity in the neocortex of preverbal infants (aged 4–11 months) while they were exposed to two distinct types of complex auditory stimuli (full sentences and musical phrases). Measurements were taken from bilateral temporal regions, including both anterior and posterior superior temporal gyri. When the infant sample was treated as a homogenous group, no significant effects emerged for stimulus type. However, when infants’ hemodynamic responses were categorized according to their overall changes in volume, two very clear neurophysiological patterns emerged. A high-responder group showed a pattern of early and increasing activation, primarily in the left hemisphere, similar to that observed in comparable studies with adults. In contrast, a low-responder group showed a pattern of gradual decreases in activation over time. Although age did track with responder type, no significant differences between these groups emerged for stimulus type, suggesting that the high- versus low-responder characterization generalizes across classes of auditory stimuli. These results highlight a new way to conceptualize the variable cortical blood flow patterns that are frequently observed across infants and stimuli, with hemodynamic response volumes potentially serving as an early indicator of developmental changes in auditory-processing sensitivity.     

The influence of infant-directed speech on 12-month-olds’ intersensory perception of fluent speech

The present study examined whether infant-directed (ID) speech facilitates intersensory matching of audio–visual fluent speech in 12-month-old infants. German-learning infants’ audio–visual matching ability of German and French fluent speech was assessed by using a variant of the intermodal matching procedure, with auditory and visual speech information presented sequentially. In Experiment 1, the sentences were spoken in an adult-directed (AD) manner. Results showed that 12-month-old infants did not exhibit a matching performance for the native, nor for the non-native language. However, Experiment 2 revealed that when ID speech stimuli were used, infants did perceive the relation between auditory and visual speech attributes, but only in response to their native language. Thus, the findings suggest that ID speech might have an influence on the intersensory perception of fluent speech and shed further light on multisensory perceptual narrowing.