Cross-modal speech perception in adults and infants using nonspeech auditory stimuli.

Adults and infants were tested for the capacity to detect correspondences between nonspeech sounds and real vowels. The /i/ and /a/ vowels were presented in 3 different ways: auditory speech, silent visual faces articulating the vowels, or mentally imagined vowels. The nonspeech sounds were either pure tones or 3-tone complexes that isolated a single feature of the vowel without allowing the vowel to be identified. Adults perceived an orderly relation between the nonspeech sounds and vowels. They matched high-pitched nonspeech sounds to /i/ vowels and low-pitched nonspeech sounds to /a/ vowels. In contrast, infants could not match nonspeech sounds to the visually presented vowels. Infants' detection of correspondence between auditory and visual speech appears to require the whole speech signal; with development, an isolated feature of the vowel is sufficient for detection of the cross-modal correspondence.     

Auditory instructional cues benefit unimanual and bimanual drawing in Parkinson's disease patients

The present study investigated performance of unimanual and bimanual anti-phase and in-phase upper limb line drawing using three different types of cues. Fifteen Parkinson’s disease (PD) patients, 15 elderly, and 15 young adults drew lines away from and towards their body on a tabletop every 1000 ms for 30 s under three different cueing conditions: (1) verbal (‘up’, ‘down’); (2) auditory (high tone, low tone); (3) visual (target line switched from top to bottom). PD patients had larger and more variable amplitudes which may be related to the finding that they also produced more curvilinear movements than young and elderly adults. Consistent with previous research, when compared to the elderly and young adult group PD patients produced a mean relative phase which deviated more from the instructed coordination modes and they showed larger variability of relative phase in bimanual coordination, especially in anti-phase conditions. For all groups, auditory and verbal cues resulted in lower coefficient of variance of cycle time, lower variability of amplitude and lower variability of relative phase than visual cues. The benefit of auditory cues may be related to the timing nature of the task or factors related to the auditory cues (e.g., reduced attentional demands, more kinesthetic focus).     

Object labeling influences infant phonetic learning and generalization

Different kinds of speech sounds are used to signify possible word forms in every language. For example, lexical stress is used in Spanish (/‘be.be/, ‘he/she drinks’ versus /be.’be/, ‘baby’), but not in French (/‘be.be/ and /be.’be/ both mean ‘baby’). Infants learn many such native language phonetic contrasts in their first year of life, likely using a number of cues from parental speech input. One such cue could be parents’ object labeling, which can explicitly highlight relevant contrasts. Here we ask whether phonetic learning from object labeling is abstract—that is, if learning can generalize to new phonetic contexts. We investigate this issue in the prosodic domain, as the abstraction of prosodic cues (like lexical stress) has been shown to be particularly difficult. One group of 10-month-old French-learners was given consistent word labels that contrasted on lexical stress (e.g., Object A was labeled /‘ma.bu/, and Object B was labeled /ma.’bu/). Another group of 10-month-olds was given inconsistent word labels (i.e., mixed pairings), and stress discrimination in both groups was measured in a test phase with words made up of new syllables. Infants trained with consistently contrastive labels showed an earlier effect of discrimination compared to infants trained with inconsistent labels. Results indicate that phonetic learning from object labeling can indeed generalize, and suggest one way infants may learn the sound properties of their native language(s).     

Exploring early developmental changes in face scanning patterns during the perception of audiovisual mismatch of speech cues

Young infants are capable of integrating auditory and visual information and their speech perception can be influenced by visual cues, while 5-month-olds detect mismatch between mouth articulations and speech sounds. From 6 months of age, infants gradually shift their attention away from eyes and towards the mouth in articulating faces, potentially to benefit from intersensory redundancy of audiovisual (AV) cues. Using eye tracking, we investigated whether 6- to 9-month-olds showed a similar age-related increase of looking to the mouth, while observing congruent and/or redundant versus mismatched and non-redundant speech cues. Participants distinguished between congruent and incongruent AV cues as reflected by the amount of looking to the mouth. They showed an age-related increase in attention to the mouth, but only for non-redundant, mismatched AV speech cues. Our results highlight the role of intersensory redundancy and audiovisual mismatch mechanisms in facilitating the development of speech processing in infants under 12 months of age.     

Assessing the double phonemic representation in bilingual speakers of Spanish and English: an electrophysiological study

Event Related Potentials (ERPs) were recorded from Spanish-English bilinguals (N = 10) to test pre-attentive speech discrimination in two language contexts. ERPs were recorded while participants silently read magazines in English or Spanish. Two speech contrast conditions were recorded in each language context. In the phonemic in English condition, the speech sounds represented two different phonemic categories in English, but represented the same phonemic category in Spanish. In the phonemic in Spanish condition, the speech sounds represented two different phonemic categories in Spanish, but represented the same phonemic categories in English. Results showed pre-attentive discrimination when the acoustics/phonetics of the speech sounds match the language context (e.g., phonemic in English condition during the English language context). The results suggest that language contexts can affect pre-attentive auditory change detection. Specifically, bilinguals’ mental processing of stop consonants relies on contextual linguistic information.     

Learning words’ sounds before learning how words sound: 9-Month-olds use distinct objects as cues to categorize speech information

One of the central themes in the study of language acquisition is the gap between the linguistic knowledge that learners demonstrate, and the apparent inadequacy of linguistic input to support induction of this knowledge. One of the first linguistic abilities in the course of development to exemplify this problem is in speech perception: specifically, learning the sound system of one’s native language. Native-language sound systems are defined by meaningful contrasts among words in a language, yet infants learn these sound patterns before any significant numbers of words are acquired. Previous approaches to this learning problem have suggested that infants can learn phonetic categories from statistical analysis of auditory input, without regard to word referents. Experimental evidence presented here suggests instead that young infants can use visual cues present in word-labeling situations to categorize phonetic information. In Experiment 1, 9-month-old English-learning infants failed to discriminate two non-native phonetic categories, establishing baseline performance in a perceptual discrimination task. In Experiment 2, these infants succeeded at discrimination after watching contrasting visual cues (i.e., videos of two novel objects) paired consistently with the two non-native phonetic categories. In Experiment 3, these infants failed at discrimination after watching the same visual cues, but paired inconsistently with the two phonetic categories. At an age before which memory of word labels is demonstrated in the laboratory, 9-month-old infants use contrastive pairings between objects and sounds to influence their phonetic sensitivity. Phonetic learning may have a more functional basis than previous statistical learning mechanisms assume: infants may use cross-modal associations inherent in social contexts to learn native-language phonetic categories.     

Infant-directed speech supports phonetic category learning in English and Japanese

Across the first year of life, infants show decreased sensitivity to phonetic differences not used in the native language [Werker, J. F., & Tees, R. C. (1984). Cross-language speech perception: evidence for perceptual reorganization during the first year of life. Infant Behaviour and Development, 7, 49-63]. In an artificial language learning manipulation, Maye, Werker, and Gerken [Maye, J., Werker, J. F., & Gerken, L. (2002). Infant sensitivity to distributional information can affect phonetic discrimination. Cognition, 82(3), B101-B111] found that infants change their speech sound categories as a function of the distributional properties of the input. For such a distributional learning mechanism to be functional, however, it is essential that the input speech contain distributional cues to support such perceptual learning. To test this, we recorded Japanese and English mothers teaching words to their infants. Acoustic analyses revealed language-specific differences in the distributions of the cues used by mothers (or cues present in the input) to distinguish the vowels. The robust availability of these cues in maternal speech adds support to the hypothesis that distributional learning is an important mechanism whereby infants establish native language phonetic categories.     

Seeing facial motion affects auditory processing in noise

Speech perception, especially in noise, may be maximized if the perceiver observes the naturally occurring visual-plus-auditory cues inherent in the production of spoken language. Evidence is conflicting, however, about which aspects of visual information mediate enhanced speech perception in noise. For this reason, we investigated the relative contributions of audibility and the type of visual cue in three experiments in young adults with normal hearing and vision. Relative to static visual cues, access to the talker??s phonetic gestures in speech production, especially in noise, was associated with (a) faster response times and sensitivity for speech understanding in noise, and (b) shorter latencies and reduced amplitudes of auditory N1 event-related potentials. Dynamic chewing facial motion also decreased the N1 latency, but only meaningful linguistic motions reduced the N1 amplitude. The hypothesis that auditory?Cvisual facilitation is distinct to properties of natural, dynamic speech gestures was partially supported.     

Do production patterns influence the processing of speech in prelinguistic infants?

The headturn preference procedure was used to test 18 infants on their response to three different passages chosen to reflect their individual production patterns. The passages contained nonwords with consonants in one of three categories: (a) often produced by that infant (‘own’), (b) rarely produced by that infant but common at that age (‘other’), and (c) not generally produced by infants. Infants who had a single ‘own’ consonant showed no significant preference for either ‘own’ (a) or ‘other’ (b) passages. In contrast, infants’ with two ‘own’ consonants exhibited greater attention to ‘other’ passages (b). Both groups attended equally to the passage featuring consonants rarely produced by infants of that age (c). An analysis of a sample of the infant-directed speech ruled out the mothers’ speech as a source of the infant preferences. The production-based shift to a focus on the ‘other’ passage suggests that nascent production abilities combine with emergent perceptual experience to facilitate word learning.     

Visual speech contributes to phonetic learning in 6-month-old infants

Previous research has shown that infants match vowel sounds to facial displays of vowel articulation [Kuhl, P. K., & Meltzoff, A. N. (1982). The bimodal perception of speech in infancy. Science, 218, 1138–1141; Patterson, M. L., & Werker, J. F. (1999). Matching phonetic information in lips and voice is robust in 4.5-month-old infants. Infant Behaviour & Development, 22, 237–247], and integrate seen and heard speech sounds [Rosenblum, L. D., Schmuckler, M. A., & Johnson, J. A. (1997). The McGurk effect in infants. Perception & Psychophysics, 59, 347–357; Burnham, D., & Dodd, B. (2004). Auditory-visual speech integration by prelinguistic infants: Perception of an emergent consonant in the McGurk effect. Developmental Psychobiology, 45, 204–220]. However, the role of visual speech in language development remains unknown. Our aim was to determine whether seen articulations enhance phoneme discrimination, thereby playing a role in phonetic category learning. We exposed 6-month-old infants to speech sounds from a restricted range of a continuum between /ba/ and /da/, following a unimodal frequency distribution. Synchronously with these speech sounds, one group of infants (the two-category group) saw a visual articulation of a canonical /ba/ or /da/, with the two alternative visual articulations, /ba/ and /da/, being presented according to whether the auditory token was on the /ba/ or /da/ side of the midpoint of the continuum. Infants in a second (one-category) group were presented with the same unimodal distribution of speech sounds, but every token for any particular infant was always paired with the same syllable, either a visual /ba/ or a visual /da/. A stimulus-alternation preference procedure following the exposure revealed that infants in the former, and not in the latter, group discriminated the /ba/–/da/ contrast. These results not only show that visual information about speech articulation enhances phoneme discrimination, but also that it may contribute to the learning of phoneme boundaries in infancy.     

Attenuated brain responses to speech sounds in moderate preterm infants at term age

Recent findings have revealed that very preterm neonates already show the typical brain responses to place of articulation changes in stop consonants, but data on their sensitivity to other types of phonetic changes remain scarce. Here, we examined the impact of 7–8 weeks of extra‐uterine life on the automatic processing of syllables in 20 healthy moderate preterm infants (mean gestational age at birth 33 weeks) matched in maturational age with 20 full‐term neonates, thus differing in their previous auditory experience. This design allows elucidating the contribution of extra‐uterine auditory experience in the immature brain on the encoding of linguistically relevant speech features. Specifically, we collected brain responses to natural CV syllables differing in three dimensions using a multi‐feature mismatch paradigm, with the syllable/ba/ as the standard and three deviants: a pitch change, a vowel change to/bo/ and a consonant voice‐onset time (VOT) change to/pa/. No significant between‐group differences were found for pitch and consonant VOT deviants. However, moderate preterm infants showed attenuated responses to vowel deviants compared to full terms. These results suggest that moderate preterm infants' limited experience with low‐pass filtered speech prenatally can hinder vowel change detection and that exposure to natural speech after birth does not seem to contribute to improve this capacity. These data are in line with recent evidence suggesting a sequential development of a hierarchical functional architecture of speech processing that is highly sensitive to early auditory experience.     

Fourteen-month-old infants copy an action style accompanied by social-emotional cues

Human adults exaggerate their actions and facial expressions when interacting with infants. These infant-directed modifications highlight certain aspects of action sequences and attract infants’ attention. This study investigated whether social-emotional aspects of infant-directed modifications, such as smiling, eye contact, and onomatopoeic vocalization, influence infants’ copying of another's action, especially action style, during the process of achieving an outcome. In Study 1, 14-month-old infants (n = 22) saw an experimenter demonstrate goal-directed actions in an exaggerated manner. Either the style or the end state of the actions was accompanied by social-emotional cues from the experimenter. Infants copied the style of the action more often when social-emotional cues accompanied the style than when they accompanied the end state. In Study 2, a different group of 14-month-old infants (n = 22) watched the same exaggerated actions as in Study 1, except that either the style or the end state was accompanied by a physical sound instead of social-emotional cues. The infants copied the end state consistently more often than the style. Taken together, these two studies showed that accompanying social-emotional cues provided by a demonstrator, but not accompanying physical sound, increased infants’ copying of action style. These findings suggest that social-emotional cues facilitate efficient social learning through the adult–infant interaction.     

Perception of audiovisual infant directed speech

Infant perception often deals with audiovisual speech input and a first step in processing this input is to perceive both visual and auditory information. The speech directed to infants has special characteristics and may enhance visual aspects of speech. The current study was designed to explore the impact of visual enhancement in infant-directed speech (IDS) on audiovisual mismatch detection in a naturalistic setting. Twenty infants participated in an experiment with a visual fixation task conducted in participants’ homes. Stimuli consisted of IDS and adult-directed speech (ADS) syllables with a plosive and the vowel /a:/, /i:/ or /u:/. These were either audiovisually congruent or incongruent. Infants looked longer at incongruent than congruent syllables and longer at IDS than ADS syllables, indicating that IDS and incongruent stimuli contain cues that can make audiovisual perception challenging and thereby attract infants’ gaze.     

Distinct Facial Characteristics Differentiate Communicative Intent of Infant‐Directed Speech

Adults and infants can differentiate communicative messages using the nonlinguistic acoustic properties of infant‐directed (ID) speech. Although the distinct prosodic properties of ID speech have been explored extensively, it is currently unknown whether the visual properties of the face during ID speech similarly convey communicative intent and thus represent an additional source of social information for infants during early interactions. To examine whether the dynamic facial movement associated with ID speech confers affective information independent of the acoustic signal, adults' differentiation of the visual properties of speakers' communicative messages was examined in two experiments in which the adults rated silent videos of approving and comforting ID and neutral adult‐directed speech. In Experiment 1, adults differentiated the facial speech groups on ratings of the intended recipient and the speaker's message. In Experiment 2, an original coding scale identified facial characteristics of the speakers. Discriminant correspondence analysis revealed two factors differentiating the facial speech groups on various characteristics. Implications for perception of ID facial movements in relation to speakers' communicative intent are discussed for both typically and atypically developing infants. Copyright © 2012 John Wiley & Sons, Ltd.     

To think or not to think,that is the question: Individual differences in suppression and rebound effects in autobiographical memory

Two studies explored the effects of forget instructions on autobiographical memory at immediate test and following delays of either 12–13 months, or 3–4 months. Using the Autobiographical Think/No-Think procedure (cf., Noreen & MacLeod, 2013), 24 never-depressed participants (Study 1) first generated 12 positive and 12 negative autobiographical memories and associated cues. Participants were then asked to recall the memory associated with some of the cues (i.e., ‘think’ condition), or to avoid saying or thinking about the memory associated with other cues (i.e., ‘no-think’ condition). Participants were then asked to recall the memories associated with all the cues at immediate test and following a delay of 12–13 months. Participants were found to be successful at forgetting both positive and negative autobiographical memories following ‘no-think’ instructions at immediate test but this forgetting effect did not persist following a 12–13 month delay. This pattern of remembering and forgetting was replicated in a second study (using 27 never-depressed participants) following a 3–4 month delay. Participants who had been less successful at forgetting ‘no-think’ memories at immediate test, were more likely to show rebound effects for those memories following a delay compared to memories which received neither ‘think’ nor ‘no-think’ instructions. Individual differences in inhibitory control and the efficacy of potential therapeutic interventions of ‘no-think’ instructions are considered.     

Noise on, voicing off: Speech perception deficits in children with specific language impairment

Speech perception of four phonetic categories (voicing, place, manner, and nasality) was investigated in children with specific language impairment (SLI) (n = 20) and age-matched controls (n = 19) in quiet and various noise conditions using an AXB two-alternative forced-choice paradigm. Children with SLI exhibited robust speech perception deficits in silence, stationary noise, and amplitude-modulated noise. Comparable deficits were obtained for fast, intermediate, and slow modulation rates, and this speaks against the various temporal processing accounts of SLI. Children with SLI exhibited normal “masking release” effects (i.e., better performance in fluctuating noise than in stationary noise), again suggesting relatively spared spectral and temporal auditory resolution. In terms of phonetic categories, voicing was more affected than place, manner, or nasality. The specific nature of this voicing deficit is hard to explain with general processing impairments in attention or memory. Finally, speech perception in noise correlated with an oral language component but not with either a memory or IQ component, and it accounted for unique variance beyond IQ and low-level auditory perception. In sum, poor speech perception seems to be one of the primary deficits in children with SLI that might explain poor phonological development, impaired word production, and poor word comprehension.     

My face or yours? Event-related potential correlates of self-face processing

The neural basis of self-recognition is mainly studied using brain-imaging techniques which reveal much about the localization of self-processing in the brain. There are comparatively few studies using EEG which allow us to study the time course of self-recognition. In this study, participants monitored a sequence of images, including 20 distinct images of their own face, a friend’s face and a stranger’s face articulating different speech sounds, while EEG was recorded from 64 scalp electrodes. Differences in the ERP waveforms were observed very early on, with increased N170 and VPP amplitude to self relative to both friend and stranger measured over posterior and fronto-central sites, respectively. This ‘self effect’ was also marked at ∼250 ms where P2/N2 amplitude was significantly reduced for self-faces. By comparison, differences between friend and stranger faces did not emerge until 250 ms and beyond, where a more conventional ‘familiarity effect’ was observed. The data also point to a ‘less lateralized’ representation of self over posterior sites. These findings are consistent with both behavioral and fMRI studies which suggest that self-face processing is ‘special’ and are discussed with reference to EEG studies of face processing.     

Audiovisual perception of noise vocoded speech in dyslexic and non-dyslexic adults: The role of low-frequency visual modulations

Visual speech inputs can enhance auditory speech information, particularly in noisy or degraded conditions. The natural statistics of audiovisual speech highlight the temporal correspondence between visual and auditory prosody, with lip, jaw, cheek and head movements conveying information about the speech envelope. Low-frequency spatial and temporal modulations in the 2–7 Hz range are of particular importance. Dyslexic individuals have specific problems in perceiving speech envelope cues. In the current study, we used an audiovisual noise-vocoded speech task to investigate the contribution of low-frequency visual information to intelligibility of 4-channel and 16-channel noise vocoded speech in participants with and without dyslexia. For the 4-channel speech, noise vocoding preserves amplitude information that is entirely congruent with dynamic visual information. All participants were significantly more accurate with 4-channel speech when visual information was present, even when this information was purely spatio-temporal (pixelated stimuli changing in luminance). Possible underlying mechanisms are discussed.     

Language experience influences audiovisual speech integration in unimodal and bimodal bilingual infants

Infants as young as 2 months can integrate audio and visual aspects of speech articulation. A shift of attention from the eyes towards the mouth of talking faces occurs around 6 months of age in monolingual infants. However, it is unknown whether this pattern of attention during audiovisual speech processing is influenced by speech and language experience in infancy. The present study investigated this question by analysing audiovisual speech processing in three groups of 4‐ to 8‐month‐old infants who differed in their language experience: monolinguals, unimodal bilinguals (infants exposed to two or more spoken languages) and bimodal bilinguals (hearing infants with Deaf mothers). Eye‐tracking was used to study patterns of face scanning while infants were viewing faces articulating syllables with congruent, incongruent and silent auditory tracks. Monolinguals and unimodal bilinguals increased their attention to the mouth of talking faces between 4 and 8 months, while bimodal bilinguals did not show any age difference in their scanning patterns. Moreover, older (6.6 to 8 months), but not younger, monolinguals (4 to 6.5 months) showed increased visual attention to the mouth of faces articulating audiovisually incongruent rather than congruent faces, indicating surprise or novelty. In contrast, no audiovisual congruency effect was found in unimodal or bimodal bilinguals. Results suggest that speech and language experience influences audiovisual integration in infancy. Specifically, reduced or more variable experience of audiovisual speech from the primary caregiver may lead to less sensitivity to the integration of audio and visual cues of speech articulation.     

Preliminary evidence for selective cortical responses to music in one-month-old infants

Prior studies have observed selective neural responses in the adult human auditory cortex to music and speech that cannot be explained by the differing lower-level acoustic properties of these stimuli. Does infant cortex exhibit similarly selective responses to music and speech shortly after birth? To answer this question, we attempted to collect functional magnetic resonance imaging (fMRI) data from 45 sleeping infants (2.0- to 11.9-weeks-old) while they listened to monophonic instrumental lullabies and infant-directed speech produced by a mother. To match acoustic variation between music and speech sounds we (1) recorded music from instruments that had a similar spectral range as female infant-directed speech, (2) used a novel excitation-matching algorithm to match the cochleagrams of music and speech stimuli, and (3) synthesized “model-matched” stimuli that were matched in spectrotemporal modulation statistics to (yet perceptually distinct from) music or speech. Of the 36 infants we collected usable data from, 19 had significant activations to sounds overall compared to scanner noise. From these infants, we observed a set of voxels in non-primary auditory cortex (NPAC) but not in Heschl's Gyrus that responded significantly more to music than to each of the other three stimulus types (but not significantly more strongly than to the background scanner noise). In contrast, our planned analyses did not reveal voxels in NPAC that responded more to speech than to model-matched speech, although other unplanned analyses did. These preliminary findings suggest that music selectivity arises within the first month of life. A video abstract of this article can be viewed at https://youtu.be/c8IGFvzxudk .

Research Highlights

• Responses to music, speech, and control sounds matched for the spectrotemporal modulation-statistics of each sound were measured from 2- to 11-week-old sleeping infants using fMRI.
• Auditory cortex was significantly activated by these stimuli in 19 out of 36 sleeping infants.
• Selective responses to music compared to the three other stimulus classes were found in non-primary auditory cortex but not in nearby Heschl's Gyrus.
• Selective responses to speech were not observed in planned analyses but were observed in unplanned, exploratory analyses.