Levels of representation in the electrophysiology of speech perception

Mapping from acoustic signals to lexical representations is a complex process mediated by a number of different levels of representation. This paper reviews properties of the phonetic and phonological levels, and hypotheses about how category structure is represented at each of these levels, and evaluates these hypotheses in light of relevant electrophysiological studies of phonetics and phonology. The paper examines evidence for two alternative views of how infant phonetic representations develop into adult representations, a structure-changing view and a structure-adding view, and suggests that each may be better suited to different kinds of phonetic categories. Electrophysiological results are beginning to provide information about phonological representations, but less is known about how the more abstract representations at this level could be coded in the brain.     

Different kinds of bilinguals: different kinds of brains: the neural organisation of two languages in one brain

The aim of this study was to determine whether the type of bilingualism affects neural organisation. We performed identification experiments and mismatch negativity (MMN) registrations in Finnish and Swedish language settings to see, whether behavioural identification and neurophysiological discrimination of vowels depend on the linguistic context, and whether there is a difference between two kinds of bilinguals. The stimuli were two vowels, which differentiate meaning in Finnish, but not in Swedish. The results indicate that Balanced Bilinguals are inconsistent in identification performance, and they have a longer MMN latency. Moreover, their MMN amplitude is context-independent, while Dominant Bilinguals show a larger MMN in the Finnish context. These results indicate that Dominant Bilinguals inhibit the preattentive discrimination of native contrast in a context where the distinction is non-phonemic, but this is not possible for Balanced Bilinguals. This implies that Dominant Bilinguals have separate systems, while Balanced Bilinguals have one inseparable system.     

Neurophysiological evidence for categorical perception of color

The aim of this investigation was to examine the time course and the relative contributions of perceptual and post-perceptual processes to categorical perception (CP) of color. A visual oddball task was used with standard and deviant stimuli from same (within-category) or different (between-category) categories, with chromatic separations for within- and between-category stimuli equated in Munsell Hue. CP was found on a behavioral version of the task, with faster RTs and greater accuracy for between- compared to within-category stimuli. On a neurophysiological version of the task, event-related potentials (ERPs) showed earlier latencies for P1 and N1 components at posterior locations to between- relative to within-category deviants, providing novel evidence for early perceptual processes on color CP. Enhanced P2 and P3 waves were also found for between- compared to within-category stimuli, indicating a role for later post-perceptual processes.     

Regional differences in the listener's phonemic inventory affect semantic processing: A mismatch negativity (MMN) study

Using the mismatch negativity (MMN) response, we examined how Standard French and Southern French speakers access the meaning of words ending in /e/ or /ε/ vowels which are contrastive in Standard French but not in Southern French. In Standard French speakers, there was a significant difference in the amplitude of the brain response after the deviant-minus-standard subtraction between the frontocentral (FC) and right lateral (RL) recording sites for the final-/ε/ word but not the final-/e/ word. In contrast, the difference in the amplitude of the brain response between the FC and RL recording sites did not significantly vary as a function of the word’s final vowel in Southern French speakers. Our findings provide evidence that access to lexical meaning in spoken word recognition depends on the speaker’s native regional accent.     

Assessing the double phonemic representation in bilingual speakers of Spanish and English: an electrophysiological study

Event Related Potentials (ERPs) were recorded from Spanish-English bilinguals (N = 10) to test pre-attentive speech discrimination in two language contexts. ERPs were recorded while participants silently read magazines in English or Spanish. Two speech contrast conditions were recorded in each language context. In the phonemic in English condition, the speech sounds represented two different phonemic categories in English, but represented the same phonemic category in Spanish. In the phonemic in Spanish condition, the speech sounds represented two different phonemic categories in Spanish, but represented the same phonemic categories in English. Results showed pre-attentive discrimination when the acoustics/phonetics of the speech sounds match the language context (e.g., phonemic in English condition during the English language context). The results suggest that language contexts can affect pre-attentive auditory change detection. Specifically, bilinguals’ mental processing of stop consonants relies on contextual linguistic information.     

Phonological abstraction in processing lexical-tone variation: evidence from a learning paradigm

There is a growing consensus that the mental lexicon contains both abstract and word-specific acoustic information. To investigate their relative importance for word recognition, we tested to what extent perceptual learning is word specific or generalizable to other words. In an exposure phase, participants were divided into two groups; each group was semantically biased to interpret an ambiguous Mandarin tone contour as either tone1 or tone2. In a subsequent test phase, the perception of ambiguous contours was dependent on the exposure phase: Participants who heard ambiguous contours as tone1 during exposure were more likely to perceive ambiguous contours as tone1 than participants who heard ambiguous contours as tone2 during exposure. This learning effect was only slightly larger for previously encountered than for not previously encountered words. The results speak for an architecture with prelexical analysis of phonological categories to achieve both lexical access and episodic storage of exemplars.     

The link between speech perception and production is phonological and abstract: evidence from the shadowing task

This study reports a shadowing experiment, in which one has to repeat a speech stimulus as fast as possible. We tested claims about a direct link between perception and production based on speech gestures, and obtained two types of counterevidence. First, shadowing is not slowed down by a gestural mismatch between stimulus and response. Second, phonetic detail is more likely to be imitated in a shadowing task if it is phonologically relevant. This is consistent with the idea that speech perception and speech production are only loosely coupled, on an abstract phonological level.     

The role of variation in the perception of accented speech

Phonetic variation has been considered a barrier that listeners must overcome in speech perception, but has been proved beneficial in category learning. In this paper, I show that listeners use within-speaker variation to accommodate gross categorical variation. Within the perceptual learning paradigm, listeners are exposed to p-initial words in English produced by a native speaker of French. Critically, listeners are trained on these words with either invariant or highly-variable VOTs. While a gross boundary shift is made for participants exposed to the variable VOTs, no such shift is observed after exposure to the invariant stimuli. These data suggest that increasing variation improves the mapping of perceptually mismatched stimuli.     

Us versus them: Understanding the process of race perception with event-related brain potentials

Race is powerful social signal, imbued with a great deal of meaning and capable of affecting a wide range of behaviours and judgements. This review focuses on the use of event-related brain potentials (ERPs) to better understand the process and implications of race perception. Research to date repeatedly demonstrates the speed and automaticity with which racial category membership is encoded. The quick and relatively obligatory encoding of racial ingroup and outgroup distinctions in turn affects a variety of subsequent racially biased behaviours. Such findings not only speak to the close link between social categorization and behaviour, but also have been used to understand how the effects of race perception can be moderated.     

Mental rotation of mirrored letters: evidence from event-related brain potentials

Event-related brain potentials (ERPs) were recorded while participants (n=13) were presented with mirrored and normal letters at different orientations and were asked to make mirror-normal letter discriminations. As it has been suggested that a mental rotation out of the plane might be necessary to decide on mirrored letters, we wanted to determine whether this rotation occurs after the plane rotation in mirror rotated letters. The results showed that mirrored letters in the upright position elicited a negative-going waveform over the right hemisphere in the 400-500 ms window. A similar negativity was also present in mirrored letters at 30 degrees, 60 degrees, and 90 degrees, but in these cases it was delayed. Moreover, the well-known orientation effect on the amplitude of the rotation-related negativity was also found, although it was more evident for normal than for mirrored letters. These results indicate that the processing of mirrored letters differs from that of normal letters, and suggest that a rotation out of the plane after the plane rotation may be involved in the processing of mirror rotated letters.     

Testing the concurrent and predictive relations among articulation accuracy, speech perception, and phoneme awareness

The relations among articulation accuracy, speech perception, and phoneme awareness were examined in a sample of 97 typically developing children ages 48 to 66 months. Of these 97 children, 46 were assessed twice at ages 4 and 5 years. Children completed two tasks for each of the three skills, assessing these abilities for the target phoneme /r/ and the control phoneme /m/ in the word-initial position. Concurrent analyses revealed that phoneme-specific relations existed among articulation, awareness, and perception. Articulation accuracy of /r/ predicted speech perception and phoneme awareness for /r/ after controlling for age, vocabulary, letter-word knowledge, and speech perception or phoneme awareness for the control phoneme /m/. The longitudinal analyses confirmed the pattern of relations. The findings are consistent with a model whereby children's articulation accuracy affects preexisting differences in phonological representations and, consequently, affects how children perceive, discriminate, and manipulate speech sounds.     

Brief pitch-priming facilitates infants’ discrimination of pitch-evoking noise: Evidence from event-related potentials

Pitch is derived by the auditory system through complex spectrotemporal processing. Pitch extraction is thought to depend on both spectral cues arising from lower harmonics that are resolved by cochlear filters in the inner ear, and on temporal cues arising from the pattern of action potentials contained in the cochlear output. Adults are capable of extracting pitch in the absence of robust spectral cues, taking advantage of the temporal cues that remain. However, recent behavioral evidence suggests that infants have difficulty discriminating between stimuli with different pitches when resolvable spectral cues are absent. In the current experiments, we used the mismatch negativity (MMN) component of the event related potential derived from electroencephalographic (EEG) recordings to examine a cortical representation of pitch discrimination for iterated rippled noise (IRN) stimuli in 4- and 8-month-old infants. IRN stimuli are pitch-evoking sounds generated by repeatedly adding a segment of white noise to itself at a constant delay. We created IRN stimuli (delays of 5 and 6 ms creating pitch percepts of 200 and 167 Hz) and high-pass filtered them to remove all resolvable spectral pitch cues. In experiment 1, we did not find EEG evidence that infants could detect the change in the pitch of these IRN stimuli. However, in Experiment 2, after a brief period of pitch-priming during which we added a sine wave component to the IRN stimulus at its perceived pitch, infants did show significant MMN in response to pitch changes in the IRN stimuli with sine waves removed. This suggests that (1) infants can use temporal cues to process pitch, although such processing is not mature and (2) that a short amount of pitch-priming experience can alter pitch representations in auditory cortex during infancy.     

The chronometry of mental ability: An event-related potential analysis of an auditory oddball discrimination task

The relation between intelligence and speed of auditory discrimination was investigated during an auditory oddball task with backward masking. In target discrimination conditions that varied in the interval between the target and the masking stimuli and in the tonal frequency of the target and masking stimuli, higher ability participants (HA) displayed more accurate discriminations, faster response time, larger P300 amplitude, and shorter P300 and mismatch negativity (MMN) latency than lower ability participants (LA). Task difficulty effects demonstrated with variation in mask type indicate that the mask does not interfere with the detection of the deviant target stimulus, but rather that the target and mask are integrated as a single compound stimulus. The temporal effects suggest that the speed of accessing short-term memory is faster for HA than LA and, on the basis of the MMN latency, the effect is accomplished automatically, without focused attention. Moreover, the pattern of results obtained with these data support the view that the accuracy effects are determined by processing speed rather than discrimination ability. Comparator models that accommodate these effects are discussed.     

The relative importance of spatial versus temporal structure in the perception of biological motion: an event-related potential study

We investigated how the spatiotemporal structure of animations of biological motion (BM) affects brain activity. We measured event-related potentials (ERPs) during the perception of BM under four conditions: normal spatial and temporal structure; scrambled spatial and normal temporal structure; normal spatial and scrambled temporal structure; and scrambled spatial and temporal structure. As in a previous study, we identified two negative components at both occipitotemporal regions: N210 reflected general motion processing while N280 reflected the processing of BM. We analyzed the averaged ERPs in the 200-300 ms response time window and found that spatial structure had a substantial effect on the magnitude of the averaged response amplitude in both hemispheres. This finding suggests that spatial structure of point-lights elicits a stronger response in the occipitotemporal region than temporal structure for the BM perception.     

The organization of words and environmental sounds in the second year: Behavioral and electrophysiological evidence

The majority of research examining early auditory‐semantic processing and organization is based on studies of meaningful relations between words and referents. However, a thorough investigation into the fundamental relation between acoustic signals and meaning requires an understanding of how meaning is associated with both lexical and non‐lexical sounds. Indeed, it is unknown how meaningful auditory information that is not lexical (e.g., environmental sounds) is processed and organized in the young brain. To capture the structure of semantic organization for words and environmental sounds, we record event‐related potentials as 20‐month‐olds view images of common nouns (e.g., dog) while hearing words or environmental sounds that match the picture (e.g., “dog” or barking), that are within‐category violations (e.g., “cat” or meowing), or that are between‐category violations (e.g., “pen” or scribbling). Results show both words and environmental sounds exhibit larger negative amplitudes to between‐category violations relative to matches. Unlike words, which show a greater negative response early and consistently to within‐category violations, such an effect for environmental sounds occurs late in semantic processing. Thus, as in adults, the young brain represents semantic relations between words and between environmental sounds, though it more readily differentiates semantically similar words compared to environmental sounds.     

Infant perception of audio-visual speech synchrony in familiar and unfamiliar fluent speech

We investigated the effects of linguistic experience and language familiarity on the perception of audio-visual (A-V) synchrony in fluent speech. In Experiment 1, we tested a group of monolingual Spanish- and Catalan-learning 8-month-old infants to a video clip of a person speaking Spanish. Following habituation to the audiovisually synchronous video, infants saw and heard desynchronized clips of the same video where the audio stream now preceded the video stream by 366, 500, or 666 ms. In Experiment 2, monolingual Catalan and Spanish infants were tested with a video clip of a person speaking English. Results indicated that in both experiments, infants detected a 666 and a 500 ms asynchrony. That is, their responsiveness to A-V synchrony was the same regardless of their specific linguistic experience or familiarity with the tested language. Compared to previous results from infant studies with isolated audiovisual syllables, these results show that infants are more sensitive to A-V temporal relations inherent in fluent speech. Furthermore, the absence of a language familiarity effect on the detection of A-V speech asynchrony at eight months of age is consistent with the broad perceptual tuning usually observed in infant response to linguistic input at this age.     

The electrophysiological correlates of integrated face and body-part perception

Previous studies have suggested that the human visual system processes faces and bodies holistically—that is, the different body parts are integrated into a unified representation. However, the time course of this integrative process is less known. In the present study, we investigated this issue by recording event-related potentials evoked by a face and two hands presented simultaneously and in different configurations. When the hands were rotated to obtain a biologically implausible configuration, a reduction of the P2 amplitude was observed relative to the condition in which the face and hands were retained in their veridical configuration and were supplemented with visual cues to highlight further the overall body posture. Our results show that the P2 component is sensitive to manipulations affecting the configuration of face and hand stimuli and suggest that the P2 reflects the operation of perceptual mechanisms responsible for the integrated processing of visually presented body parts.     

The role of Broca's area in speech perception: evidence from aphasia revisited

Motor theories of speech perception have been re-vitalized as a consequence of the discovery of mirror neurons. Some authors have even promoted a strong version of the motor theory, arguing that the motor speech system is critical for perception. Part of the evidence that is cited in favor of this claim is the observation from the early 1980s that individuals with Broca’s aphasia, and therefore inferred damage to Broca’s area, can have deficits in speech sound discrimination. Here we re-examine this issue in 24 patients with radiologically confirmed lesions to Broca’s area and various degrees of associated non-fluent speech production. Patients performed two same-different discrimination tasks involving pairs of CV syllables, one in which both CVs were presented auditorily, and the other in which one syllable was auditorily presented and the other visually presented as an orthographic form; word comprehension was also assessed using word-to-picture matching tasks in both auditory and visual forms. Discrimination performance on the all-auditory task was four standard deviations above chance, as measured using d′, and was unrelated to the degree of non-fluency in the patients’ speech production. Performance on the auditory–visual task, however, was worse than, and not correlated with, the all-auditory task. The auditory–visual task was related to the degree of speech non-fluency. Word comprehension was at ceiling for the auditory version (97% accuracy) and near ceiling for the orthographic version (90% accuracy). We conclude that the motor speech system is not necessary for speech perception as measured both by discrimination and comprehension paradigms, but may play a role in orthographic decoding or in auditory–visual matching of phonological forms.     

EEG correlates of cognitive time scales in the Necker-Zeno model for bistable perception

The Necker-Zeno model of bistable perception provides a formal relation between the average duration of meta-stable percepts (dwell times T) of ambiguous figures and two other basic time scales (t₀, ΔT) underlying cognitive processing. The model predicts that dwell times T covary with t₀, ΔT or both. We tested this prediction by exploiting that observers, in particular experienced meditators, can volitionally control dwell times T.Meditators and non-meditators observed bistable Necker cubes either passively or tried to hold their current percept. The latencies of a centro-parietal event-related potential (CPP) were recorded as a physiological correlate of t₀.Dwell times T and the CPP latencies, correlated with t₀, differed between conditions and observer groups, while ΔT remained constant in the range predicted by the model. The covariation of CPP latencies and dwell times, as well as their quadratic functional dependence extends previous psychophysical confirmation of the Necker-Zeno model to psychophysiological measures.     

Structural constraints in the perception of English stop-sonorant clusters

Native-language phonemes combined in a non-native way can be misperceived so as to conform to native phonotactics, e.g. English listeners are biased to hear syllable-initial [tr] rather than the illegal [tl] (Perception and Psychophysics 34 (1983) 338; Perception and Psychophysics 60 (1998) 941). What sort of linguistic knowledge causes phonotactic perceptual bias? Two classes of models were compared: unit models, which attribute bias to the listener's differing experience of each cluster (such as their different frequencies), and structure models, which use abstract phonological generalizations (such as a ban on [coronal][coronal] sequences). Listeners (N=16 in each experiment) judged synthetic 6 x 6 arrays of stop-sonorant clusters in which both consonants were ambiguous. The effect of the stop judgment on the log odds ratio of the sonorant judgment was assessed separately for each stimulus token to provide a stimulus-independent measure of bias. Experiment 1 compared perceptual bias against the onsets [bw] and [dl], which violate different structural constraints but are both of zero frequency. Experiment 2 compared bias against [dl] in CCV and VCCV contexts, to investigate the interaction of syllabification with segmentism and to rule out a compensation-for-coarticulation account of Experiment 1. Results of both experiments favor the structure models (supported by NSF).