An Outline of a Theory of Affordances

The principle of auditory enhancement is a valuable extension of the theory of adaptive dispersion. However, neither the principle nor the theory has any bearing on what we perceive in speech. All perceptual objects, including those of speech, are abstract, amodal structures made available to cognition through one or more sensory modalities. A focus on the modalities or media of information transfer in speech perception diverts attention from two central and related aspects of speech function: perceptuomotor functional equivalence and imitation. Arguments and evidence from studies of lipreading, short-term memory, and infant behavior are put forward to support the postulation of an output unit, the phonetic gesture, at a level in the communication chain corresponding on the input side to the acoustic cue. The object of speech perception is then taken to be the amodal phonetic segment, a cohesive set of direct mappings between sound and gesture.     

Self-Induced Motion Sickness and Body Movement During Passive Restraint

From both direct-realist and motor-theoretic perspectives, the objects of speech perception are construed as articulatory events. We offer the alternative view that it is the auditory properties of phonetic events to which listeners are primarily sensitive. Evidence for this alternative view comes from a variety of sources, including studies of phonetic and phonological universals, comparisons of the perception of speech and nonspeech signals, cross-language perceptual experi- ments, and studies of phonetic categorization by nonhuman animals.     

Auditory and phonetic levels of processing in speech perception: neurophysiological and information-processing analyses.

Two new experimental operations were used to distinguish between auditory and phonetic levels of processing in speech perception: the first based on reaction time data in speeded classification tasks with synthetic speech stimuli, and the second based on average evoked potentials recorded concurrently in the same tasks. Each of four experiments compared the processing of two different dimensions of the same synthetic consonant-vowel syllables. When a phonetic dimensions was compared to an auditory dimension, different patterns of results were obtained in both the reaction time and evoked potential data. No such differences were obtained for isolated acoustic components of the phonetic dimension or for two purely auditory dimensions. Together with other recent evidence, the present results constitute additional converging operations on the distinction between auditory and phonetic processes in speech perception and on the idea that phonetic processing involves mechanisms that are lateralized in one cerebral hemisphere.     

Discovering phonetic coherence in acoustic patterns

Despite spectral and temporal discontinuities in the speech signal, listeners normally report coherent phonetic patterns corresponding to the phonemes of a language that they know. What is the basis for the internal coherence of phonetic segments? According to one account, listeners achieve coherence by extracting and integrating discrete cues; according to another, coherence arises automatically from general principles of auditory form perception; according to a third, listeners perceive speech patterns as coherent because they are the acoustic consequences of coordinated articulatory gestures in a familiar language. We tested these accounts in three experiments by training listeners to hear a continuum of three-tone, modulated sine wave patterns, modeled after a minimal pair contrast between three-formant synthetic speech syllables, either as distorted speech signals carrying a phonetic contrast (speech listeners) or as distorted musical chords carrying a nonspeech auditory contrast (music listeners). The music listeners could neither integrate the sine wave patterns nor perceive their auditory coherence to arrive at consistent, categorical percepts, whereas the speech listeners judged the patterns as speech almost as reliably as the synthetic syllables on which they were modeled. The outcome is consistent with the hypothesis that listeners perceive the phonetic coherence of a speech signal by recognizing acoustic patterns that reflect the coordinated articulatory gestures from which they arose.     

Perceptual-motor processing of phonetic features in speech

Some reaction time experiments are reported on the relation between the perception and production of phonetic features in speech. Subjects had to produce spoken consonant-vowel syllables rapidly in response to other consonant-vowel stimulus syllables. The stimulus syllables were presented auditorily in one condition and visually in another. Reaction time was measured as a function of the phonetic features shared by the consonants of the stimulus and response syllables. Responses to auditory stimulus syllables were faster when the response syllables started with consonants that had the same voicing feature as those of the stimulus syllables. A shared place-of-articulation feature did not affect the speed of responses to auditory stimulus syllables, even though the place feature was highly salient. For visual stimulus syllables, performance was independent of whether the consonants of the response syllables had the same voicing, same place of articulation, or no shared features. This pattern of results occurred in cases where the syllables contained stop consonants and where they contained fricatives. It held for natural auditory stimuli as well as artificially synthesized ones. The overall data reveal a close relation between the perception and production of voicing features in speech. It does not appear that such a relation exists between perceiving and producing places of articulation. The experiments are relevant to the motor theory of speech perception and to other models of perceptual-motor interactions.     

婴儿语音感知发展及其机制

语音感知的发展状况对个体的语言发展有着深远影响。生命的第一年中, 在语言经验的作用下, 婴儿的语音感知从最初的普遍性感知逐渐发展为对母语的特异性感知。研究者们提出统计学习机制对这一过程加以解释, 即婴儿对语言环境中语音的频次分布十分敏感, 可以通过对频次分布的计算, 从语音的连续体中区分出在母语中起区别意义作用的各个语音范畴。同时, 功能性重组机制和一些社会性线索也会对婴儿语音感知的发展产生重要影响。     

Real Objects of Speech Perception: A Commentary on Diehl and Kluender

I agree with Diehl and Kluender (1989) that perceptual constraints guide the development of sound inventories and of phonological processes in languages. I disagree that these constraints are primary in comparison with other influences on sound inventories, such as articulatory ones. More important, I disagree that any of the evidence that Diehl and Kluender cite, indicates that objects of perception are auditory or acoustic rather than phonetic gestural. None of the evidence is persuasive; all of it is consistent with a view that perceptual objects are gestural. Viewed in a larger context-of a universal theory of perception-a theory that perceptual objects are gestural, whereas acoustic structure serves as information for gestures, is strongly promoted.     

Mandarin speech perception by ear and eye follows a universal principle

In this study, the nature of speech perception of native Mandarin Chinese was compared with that of American English speakers, using synthetic visual and auditory continua (from /ba/ to /da/) in an expanded factorial design. In Experiment 1, speakers identified synthetic unimodal and bimodal speech syllables as either /ba/ or /da/. In Experiment 2, Mandarin speakers were given nine possible response alternatives. Syllable identification was influenced by both visual and auditory sources of information for both Mandarin and English speakers. Performance was better described by the fuzzy logical model of perception than by an auditory dominance model or a weighted-averaging model. Overall, the results are consistent with the idea that although there may be differences in information (which reflect differences in phonemic repertoires, phonetic realizations of the syllables, and the phonotactic constraints of languages), the underlying nature of audiovisual speech processing is similar across languages.     

A Theory of Perceptual Objects

Objects are central in visual, auditory, and tactual perception. But what counts as a perceptual object? I address this question via a structural unity schema, which specifies how a collection of parts must be arranged to compose an object for perception. On the theory I propose, perceptual objects are composed of parts that participate in causally sustained regularities. I argue that this theory falls out of a compelling account of the function of object perception, and illustrate its applications to multisensory perception. I also argue that the account avoids problems faced by standard views of visual and auditory objects.     

An investigation of categorical speech discrimination by rhesus monkeys

The categorical discrimination of synthetic human speech sounds by rhesus macaques was examined using the cardiac component of the orienting response. A within-category change which consisted of stimuli differing acoustically in the onset of F2 and F3 transitions, but which are identified by humans as belonging to thesame phonetic category, were responded to differently from a no-change control condition. Stimuli which differed by the same amount in the onset of F2 and F3 transitions, but which human observers identify as belonging toseparate phonetic categories, were differentiated to an even greater degree than the within-category stimuli. The results provide ambiguous data for an articulatory model of human speech perception and are interpreted instead in terms of a feature-detector model of auditory perception.     

Speech perception without hearing

In this study of visual phonetic speech perception without accompanying auditory speech stimuli, adults with normal hearing (NH; n = 96) and with severely to profoundly impaired hearing (IH; n = 72) identified consonant-vowel (CV) nonsense syllables and words in isolation and in sentences. The measures of phonetic perception were the proportion of phonemes correct and the proportion of transmitted feature information for CVs, the proportion of phonemes correct for words, and the proportion of phonemes correct and the amount of phoneme substitution entropy for sentences. The results demonstrated greater sensitivity to phonetic information in the IH group. Transmitted feature information was related to isolated word scores for the IH group, but not for the NH group. Phoneme errors in sentences were more systematic in the IH than in the NH group. Individual differences in phonetic perception for CVs were more highly associated with word and sentence performance for the IH than for the NH group. The results suggest that the necessity to perceive speech without hearing can be associated with enhanced visual phonetic perception in some individuals.     

Auditory and phonetic memory codes in the discrimination of consonants and vowels

Recognition memory for consonants and vowels selected from within and between phonetic categories was examined in a delayed comparison discrimination task. Accuracy of discrimination for synthetic vowels selected from both within and between categories was inversely related to the magnitude of the comparison interval. In contrast, discrimination of synthetic stop consonants remained relatively stable both within and between categories. The results indicate that differences in discrimination between consonants and vowels are primarily due to the differential availability of auditory short-term memory for the acoustic cues distinguishing these two classes of speech sounds. The findings provide evidence for distinct auditory and phonetic memory codes in speech perception.     

Learning words’ sounds before learning how words sound: 9-Month-olds use distinct objects as cues to categorize speech information

One of the central themes in the study of language acquisition is the gap between the linguistic knowledge that learners demonstrate, and the apparent inadequacy of linguistic input to support induction of this knowledge. One of the first linguistic abilities in the course of development to exemplify this problem is in speech perception: specifically, learning the sound system of one’s native language. Native-language sound systems are defined by meaningful contrasts among words in a language, yet infants learn these sound patterns before any significant numbers of words are acquired. Previous approaches to this learning problem have suggested that infants can learn phonetic categories from statistical analysis of auditory input, without regard to word referents. Experimental evidence presented here suggests instead that young infants can use visual cues present in word-labeling situations to categorize phonetic information. In Experiment 1, 9-month-old English-learning infants failed to discriminate two non-native phonetic categories, establishing baseline performance in a perceptual discrimination task. In Experiment 2, these infants succeeded at discrimination after watching contrasting visual cues (i.e., videos of two novel objects) paired consistently with the two non-native phonetic categories. In Experiment 3, these infants failed at discrimination after watching the same visual cues, but paired inconsistently with the two phonetic categories. At an age before which memory of word labels is demonstrated in the laboratory, 9-month-old infants use contrastive pairings between objects and sounds to influence their phonetic sensitivity. Phonetic learning may have a more functional basis than previous statistical learning mechanisms assume: infants may use cross-modal associations inherent in social contexts to learn native-language phonetic categories.     

Processing steps in perception of speech sounds]

The aim of this study is to investigate whether speech sounds--as is stated by the widely accepted theory of categorical perception of speech--can be perceived only as instances of phonetic categories, or whether physical differences between speech sounds lead to perceptual differences regardless of their phonetic categorization. Subjects listened to pairs of synthetically generated speech sounds that correspond to realizations of the syllables "ba" and "pa" in natural German, and they were instructed to decide as fast as possible whether they perceived them as belonging to the same or to different phonetic categories. For 'same'-responses reaction times become longer when the physical distance between the speech sounds is increased; for 'different'-responses reaction times become shorter with growing physical distance between the stimuli. The results show that subjects can judge speech sounds on the basis of perceptual continua, which is inconsistent with the theory of categorical perception. A mathematical model is presented that attempts to explain the results by postulating two interacting stages of processing, a psychoacoustical and a phonetic one. The model is not entirely confirmed by the data, but it seems to deserve further consideration.     

Identification of consonants and vowels presented to left and right ears

The results of earlier studies by several authors suggest that speech and nonspeech auditory patterns are processed primarily in different places in the brain and perhaps by different modes. The question arises in studies of speech perception whether all phonetic elements or all features of phonetic elements are processed in the same way. The technique of dichotic presentation was used to examine this question.

The present study compared identifications of dichotically presented pairs of synthetic CV syllables and pairs of steady-state vowels. The results show a significant right-ear advantage for CV syllables but not for steady-state vowels. Evidence for analysis by feature in the perception of consonants is discussed.     

Unattended exposure to components of speech sounds yields same benefits as explicit auditory training

Learning a second language as an adult is particularly effortful when new phonetic representations must be formed. Therefore the processes that allow learning of speech sounds are of great theoretical and practical interest. Here we examined whether perception of single formant transitions, that is, sound components critical in speech perception, can be enhanced through an implicit task-irrelevant learning procedure that has been shown to produce visual perceptual learning. The single-formant sounds were paired at subthreshold levels with the attended targets in an auditory identification task. Results showed that task-irrelevant learning occurred for the unattended stimuli. Surprisingly, the magnitude of this learning effect was similar to that following explicit training on auditory formant transition detection using discriminable stimuli in an adaptive procedure, whereas explicit training on the subthreshold stimuli produced no learning. These results suggest that in adults learning of speech parts can occur at least partially through implicit mechanisms.     

A Connectionist Model of Phonological Representation in Speech Perception

A number of recent studies have examined the effects of phonological variation on the perception of speech. These studies show that both the lexical representations of words and the mechanisms of lexical access are organized so that natural, systematic variation is tolerated by the perceptual system, while a general intolerance of random deviation is maintained. Lexical abstraction distinguishes between phonetic features that form the invariant core of a word and those that are susceptible to variation. Phonological inference relies on the context of surface changes to retrieve the underlying phonological form. In this article we present a model of these processes in speech perception, based on connectionist learning techniques. A simple recurrent network was trained on the mapping from the variant surface form of speech to the underlying form. Once trained, the network exhibited features of both abstraction and inference in its processing of normal speech, and predicted that similar behavior will be found in the perception of nonsense words. This prediction was confirmed in subsequent research (Gaskell & Marslen-Wilson, 1994).     

Perceptual dominance during lipreading

Two experiments were pedormed under visual-only and visual-auditory discrepancy conditions (dubs) to assess observers’ abilities to read speech information on a face. In the first experiment, identification and multiple choice testing were used. In addition, the relation between visual and auditory phonetic information was manipulated and related to perceptual bias. In the second experiment, the “compellingness” of the visual-auditory discrepancy as a single speech event was manipulated. Subjects also rated the confidence they had that their perception of the lipped word was accurate. Results indicated that competing visual information exerted little effect on auditory speech recognition, but visual speech recognition was substantially interfered with when discrepant auditory information was present. The extent of auditory bias was found to be related to the abilities of observers to read speech under nondiscrepancy conditions, the magnitude of the visual-auditory discrepancy, and the compellingheSS of the visual-auditory discrepancy as a single event. Auditory bias during speech was found to be a moderately compelling conscious experience, and not simply a case of confused responding or guessing. Results were discussed in terms of current models of perceptual dominance and related to results from modality discordance during space perception.     

Rise time and formant transition duration in the discrimination of speech sounds: the Ba-Wa distinction in developmental dyslexia

Across languages, children with developmental dyslexia have a specific difficulty with the neural representation of the sound structure (phonological structure) of speech. One likely cause of their difficulties with phonology is a perceptual difficulty in auditory temporal processing (Tallal, 1980). Tallal (1980) proposed that basic auditory processing of brief, rapidly successive acoustic changes is compromised in dyslexia, thereby affecting phonetic discrimination (e.g. discriminating /b/ from /d/) via impaired discrimination of formant transitions (rapid acoustic changes in frequency and intensity). However, an alternative auditory temporal hypothesis is that the basic auditory processing of the slower amplitude modulation cues in speech is compromised (Goswami et al., 2002). Here, we contrast children's perception of a synthetic speech contrast (ba/wa) when it is based on the speed of the rate of change of frequency information (formant transition duration) versus the speed of the rate of change of amplitude modulation (rise time). We show that children with dyslexia have excellent phonetic discrimination based on formant transition duration, but poor phonetic discrimination based on envelope cues. The results explain why phonetic discrimination may be allophonic in developmental dyslexia (Serniclaes et al., 2004), and suggest new avenues for the remediation of developmental dyslexia.