Attention within auditory word perception: insights from the phonemic restoration illusion

Phonemic restoration is a powerful auditory illusion that arises when a phoneme is removed from a word and replaced with noise, resulting in a percept that sounds like the intact word with a spurious bit of noise. It is hypothesized that the configurational properties of the word impair attention to the individual phonemes and thereby induce perceptual restoration of the missing phoneme. If so, this impairment might be unlearned if listeners can process individual phonemes within a word selectively. Subjects received training with the potentially restorable stimuli (972 trials with feedback); in addition, the presence or absence of an attentional cue, contained in a visual prime preceding each trial, was varied between groups of subjects. Cuing the identity and location of the critical phoneme of each test word allowed subjects to attend to the critical phoneme, thereby inhibiting the illusion, but only when the prime also identified the test word itself. When the prime provided only the identity or location of the critical phoneme, or only the identity of the word, subjects performed identically to those subjects for whom the prime contained no information at all about the test word. Furthermore, training did not produce any generalized learning about the types of stimuli used. A limited interactive model of auditory word perception is discussed in which attention operates through the lexical level.     

Testing the speech unit hypothesis with the primed matching task: Phoneme categories are perceptually basic

The basic speech unit (phoneme or syllable) problem was investigatedwith the primed matching task. In primed matching, subjects have to decide whether the elements of stimulus pairs are the same or different. The prime should facilitate matching in as far as its representation is similar to the stimuli to be matched. If stimulus representations generate graded structure, with stimulus instances being more or less prototypical for the category, priming should interact with prototypicality because prototypical instances are more similar to the activated category than are low-prototypical instances. Rosch (1975a, 1975b) showed that, by varying the matching criterion (matching for physical identity or for belonging to the same category), the specific patterns of the priming × prototypicality interaction could differentiate perceptually based from abstract categories. Bytesting this pattern forphoneme and syllable categories, the abstraction level of these categories canbe studied. After finding reliable prototypicality effects for both phoneme and syllable categories (Experiments 1 and 2), primed phoneme matching (Experiments 3 and 4) and primed syllable matching (Experiments 5 and 6) were used under both physical identity instructions and same-category instructions. The results make clear that phoneme categories are represented on the basis of perceptual information, whereas syllable representations are more abstract. The phoneme category can thus be identified as the basic speech unit. Implications for phoneme and syllable representation are discussed.     

An effect of linguistic experience: The discrimination of [r] and [l] by native speakers of Japanese and English

To test the effect of linguistic experience on the perception of a cue that is known to be effective in distinguishing between [r] and [l] in English, 21 Japanese and 39 American adults were tested on discrimination of a set of synthetic speech-like stimuli. The 13 “speech” stimuli in this set varied in the initial stationary frequency of the third formant (F3) and its subsequent transition into the vowel over a range sufficient to produce the perception of [r a] and [l a] for American subjects and to produce [r a] (which is not in phonemic contrast to [l a ]) for Japanese subjects. Discrimination tests of a comparable set of stimuli consisting of the isolated F3 components provided a “nonspeech” control. For Americans, the discrimination of the speech stimuli was nearly categorical, i.e., comparison pairs which were identified as different phonemes were discriminated with high accuracy, while pairs which were identified as the same phoneme were discriminated relatively poorly. In comparison, discrimination of speech stimuli by Japanese subjects was only slightly better than chance for all comparison pairs. Performance on nonspeech stimuli, however, was virtually identical for Japanese and American subjects; both groups showed highly accurate discrimination of all comparison pairs. These results suggest that the effect of linguistic experience is specific to perception in the “speech mode.”     

Effects of intensity on the categorical perception of stop consonants and isolated second formant transitions

Identification and discrimination of two-formant [bae-dae-gae] and [pae-tae-kae] synthetic speech stimuli and discrimination of corresponding isolated second formant transitions (chirps) were performed by six subjects. Stimuli were presented at several intensity levels such that the intensity of the F2 transition was equated between speech and nonspeech stimuli, or the overall intensity of the stimulus was equated. At higher intensity (92 dB), b-d-g and p-t-k identification and between-category discrimination performance declined and bilabial-alveolar phonetic boundaries shifted in location on the continuum towards the F2 steady-state frequency. Between-category discrimination improved from performance at 92 dB when 92-dB speech stimuli were simultaneously masked by 60-dB speech noise; alveolar-velar boundaries shifted to a higher frequency location in the 92-dB-plus-noise condition. Chirps were discriminated categorically when presented at 58 dB, but discrimination peaks declined at higher intensities. Perceptual performance for chirps and p-t-k stimuli was very similar, and slightly inferior to performance for b-d-g stimuli, where simultaneous masking by F1 resulted in a lower effective intensity of F2. The results were related to a suggested model involving pitch comparison and transitional quality perceptual strategies.     

Speech perception and the discrimination of brief auditory cues in reading disabled children

The processing of speech and nonspeech sounds by 23 reading disabled children and their age- and sex-matched controls was examined in a task requiring them to identify and report the order of pairs of stimuli. Reading disabled children were impaired in making judgments with very brief tones and with stop consonant syllables at short interstimulus intervals (ISI's). They had no unusual difficulty with vowel stimuli, vowel stimuli in a white noise background, or very brief visual figures. Poor performance on the tones and stop consonants appears to be due to specific difficulty in processing very brief auditory cues. The reading disabled children also showed deficits in the perception of naturally produced words, less sharply defined category boundaries, and a greater reliance on context in making phoneme identifications. The results suggest a perceptual deficit in some reading disabled children, which interferes with the processing of phonological information.     

Phonetic recoding of print and its effect on the detection of concurrent speech in amplitude-modulated noise.

When an amplitude-modulated noise generated from a spoken word is presented simultaneously with the word's printed version, the noise sounds more speechlike. This auditory illusion obtained by Frost, Repp, and Katz (1988) suggests that subjects detect correspondences between speech amplitude envelopes and printed stimuli. The present study investigated whether the speech envelope is assembled from the printed word or whether it is lexically addressed. In two experiments subjects were presented with speech-plus-noise and with noise-only trials, and were required to detect the speech in the noise. The auditory stimuli were accompanied with matching or non-matching Hebrew print, which was unvoweled in Experiment 1 and voweled in Experiment 2. The stimuli of both experiments consisted of high-frequency words, low-frequency words, and non-words. The results demonstrated that matching print caused a strong bias to detect speech in the noise when the stimuli were either high- or low-frequency words, whereas no bias was found for non-words. The bias effect for words or non-words was not affected by spelling to sound regularity; that is, similar effects were obtained in the voweled and the unvoweled conditions. These results suggest that the amplitude envelope of the word is not assembled from the print. Rather, it is addressed directly from the printed word and retrieved from the mental lexicon. Since amplitude envelopes are contingent on detailed phonetic structures, this outcome suggests that representations of words in the mental lexicon are not only phonological but also phonetic in character.     

Repetition priming of words and pseudowords in divided attention and in amnesia

Repetition priming of novel stimuli (pseudowords) and stimuli with preexisting representations (words) was compared in two experiments. In one, 19 normal male subjects performed a lexical decision task with either focused or divided attention. In another, lexical decision performance was compared between 8 male Korsakoff patients and 8 alcoholic control subjects. In control conditions, repetition speeded responses to both stimulus types. Experimental conditions that minimized the contribution of episodic memory to task performance eliminated reaction time priming for pseudowords but not for words. However, in these same conditions, repetition increased the likelihood that pseudowords would be incorrectly classified. These results indicate that preserved repetition priming effects in amnesia do not solely reflect activation of representations in semantic memory.     

A "voice inversion effect?"

Voice is the carrier of speech but is also an "auditory face" rich in information on the speaker's identity and affective state. Three experiments explored the possibility of a "voice inversion effect," by analogy to the classical "face inversion effect," which could support the hypothesis of a voice-specific module. Experiment 1 consisted of a gender identification task on two syllables pronounced by 90 speakers (boys, girls, men, and women). Experiment 2 consisted of a speaker discrimination task on pairs of syllables (8 men and 8 women). Experiment 3 consisted of an instrument discrimination task on pairs of melodies (8 string and 8 wind instruments). In all three experiments, stimuli were presented in 4 conditions: (1) no inversion; (2) temporal inversion (e.g., backwards speech); (3) frequency inversion centered around 4000 Hz; and (4) around 2500 Hz. Results indicated a significant decrease in performance caused by sound inversion, with a much stronger effect for frequency than for temporal inversion. Interestingly, although frequency inversion markedly affected timbre for both voices and instruments, subjects' performance was still above chance. However, performance at instrument discrimination was much higher than for voices, preventing comparison of inversion effects for voices vs. non-vocal stimuli. Additional experiments will be necessary to conclude on the existence of a possible "voice inversion effect."     

Perceptual restoration of a “missing” speech sound: Auditory induction or illusion?

This study investigated whether the apparent completeness of the acoustic speech signal during phonemic restoration derives from a process of auditory induction (Warren, 1984) or segregation, or whether it is an auditory illusion that accompanies the completion of an abstract phonological representation. Specifically, five experiments tested the prediction of the auditory induction (segregation) hypothesis that active perceptual restoration of an [s] noise that has been replaced with an extraneous noise would use up a portion of that noise’s high-frequency energy and consequently change the perceived pitch (timbre, brightness) of the extraneous noise. Listeners were required to compare the pitch of a target noise, which replaced a fricative noise in a sentence, with that of a probe noise preceding or following the speech. In the first two experiments, a significant tendency was found in favor of the auditory induction hypothesis, although the effect was small and may have been caused by variations in acoustic context. In the following three experiments, a larger variety of stimuli were used and context was controlled more carefully; this yielded negative results. Phoneme identification responses collected in the same experiments, as well as informal observations about the quality of the restored phoneme, suggested that restoration of a fricative phone distinct from the extraneous noise did not occur; rather, the spectrum of the extraneous noise itself influenced phoneme identification. These results suggest that the apparent auditory restoration which accompanies phonemic restoration is illusory, and that the schema-guided process of phoneme restoration does not interact with auditory processing.     

Vertical similarity in spoken word recognition: Multiple lexical activation,individual differences,and the role of sentence context

Four experiments investigated acoustic-phonetic similarity in the mapping process between the speech signal and lexical representations (vertical similarity). Auditory stimuli were used where ambiguous initial phonemes rendered a phoneme sequence lexically ambiguous (perceptual-lexical ambiguities). A cross-modal priming paradigm (Experiments 1, 2, and 3) showed facilitation for targets related to both interpretations of the ambiguities, indicating multiple activation. Experiment 4 investigated individual differences and the role of sentence context in vertical similarity mapping. The results support a model where spoken word recognition proceeds via goodness-of-fit mapping between speech and lexical representations that is not influenced by sentence context.     

Perceptual restoration of a "missing" speech sound: auditory induction or illusion?

This study investigated whether the apparent completeness of the acoustic speech signal during phonemic restoration derives from a process of auditory induction (Warren, 1984) or segregation, or whether it is an auditory illusion that accompanies the completion of an abstract phonological representation. Specifically, five experiments tested the prediction of the auditory induction (segregation) hypothesis that active perceptual restoration of an [s] noise that has been replaced with an extraneous noise would use up a portion of that noise's high-frequency energy and consequently change the perceived pitch (timbre, brightness) of the extraneous noise. Listeners were required to compare the pitch of a target noise, which replaced a fricative noise in a sentence, with that of a probe noise preceding or following the speech. In the first two experiments, a significant tendency was found in favor of the auditory induction hypothesis, although the effect was small and may have been caused by variations in acoustic context. In the following three experiments, a larger variety of stimuli were used and context was controlled more carefully; this yielded negative results. Phoneme identification responses collected in the same experiments, as well as informal observations about the quality of the restored phoneme, suggested that restoration of a fricative phone distinct from the extraneous noise did not occur; rather, the spectrum of the extraneous noise itself influenced phoneme identification. These results suggest that the apparent auditory restoration which accompanies phonemic restoration is illusory, and that the schema-guided process of phoneme restoration does not interact with auditory processing.     

What Are You Waiting For? Real‐Time Integration of Cues for Fricatives Suggests Encapsulated Auditory Memory

Speech unfolds over time, and the cues for even a single phoneme are rarely available simultaneously. Consequently, to recognize a single phoneme, listeners must integrate material over several hundred milliseconds. Prior work contrasts two accounts: (a) a memory buffer account in which listeners accumulate auditory information in memory and only access higher level representations (i.e., lexical representations) when sufficient information has arrived; and (b) an immediate integration scheme in which lexical representations can be partially activated on the basis of early cues and then updated when more information arises. These studies have uniformly shown evidence for immediate integration for a variety of phonetic distinctions. We attempted to extend this to fricatives, a class of speech sounds which requires not only temporal integration of asynchronous cues (the frication, followed by the formant transitions 150–350 ms later), but also integration across different frequency bands and compensation for contextual factors like coarticulation. Eye movements in the visual world paradigm showed clear evidence for a memory buffer. Results were replicated in five experiments, ruling out methodological factors and tying the release of the buffer to the onset of the vowel. These findings support a general auditory account for speech by suggesting that the acoustic nature of particular speech sounds may have large effects on how they are processed. It also has major implications for theories of auditory and speech perception by raising the possibility of an encapsulated memory buffer in early auditory processing.     

The role of facial colour and luminance in visual and audiovisual speech perception

We conducted four experiments to investigate the role of colour and luminance information in visual and audiovisual speech perception. In experiments 1a (stimuli presented in quiet conditions) and 1b (stimuli presented in auditory noise), face display types comprised naturalistic colour (NC), grey-scale (GS), and luminance inverted (LI) faces. In experiments 2a (quiet) and 2b (noise), face display types comprised NC, colour inverted (CI), LI, and colour and luminance inverted (CLI) faces. Six syllables and twenty-two words were used to produce auditory and visual speech stimuli. Auditory and visual signals were combined to produce congruent and incongruent audiovisual speech stimuli. Experiments 1a and 1b showed that perception of visual speech, and its influence on identifying the auditory components of congruent and incongruent audiovisual speech, was less for LI than for either NC or GS faces, which produced identical results. Experiments 2a and 2b showed that perception of visual speech, and influences on perception of incongruent auditory speech, was less for LI and CLI faces than for NC and CI faces (which produced identical patterns of performance). Our findings for NC and CI faces suggest that colour is not critical for perception of visual and audiovisual speech. The effect of luminance inversion on performance accuracy was relatively small (5%), which suggests that the luminance information preserved in LI faces is important for the processing of visual and audiovisual speech.     

The relation of phoneme discrimination, lexical access, and short-term memory: A case study and interactive activation account

A brain-damaged patient (AP) is reported who had a strong tendency to identify nonwords as words on auditory lexical decision and to lexicalize nonwords in repetition, yet who showed a normal ability to perceive individual phonemes. It was initially hypothesized that these findings could be accounted for in terms of disrupted lexical phonological representations. This hypothesis was rejected on the basis of an interactive activation model of word recognition which revealed that modifications at the lexical level did not mimic the patient's pattern of results. Instead, it was found that increasing the rate of decay of activation at the phoneme level produced output that was consistent with the phoneme discrimination, lexical decision, and repetition results. This hypothesis of increased phoneme level decay led to the prediction that speech discrimination would decline with increased interstimulus interval and that short-term memory performance would be impaired. Both predictions were confirmed. The results of this study provide support for an interactive activation model of word recognition with feedback from the lexical to the phonemic level and for a close connection between the processes involved in word recognition and short-term memory.     

Examining the interactivity of lexical orthographic and phonological processing.

The number and type of connections involving different levels of orthographic and phonological representations differentiate between several models of spoken and visual word recognition. At the sublexical level of processing, Borowsky, Owen, and Fonos (1999) demonstrated evidence for direct processing connections from grapheme representations to phoneme representations (i.e., a sensitivity effect) over and above any bias effects, but not in the reverse direction. Neural network models of visual word recognition implement an orthography to phonology processing route that involves the same connections for processing sublexical and lexical information, and thus a similar pattern of cross-modal effects for lexical stimuli are expected by models that implement this single type of connection (i.e., orthographic lexical processing should directly affect phonological lexical processing, but not in the reverse direction). Furthermore, several models of spoken word perception predict that there should be no direct connections between orthographic representations and phonological representations, regardless of whether the connections are sublexical or lexical. The present experiments examined these predictions by measuring the influence of a cross-modal word context on word target discrimination. The results provide constraints on the types of connections that can exist between orthographic lexical representations and phonological lexical representations.     

Perceptual reference points for form and orientation in young infants: anchors or magnets?

In some domains, certain stimuli are especially salient and efficiently encoded and are referred to as reference points. One current issue concerns whether reference points are associated with regions of increased or decreased discriminability and function as either perceptual anchors or magnets. In two experiments utilizing the familiarization/novelty-preference procedure, the question of whether 3- to 4-month-old infants' representations of form and orientation information are structured by perceptual reference points and whether such reference points serve as anchors or magnets is examined. In Experiment 1, infants displayed above-chance discrimination performance for pairs of form stimuli that were equivalently distinct on a physical basis, but only when one member of each pair was a "good" form (i.e., diamond, square, or triangle). In Experiment 2, infants displayed above-chance discrimination performance for pairs of stimuli differing by 7.5 degrees of orientation, but only when one member of each pair was either horizontal or vertical. The combined results from the two experiments suggest that "simple" gestalts and main axes (i.e., horizontal and vertical) serve as perceptual anchors in young infants' representations of form and orientation information.     

Phonemic restorations based on subsequent context

Earlier experiments have shown that when one or more speech sounds in a sentence are replaced by a noise meeting certain criteria, the listener mislocalizes the extraneous sound and believes he hears the missing phoneme(s) clearly. The present study confirms and extends these earlier reports of phonemic restorations under a variety of novel conditions. All stimuli had some of the context necessary for the appropriate phonemic restoration following the missing sound, and all sentences had the missing phoneme deliberately mispronounced before electronic deletion (so that the neighboring phonemes could not provide acoustic cues to aid phonemic restorations). The results are interpreted in terms of mechanisms normally aiding veridical perception of speech and nonspeech sounds.     

Spoken word recognition by young children and adults

The nature of the stimulus information that is important for the recognition of auditorily presented words by young (5-year-old) children and adults was studied. In Experiment 1, subjects identified and rated the extent of noise disruption for words in which white noise either was added to or replaced phoneme (fricative and nonfricative) segments in word-initial, -medial or -final position. In Experiment 2, subjects identified words as acoustic-phonetic information accumulated either from their beginnings or ends with silence or envelope-shaped noise replacing the nonpresented parts. The results point to developmental similarities in the derivation of phoneme identities from impoverished sensory input to support the component processes of recognition. However, position-specific information may play a less prominent role in recognition for children than for adults.     

Segmentation and speech perception in relation to reading skill: a developmental analysis

The present paper examines the processing of speech by dyslexic readers and compares their performance with that of age-matched (CA) and reading-ability-matched (RA) controls. In Experiment 1, subjects repeated single-syllable stimuli (words and nonwords) presented either in a favorable signal-to-noise ratio or with noise masking. Noise affected all subjects to the same extent. Dyslexic children performed as well as controls when repeating high-frequency words, but they had difficulty relative to CA-controls with low-frequency words and relative to both CA- and RA-controls when repeating nonwords. In Experiments 2 and 3, subjects made auditory lexical decisions about the stimuli presented in Experiment 1. Dyslexics performed less well than CA-controls, gaining similar scores to RA-controls. Thus, their difficulty in repeating low-frequency words could be reinterpreted as a difficulty with nonword repetition. Taken together, these results suggest that dyslexics have difficulty with the nonlexical procedures (including phoneme segmentation) involved in verbal repetition. One consequence is that they take longer to consolidate "new" words; verbal memory and reading processes are also compromised.     

Perception of speech and nonspeech stimuli by children with and without reading disability and attention deficit hyperactivity disorder

The auditory temporal deficit hypothesis predicts that children with reading disability (RD) will exhibit deficits in the perception of speech and nonspeech acoustic stimuli in discrimination and temporal ordering tasks when the interstimulus interval (ISI) is short. Initial studies testing this hypothesis did not account for the potential presence of attention deficit hyperactivity disorder (ADHD). Temporal order judgment and discrimination tasks were administered to children with (1) RD/no-ADHD (n=38), (2) ADHD (n=29), (3) RD and ADHD (RD/ADHD; n=32), and (4) no impairment (NI; n=43). Contrary to predictions, children with RD showed no specific sensitivity to ISI and performed worse relative to children without RD on speech but not nonspeech tasks. Relationships between perceptual tasks and phonological processing measures were stronger and more consistent for speech than nonspeech stimuli. These results were independent of the presence of ADHD and suggest that children with RD have a deficit in phoneme perception that correlates with reading and phonological processing ability. (c) 2002 Elsevier Science (USA).