Acoustic memory and the perception of speech

The nature of acoustic memory and its relationship to the categorizing process in speech perception is investigated in three experiments on the serial recall of lists of syllables. The first study confirms previous reports that sequences comprising the syllables, bah, dah, and gah show neither enhanced retention when presented auditorily rather than visually, nor a recency effect—both occurred with sequences in which vowel sounds differed (bee, bih, boo). This was found not to be a simple vowel-consonant difference since acoustic memory effects did occur with consonant sequences that were acoustically more discriminable (sha, ma, ga and ash, am, ag). Further experiments used the stimulus suffix effect to provide evidence of acoustic memory, and showed (1), increasing the acoustic similarity of the set grossly impairs acoustic memory effects for vowels as well as consonants, and (2) such memory effects are no greater for steady-state vowels than for continuously changing diphthongs. It is concluded that the usefulness of the information that can be retrieved from acoustic memory depends on the acoustic similarity of the items in the list rather than on their phonetic class or whether or not they have “encoded” acoustic cues. These results question whether there is any psychological evidence for “encoded” speech sounds being categorized in ways different from other speech sounds.     

Effects of intelligibility on working memory demand for speech perception

Understanding low-intelligibility speech is effortful. In three experiments, we examined the effects of intelligibility on working memory (WM) demands imposed by perception of synthetic speech. In all three experiments, a primary speeded word recognition task was paired with a secondary WM-load task designed to vary the availability of WM capacity during speech perception. Speech intelligibility was varied either by training listeners to use available acoustic cues in a more diagnostic manner (as in Experiment 1) or by providing listeners with more informative acoustic cues (i.e., better speech quality, as in Experiments 2 and 3). In the first experiment, training significantly improved intelligibility and recognition speed; increasing WM load significantly slowed recognition. A significant interaction between training and load indicated that the benefit of training on recognition speed was observed only under low memory load. In subsequent experiments, listeners received no training; intelligibility was manipulated by changing synthesizers. Improving intelligibility without training improved recognition accuracy, and increasing memory load still decreased it, but more intelligible speech did not produce more efficient use of available WM capacity. This suggests that perceptual learning modifies the way available capacity is used, perhaps by increasing the use of more phonetically informative features and/or by decreasing use of less informative ones.     

Speech perception and short-term memory deficits in persistent developmental speech disorder

Children with developmental speech disorders may have additional deficits in speech perception and/or short-term memory. To determine whether these are only transient developmental delays that can accompany the disorder in childhood or persist as part of the speech disorder, adults with a persistent familial speech disorder were tested on speech perception and short-term memory. Nine adults with a persistent familial developmental speech disorder without language impairment were compared with 20 controls on tasks requiring the discrimination of fine acoustic cues for word identification and on measures of verbal and nonverbal short-term memory. Significant group differences were found in the slopes of the discrimination curves for first formant transitions for word identification with stop gaps of 40 and 20 ms with effect sizes of 1.60 and 1.56. Significant group differences also occurred on tests of nonverbal rhythm and tonal memory, and verbal short-term memory with effect sizes of 2.38, 1.56, and 1.73. No group differences occurred in the use of stop gap durations for word identification. Because frequency-based speech perception and short-term verbal and nonverbal memory deficits both persisted into adulthood in the speech-impaired adults, these deficits may be involved in the persistence of speech disorders without language impairment.     

Significance of pauses for speech perception

Pauses can be used to facilitate certain operations involved in the production and in the perception of speech. In the case of speech perception, pauses have been found to improve the accuracy of detection and the recall of lists of digits and letters. The aim of the present experiments was to examine the effects of pause time on the perception of sentences. In experiment I, a semantic categorization task was used and in experiment II a sentence recall task. The results indicated that in sentences containing pauses between clauses, words were categorized more rapidly (experiment I) and propositions were recalled more accurately (experiment II) than in sentences containing pauses, within the clause. The results are interpreted in the context of existing models of speech processing, and the significane of pause time for cognitive activity is discussed.     

Phonological similarity and the irrelevant speech effect: implications for models of short-term verbal memory

Three experiments studied the interaction between irrelevant speech and phonological similarity within both the remembered and the irrelevant auditory material. Phonological similarity within the remembered list impaired performance in both baseline and irrelevant speech conditions, whereas phonological similarity between the remembered and ignored irrelevant items did not influence performance. Although there was a tendency for similarity within the irrelevant items to reduce interference, this proved to be a less robust finding. Implications for the theoretical interpretation of the irrelevant speech effect are discussed.     

Phonological similarity and the irrelevant speech effect: Implications for models of short-term verbal memory

Three experiments studied the interaction between irrelevant speech and phonological similarity within both the remembered and the irrelevant auditory material. Phonological similarity within the remembered list impaired performance in both baseline and irrelevant speech conditions, whereas phonological similarity between the remembered and ignored irrelevant items did not influence performance. Although there was a tendency for similarity within the irrelevant items to reduce interference, this proved to be a less robust finding. Implications for the theoretical interpretation of the irrelevant speech effect are discussed.     

Reading speech and hearing print: constraining models of visual word recognition by exploring connections with speech perception.

Current models of reading and speech perception differ widely in their assumptions regarding the interaction of orthographic and phonological information during language perception. The present experiments examined this interaction through a two-alternative, forced-choice paradigm, and explored the nature of the connections between graphemic and phonemic processing subsystems. Experiments 1 and 2 demonstrated a facilitation-dominant influence (i.e., benefits exceed costs) of graphemic contexts on phoneme discrimination, which is interpreted as a sensitivity effect. Experiments 3 and 4 demonstrated a symmetrical influence (i.e., benefits equal costs) of phonemic contexts on grapheme discrimination, which can be interpreted as either a bias effect, or an equally facilitative/inhibitory sensitivity effect. General implications for the functional architecture of language processing models are discussed, as well as specific implications for models of visual word recognition and speech perception.     

Spatial frequency requirements for audiovisual speech perception

Spatial frequency band-pass and low-pass filtered images of a talker were used in an audiovisual speech-in-noise task. Three experiments tested subjects' use of information contained in the different filter bands with center frequencies ranging from 2.7 to 44.1 cycles/face (c/face). Experiment 1 demonstrated that information from a broad range of spatial frequencies enhanced auditory intelligibility. The frequency bands differed in the degree of enhancement, with a peak being observed in a mid-range band (11-c/face center frequency). Experiment 2 showed that this pattern was not influenced by viewing distance and, thus, that the results are best interpreted in object spatial frequency, rather than in retinal coordinates. Experiment 3 showed that low-pass filtered images could produce a performance equivalent to that produced by unfiltered images. These experiments are consistent with the hypothesis that high spatial resolution information is not necessary for audiovisual speech perception and that a limited range of spatial frequency spectrum is sufficient.     

Visual contribution to the multistable perception of speech

The multistable perception of speech, or verbal transformation effect, refers to perceptual changes experienced while listening to a speech form that is repeated rapidly and continuously. In order to test whether visual information from the speaker's articulatory gestures may modify the emergence and stability of verbal auditory percepts, subjects were instructed to report any perceptual changes during unimodal, audiovisual, and incongruent audiovisual presentations of distinct repeated syllables. In a first experiment, the perceptual stability of reported auditory percepts was significantly modulated by the modality of presentation. In a second experiment, when audiovisual stimuli consisting of a stable audio track dubbed with a video track that alternated between congruent and incongruent stimuli were presented, a strong correlation between the timing of perceptual transitions and the timing of video switches was found. Finally, a third experiment showed that the vocal tract opening onset event provided by the visual input could play the role of a bootstrap mechanism in the search for transformations. Altogether, these results demonstrate the capacity of visual information to control the multistable perception of speech in its phonetic content and temporal course. The verbal transformation effect thus provides a useful experimental paradigm to explore audiovisual interactions in speech perception.     

Modeling memory and perception

I present a framework for modeling memory, retrieval, perception, and their interactions. Recent versions of the models were inspired by Bayesian induction: We chose models that make optimal decisions conditioned on a memory/perceptual system with inherently noisy storage and retrieval. The resultant models are, fortunately, largely consistent with my models dating back to the 1960s, and are therefore natural successors. My recent articles have presented simplified models in order to focus on particular applications. This article takes a larger perspective and places the individual models in a more global framework. I will discuss (1) the storage of episodic traces, the accumulation of these into knowledge (e.g., lexical/semantic traces in the case of words), and the changes in knowledge caused by learning; (2) the retrieval of information from episodic memory and from general knowledge; (3) decisions concerning storage, retrieval, and responding. Examples of applications include episodic recognition and cued and free recall, perceptual identification (naming, yes–no and forced‐choice), lexical decision, and long‐term and short‐term priming.     

Hearing-impaired children's rhythm perception and speech

The role of rhythm in the speech intelligibility of 18 hearing-impaired children, aged 15 years with hearing losses from 40 to 108 db, was investigated. Their perceptual judgement of visual rhythm sequences was superior to that of the hearing controls, but their scores were not correlated with their speech intelligibility.     

Subliminal speech perception and auditory streaming

Current theories of consciousness assume a qualitative dissociation between conscious and unconscious processing: while subliminal stimuli only elicit a transient activity, supraliminal stimuli have long-lasting influences. Nevertheless, the existence of this qualitative distinction remains controversial, as past studies confounded awareness and stimulus strength (energy, duration). Here, we used a masked speech priming method in conjunction with a submillisecond interaural delay manipulation to contrast subliminal and supraliminal processing at constant prime, mask and target strength. This delay induced a perceptual streaming effect, with the prime popping out in the supraliminal condition. By manipulating the prime-target interval (ISI), we show a qualitatively distinct profile of priming longevity as a function of prime awareness. While subliminal priming disappeared after half a second, supraliminal priming was independent of ISI. This shows that the distinction between conscious and unconscious processing depends on high-level perceptual streaming factors rather than low-level features (energy, duration).     

Asymmetries for the visual expression and perception of speech

This study explored asymmetries for movement, expression and perception of visual speech. Sixteen dextral models were videoed as they articulated: 'bat,' 'cat,' 'fat,' and 'sat.' Measurements revealed that the right side of the mouth was opened wider and for a longer period than the left. The asymmetry was accentuated at the beginning and ends of the vocalization and was attenuated for words where the lips did not articulate the first consonant. To measure asymmetries in expressivity, 20 dextral observers watched silent videos and reported what was said. The model's mouth was covered so that the left, right or both sides were visible. Fewer errors were made when the right mouth was visible compared to the left--suggesting that the right side is more visually expressive of speech. Investigation of asymmetries in perception using mirror-reversed clips revealed that participants did not preferentially attend to one side of the speaker's face. A correlational analysis revealed an association between movement and expressivity whereby a more motile right mouth led to stronger visual expressivity of the right mouth. The asymmetries are most likely driven by left hemisphere specialization for language, which causes a rightward motoric bias.     

A software tool for auditory and speech perception experimentation

A computer program capable of supporting auditory and speech perception experimentation is described. Given a continuum of acoustic stimuli, the program (Paradigm) allows the user to present those stimuli under different, well-known psychophysical paradigms (simple identification, identification with a rating scale, 2IAX, ABX, AXB, and oddity task). For discrimination tests, both high uncertainty (roving designs) and minimal psychophysical uncertainty (fixed designs) procedures are available. All the relevant time intervals can be precisely specified, and feedback is also available. Response times can be measured as well. Furthermore, the program stores subjects’ responses and provides summaries of experimental results for both individual subjects and groups. The program runs on Microsoft Windows (3.1 or 95) on personal computers equipped with any soundboard.