Can you McGurk yourself? Self-face and self-voice in audiovisual speech

We are constantly exposed to our own face and voice, and we identify our own faces and voices as familiar. However, the influence of self-identity upon self-speech perception is still uncertain. Speech perception is a synthesis of both auditory and visual inputs; although we hear our own voice when we speak, we rarely see the dynamic movements of our own face. If visual speech and identity are processed independently, no processing advantage would obtain in viewing one’s own highly familiar face. In the present experiment, the relative contributions of facial and vocal inputs to speech perception were evaluated with an audiovisual illusion. Our results indicate that auditory self-speech conveys a processing advantage, whereas visual self-speech does not. The data thereby support a model of visual speech as dynamic movement processed separately from speaker recognition.     

Crossmodal binding: Evaluating the “unity assumption” using audiovisual speech stimuli

We investigated whether the “unity assumption,” according to which an observer assumes that two different sensory signals refer to the same underlying multisensory event, influences the multisensory integration of audiovisual speech stimuli. Syllables (Experiments 1, 3, and 4) or words (Experiment 2) were presented to participants at a range of different stimulus onset asynchronies using the method of constant stimuli. Participants made unspeeded temporal order judgments regarding which stream (either auditory or visual) had been presented first. The auditory and visual speech stimuli in Experiments 1–3 were either gender matched (i.e., a female face presented together with a female voice) or else gender mismatched (i.e., a female face presented together with a male voice). In Experiment 4, different utterances from the same female speaker were used to generate the matched and mismatched speech video clips. Measuring in terms of the just noticeable difference the participants in all four experiments found it easier to judge which sensory modality had been presented first when evaluating mismatched stimuli than when evaluating the matched-speech stimuli. These results therefore provide the first empirical support for the “unity assumption” in the domain of the multisensory temporal integration of audiovisual speech stimuli.     

Do age and linguistic background alter the audiovisual advantage when listening to speech in the presence of energetic and informational masking?

We examined how the type of masker presented in the background affected the extent to which visual information enhanced speech recognition, and whether the effect was dependent on or independent of age and linguistic competence. In the present study, young speakers of English as a first language (YEL1) and English as a second language (YEL2), as well as older speakers of English as a first language (OEL1), were asked to complete an audio (A) and an audiovisual (AV) speech recognition task in which they listened to anomalous target sentences presented against a background of one of three masker types (noise, babble, and competing speech). All three main effects were found to be statistically significant (group, masker type, A vs. AV presentation type). Interesting two-way interactions were found between masker type and group and between masker type and presentation type; however, no interactions were found between group (age and/or linguistic competence) and presentation type (A vs. AV). The results of this study, while they shed light on the effect of masker type on the AV advantage, suggest that age and linguistic competence have no significant effects on the extent to which a listener is able to use visual information to improve speech recognition in background noise.     

The influence of infant-directed speech on 12-month-olds’ intersensory perception of fluent speech

The present study examined whether infant-directed (ID) speech facilitates intersensory matching of audio–visual fluent speech in 12-month-old infants. German-learning infants’ audio–visual matching ability of German and French fluent speech was assessed by using a variant of the intermodal matching procedure, with auditory and visual speech information presented sequentially. In Experiment 1, the sentences were spoken in an adult-directed (AD) manner. Results showed that 12-month-old infants did not exhibit a matching performance for the native, nor for the non-native language. However, Experiment 2 revealed that when ID speech stimuli were used, infants did perceive the relation between auditory and visual speech attributes, but only in response to their native language. Thus, the findings suggest that ID speech might have an influence on the intersensory perception of fluent speech and shed further light on multisensory perceptual narrowing.     

On morality of speech: Cavell’s critique of Derrida

This article tries to bring out the implication of Cavell’s critical comments on Derrida, clustered around Cavell’s charge that deconstruction entails a flight from the ordinary. Cavell’s and Derrida’s different readings of Austin’s ordinary language philosophy provide a common ground for elaborating their respective positions. Their writings are at once the closest but also the most divergent when addressing the moral implication of speech, or more precisely, when addressing their understanding of responsibility and voice. Employing Derrida’s so-called ‘double reading’ as a leitmotif will not only shed light on the moral dimension of deconstruction, but also bring the central target of Cavell’s critique into the open.     

Perceptual study of the speech of “successfully therapeutized” stutterers

The purpose of this investigation was to determine if the fluent speech of “successfully therapeutized” stutterers and a group of partially treated stutterers was perceptually different from the fluent speech of normal speaking subjects. Tape recorded speech samples of treated stutterers were obtained from leading exponents of (1) Van Riperian, (2) metronome-conditioned speech retraining, (3) delayed auditory feedback, (4) operant conditioning, (5) precision fluency shaping, and (6) “holistic” therapy programs. All forms of disfluency were edited out of these tapes. The remaining samples were then paired with matched fluent samples of normal talkers and presented to a group of 20 sophisticated judges. The judges were instructed to select from each paired speech sample presented to them the one produced by the stuttering subject. The results of the analyses of variance for correct identification of the stutterers showed that the partially treated stutterers, as well as each group of therapeutized stutterers, were identified at levels significantly above chance. This meant that the fluent speech of the partially and successfully treated stutterers was perceptibly different from the utterances of the normal speakers sampled. In addition, the analyses of variance also revealed that all stuttering groups had a significant severity factor. This finding indicated there was a significant difference between the severity subgroup in all the samples investigated. These results were discussed in terms of their clinical implications.     

Does awareness of speech as a sequence of phones arise spontaneously?

It was found that illiterate adults could neither delete nor add a phone at the beginning of a non-word; but these tasks were rather easily performed by people with similar environment and childhood experiences, who learned to read rudimentarily as adults. Awareness of speech as a sequence of phones is thus not attained spontaneously in the course of general cognitive growth, but demands some specific training, which, for most persons, is probably provided by learning to read in the alphabetic system.     

Influence of auditory and audiovisual stimuli on the right–left prevalence effect

When auditory stimuli are used in two-dimensional spatial compatibility tasks, where the stimulus and response configurations vary along the horizontal and vertical dimensions simultaneously, a right–left prevalence effect occurs in which horizontal compatibility dominates over vertical compatibility. The right–left prevalence effects obtained with auditory stimuli are typically larger than that obtained with visual stimuli even though less attention should be demanded from the horizontal dimension in auditory processing. In the present study, we examined whether auditory or visual dominance occurs when the two-dimensional stimuli are audiovisual, as well as whether there will be cross-modal facilitation of response selection for the horizontal and vertical dimensions. We also examined whether there is an additional benefit of adding a pitch dimension to the auditory stimulus to facilitate vertical coding through use of the spatial-musical association of response codes (SMARC) effect, where pitch is coded in terms of height in space. In Experiment 1, we found a larger right–left prevalence effect for unimodal auditory than visual stimuli. Neutral, non-pitch coded, audiovisual stimuli did not result in cross-modal facilitation, but did show evidence of visual dominance. The right–left prevalence effect was eliminated in the presence of SMARC audiovisual stimuli, but the effect influenced horizontal rather than vertical coding. Experiment 2 showed that the influence of the pitch dimension was not in terms of influencing response selection on a trial-to-trial basis, but in terms of altering the salience of the task environment. Taken together, these findings indicate that in the absence of salient vertical cues, auditory and audiovisual stimuli tend to be coded along the horizontal dimension and vision tends to dominate audition in this two-dimensional spatial stimulus–response task.     

Infant directed speech and the development of speech perception: Enhancing development or an unintended consequence?

Infant directed speech (IDS) is a speech register characterized by simpler sentences, a slower rate, and more variable prosody. Recent work has implicated it in more subtle aspects of language development. Kuhl et al. (1997) demonstrated that segmental cues for vowels are affected by IDS in a way that may enhance development: the average locations of the extreme “point” vowels (/a/, /i/ and /u/) are further apart in acoustic space. If infants learn speech categories, in part, from the statistical distributions of such cues, these changes may specifically enhance speech category learning. We revisited this by asking (1) if these findings extend to a new cue (Voice Onset Time, a cue for voicing); (2) whether they extend to the interior vowels which are much harder to learn and/or discriminate; and (3) whether these changes may be an unintended phonetic consequence of factors like speaking rate or prosodic changes associated with IDS. Eighteen caregivers were recorded reading a picture book including minimal pairs for voicing (e.g., beach/peach) and a variety of vowels to either an adult or their infant. Acoustic measurements suggested that VOT was different in IDS, but not in a way that necessarily supports better development, and that these changes are almost entirely due to slower rate of speech of IDS. Measurements of the vowel suggested that in addition to changes in the mean, there was also an increase in variance, and statistical modeling suggests that this may counteract the benefit of any expansion of the vowel space. As a whole this suggests that changes in segmental cues associated with IDS may be an unintended by-product of the slower rate of speech and different prosodic structure, and do not necessarily derive from a motivation to enhance development.     

An adult model of preschool children’s speech memory

In previous work, it has been demonstrated that phonetic similarity among the items in a spoken list interferes with recall much more in school-aged children than in preschool children. The basis of this developmental change, however, is unclear. In the present study we examined the possibility that a developmental increase in the use of covert verbal rehearsal accounts for the change in the effects of phonetic similarity. Adults who recalled lists of spoken words during articulatory suppression tasks that blocked covert rehearsal were found to display patterns of recall that resembled those ordinarily found in 5-year-old children. The specific aspects of rehearsal responsible for these effects also were investigated.     

Is infant-directed speech prosody a result of the vocal expression of emotion?

Many studies have found that infant-directed (ID) speech has higher pitch, has more exaggerated pitch contours, has a larger pitch range, has a slower tempo, and is more rhythmic than typical adult-directed (AD) speech. We show that the ID speech style reflects free vocal expression of emotion to infants, in comparison with more inhibited expression of emotion in typical AD speech. When AD speech does express emotion, the same acoustic features are used as in ID speech. We recorded ID and AD samples of speech expressing love-comfort, fear, and surprise. The emotions were equally discriminable in the ID and AD samples. Acoustic analyses showed few differences between the ID and AD samples, but robust differences across the emotions. We conclude that ID prosody itself is not special. What is special is the widespread expression of emotion to infants in comparison with the more inhibited expression of emotion in typical adult interactions.     

Cortical integration of audio–visual speech and non-speech stimuli

Using fMRI we investigated the neural basis of audio–visual processing of speech and non-speech stimuli using physically similar auditory stimuli (speech and sinusoidal tones) and visual stimuli (animated circles and ellipses). Relative to uni-modal stimuli, the different multi-modal stimuli showed increased activation in largely non-overlapping areas. Ellipse-Speech, which most resembles naturalistic audio–visual speech, showed higher activation in the right inferior frontal gyrus, fusiform gyri, left posterior superior temporal sulcus, and lateral occipital cortex. Circle-Tone, an arbitrary audio–visual pairing with no speech association, activated middle temporal gyri and lateral occipital cortex. Circle-Speech showed activation in lateral occipital cortex, and Ellipse-Tone did not show increased activation relative to uni-modal stimuli. Further analysis revealed that middle temporal regions, although identified as multi-modal only in the Circle-Tone condition, were more strongly active to Ellipse-Speech or Circle-Speech, but regions that were identified as multi-modal for Ellipse-Speech were always strongest for Ellipse-Speech. Our results suggest that combinations of auditory and visual stimuli may together be processed by different cortical networks, depending on the extent to which multi-modal speech or non-speech percepts are evoked.     

Is desynchrony tolerance adaptable in the perceptual organization of speech?

Speech signal components that are desynchronized from the veridical temporal pattern lose intelligibility. In contrast, audiovisual presentations with large desynchrony in visible and audible speech streams are perceived without loss of integration. Under such conditions, the limit of desynchrony that permits audiovisual integration is also adaptable. A new project directly investigated the potential for adaptation to consistent desynchrony with unimodal auditory sine-wave speech. Listeners transcribed sentences that are highly intelligible, with veridical temporal properties. Desynchronized variants were created by leading or lagging the tone analog of the second formant relative to the rest of the tones composing the sentences, in 50-msec steps, ranging from 250-msec lead to 250-msec lag. In blocked trials, listeners only tolerated desynchronies <50 msec, and exhibited no gain in intelligibility to consistent desynchrony. Unimodal auditory and bimodal audiovisual forms of perceptual integration evidently exhibit different temporal characteristics, an indication of distinct perceptual functions.     

Speaking of grief and the grief of speaking: martyrs’ speech and the perils of translation

In Derrida’s Monolinguism of the Other, a theory about the universal and constitutive alienation of the speaking subject from language finds its exemplary grounding in Derrida’s own experience as an Algerian Jew, one whose relationship to the French language is both totalizing and exiled (‘I have only one language, it is not mine.’). He equates speaking not only with contingent citizenship and a divestment of what one never really had in the first place, but also with the extreme experiences of torture, threat and physical violence. He indeed uses the words ‘passion’ and ‘martyr’ to describe his experience. In this paper, I will read Derrida ‘backwards,’ and against the universalizing move Derrida and those following him make in order to suggest a way of reading some scenes of violent death as scenes about diasporic cultural divestment. I’ll specifically attend to martyrs’ speech, and do so reading them as archives of the perils and inescapable expenses of entering dominant cultural ‘languages.’     

Disruption of proofreading by irrelevant speech: Effects of attention,arousal or memory?

A series of five experiments examined the effects of irrelevant speech on proofreading and memory. Four of the experiments used a proofreading task and showed that the deleterious effects of irrelevant speech: (1) depend on the speech being meaningful, (2) are only present when the burden on short-term memory is low and (3) are manifested in a lower detection rate for non-contextual as opposed to contextual errors. Neither the spatial location of the speech (either in terms of spatial dispersion of sources or spatial movement of a single source) nor the intensity of the speech (in a range bounded by 50 dB(A) and 70 dB(A)) had any effect on proofreading. Late selection models of attention are favoured by the results in preference to models having arousal, short-term memory or early selection in attention as their basis. A final experiment showed serial recall for visual lists to be impaired by the presence of any speech-like sound (including reversed speech and speech in an unfamiliar language) which suggests a set of phenomena qualitatively different from those associated with proofreading. Throughout the article the practical consequences of the findings are emphasized.     

Unsophisticated judges' perceptual evaluations of the speech of “successfully treated” stutterers

The purpose of this investigation was to determine if the speech of “successfully therapeutized” stutterers and a group of partially treated stutterers was perceptually different from the speech of normal speakers when judged by unsophisticated listeners. Tape-recorded speech samples of treated stutterers were obtained from leading proponents of (1) Van Riperian, (2) metronome-conditioned speech retraining, (3) delayed auditory feedback, (4) operant conditioning, (5) precision fluency shaping, and (6) “holistic” therapy programs. Fluent speech samples from these groups of stutterers were paired with matched fluent samples of normal talkers and presented to a group of 20 unsophisticated judges. The judges were instructed to select from each paired speech sample presented to them the one produced by the stuttering subject. The results of the analyses showed that five of seven experimental groups were identified at levels significantly above chance. It can be concluded that the fluent speech of the partially and successfully treated stutterers was perceptibly different from the utterances of the normal speakers and that the perceptual disparity can be detected, even by unsophisticated listeners.