The role of the right hemisphere in speech act comprehension

In this research the role of the RH in the comprehension of speech acts (or illocutionary force) was examined. Two split-screen experiments were conducted in which participants made lexical decisions for lateralized targets after reading a brief conversation remark. On one-half of the trials the target word named the speech act performed with the preceding conversation remark; on the remaining trials the target did not name the speech act that the remark performed. In both experiments, lexical decisions were facilitated for targets representing the speech act performed with the prior utterance, but only when the target was presented to the left visual field (and hence initially processed by the RH) and not when presented to the right visual field. This effect occurred at both short (Experiment 1: 250 ms) and long (Experiment 2: 1000 ms) delays. The results demonstrate the critical role played by the RH in conversation processing.     

Language identification from visual-only speech signals

Our goal in the present study was to examine how observers identify English and Spanish from visual-only displays of speech. First, we replicated the recent findings of Soto-Faraco et al. (2007) with Spanish and English bilingual and monolingual observers using different languages and a different experimental paradigm (identification). We found that prior linguistic experience affected response bias but not sensitivity (Experiment 1). In two additional experiments, we investigated the visual cues that observers use to complete the languageidentification task. The results of Experiment 2 indicate that some lexical information is available in the visual signal but that it is limited. Acoustic analyses confirmed that our Spanish and English stimuli differed acoustically with respect to linguistic rhythmic categories. In Experiment 3, we tested whether this rhythmic difference could be used by observers to identify the language when the visual stimuli is temporally reversed, thereby eliminating lexical information but retaining rhythmic differences. The participants performed above chance even in the backward condition, suggesting that the rhythmic differences between the two languages may aid language identification in visual-only speech signals. The results of Experiments 3A and 3B also confirm previous findings that increased stimulus length facilitates language identification. Taken together, the results of these three experiments replicate earlier findings and also show that prior linguistic experience, lexical information, rhythmic structure, and utterance length influence visual-only language identification.     

Alignment to visual speech information

Speech alignment is the tendency for interlocutors to unconsciously imitate one another’s speaking style. Alignment also occurs when a talker is asked to shadow recorded words (e.g., Shockley, Sabadini, & Fowler, 2004). In two experiments, we examined whether alignment could be induced with visual (lipread) speech and with auditory speech. In Experiment 1, we asked subjects to lipread and shadow out loud a model silently uttering words. The results indicate that shadowed utterances sounded more similar to the model’s utterances than did subjects’ nonshadowed read utterances. This suggests that speech alignment can be based on visual speech. In Experiment 2, we tested whether raters could perceive alignment across modalities. Raters were asked to judge the relative similarity between a model’s visual (silent video) utterance and subjects’ audio utterances. The subjects’ shadowed utterances were again judged as more similar to the model’s than were read utterances, suggesting that raters are sensitive to cross-modal similarity between aligned words.     

Use of visual information in speech perception: evidence for a visual rate effect both with and without a McGurk effect

The McGurk effect, where an incongruent visual syllable influences identification of an auditory syllable, does not always occur, suggesting that perceivers sometimes fail to use relevant visual phonetic information. We tested whether another visual phonetic effect, which involves the influence of visual speaking rate on perceived voicing (Green & Miller, 1985), would occur in instances when the McGurk effect does not. In Experiment 1, we established this visual rate effect using auditory and visual stimuli matching in place of articulation, finding a shift in the voicing boundary along an auditory voice-onset-time continuum with fast versus slow visual speech tokens. In Experiment 2, we used auditory and visual stimuli differing in place of articulation and found a shift in the voicing boundary due to visual rate when the McGurk effect occurred and, more critically, when it did not. The latter finding indicates that phonetically relevant visual information is used in speech perception even when the McGurk effect does not occur, suggesting that the incidence of the McGurk effect underestimates the extent of audio-visual integration.     

Phonetic recoding of print and its effect on the detection of concurrent speech in amplitude-modulated noise.

When an amplitude-modulated noise generated from a spoken word is presented simultaneously with the word's printed version, the noise sounds more speechlike. This auditory illusion obtained by Frost, Repp, and Katz (1988) suggests that subjects detect correspondences between speech amplitude envelopes and printed stimuli. The present study investigated whether the speech envelope is assembled from the printed word or whether it is lexically addressed. In two experiments subjects were presented with speech-plus-noise and with noise-only trials, and were required to detect the speech in the noise. The auditory stimuli were accompanied with matching or non-matching Hebrew print, which was unvoweled in Experiment 1 and voweled in Experiment 2. The stimuli of both experiments consisted of high-frequency words, low-frequency words, and non-words. The results demonstrated that matching print caused a strong bias to detect speech in the noise when the stimuli were either high- or low-frequency words, whereas no bias was found for non-words. The bias effect for words or non-words was not affected by spelling to sound regularity; that is, similar effects were obtained in the voweled and the unvoweled conditions. These results suggest that the amplitude envelope of the word is not assembled from the print. Rather, it is addressed directly from the printed word and retrieved from the mental lexicon. Since amplitude envelopes are contingent on detailed phonetic structures, this outcome suggests that representations of words in the mental lexicon are not only phonological but also phonetic in character.     

The effect of psychological stress and expectation on auditory perception: A signal detection analysis

Both psychological stress and predictive signals relating to expected sensory input are believed to influence perception, an influence which, when disrupted, may contribute to the generation of auditory hallucinations. The effect of stress and semantic expectation on auditory perception was therefore examined in healthy participants using an auditory signal detection task requiring the detection of speech from within white noise. Trait anxiety was found to predict the extent to which stress influenced response bias, resulting in more anxious participants adopting a more liberal criterion, and therefore experiencing more false positives, when under stress. While semantic expectation was found to increase sensitivity, its presence also generated a shift in response bias towards reporting a signal, suggesting that the erroneous perception of speech became more likely. These findings provide a potential cognitive mechanism that may explain the impact of stress on hallucination‐proneness, by suggesting that stress has the tendency to alter response bias in highly anxious individuals. These results also provide support for the idea that top‐down processes such as those relating to semantic expectation may contribute to the generation of auditory hallucinations.     

Visual influences on speech perception processes

An experiment is reported, the results of which confirm and extend an earlier observation that visual information for the speaker’s lip movements profoundly modifies the auditorv perception of natural speech by normally hearing subjects. The effect is most pronounced when there is auditory information for a bilabial utterance combined with visual information for a nonlabial utterance. However, the effect is also obtained with the reverse combination, although to a lesser extent. These findings are considered for their relevance to auditory theories of speech perception.     

Audio-visual interactions with intact clearly audible speech

The effects of viewing the face of the talker (visual speech) on the processing of clearly presented intact auditory stimuli were investigated using two measures likely to be sensitive to the articulatory motor actions produced in speaking. The aim of these experiments was to highlight the need for accounts of the effects of audio-visual (AV) speech that explicitly consider the properties of articulated action. The first experiment employed a syllable-monitoring task in which participants were required to monitor for target syllables within foreign carrier phrases. An AV effect was found in that seeing a talker's moving face (moving face condition) assisted in more accurate recognition (hits and correct rejections) of spoken syllables than of auditory-only still face (still face condition) presentations. The second experiment examined processing of spoken phrases by investigating whether an AV effect would be found for estimates of phrase duration. Two effects of seeing the moving face of the talker were found. First, the moving face condition had significantly longer duration estimates than the still face auditory-only condition. Second, estimates of auditory duration made in the moving face condition reliably correlated with the actual durations whereas those made in the still face auditory condition did not. The third experiment was carried out to determine whether the stronger correlation between estimated and actual duration in the moving face condition might have been due to generic properties of AV presentation. Experiment 3 employed the procedures of the second experiment but used stimuli that were not perceived as speech although they possessed the same timing cues as those of the speech stimuli of Experiment 2. It was found that simply presenting both auditory and visual timing information did not result in more reliable duration estimates. Further, when released from the speech context (used in Experiment 2), duration estimates for the auditory-only stimuli were significantly correlated with actual durations. In all, these results demonstrate that visual speech can assist in the analysis of clearly presented auditory stimuli in tasks concerned with information provided by viewing the production of an utterance. We suggest that these findings are consistent with there being a processing link between perception and action such that viewing a talker speaking will activate speech motor schemas in the perceiver.     

Irrelevant Pictures in Visual Working Memory

The aim of the paper is to establish firmly the phenomenon of an irrelevant picture effect and to demonstrate that the phenomenon can be used to investigate the characteristics of the visuo-spatial sketchpad. Experiment 1 introduces the use of dynamic visual noise as an interfering technique. This technique is shown to cause interference with a word list learned under visual mnemonic mediation instructions but to cause no interference when the word list is learned under verbal mediation instructions. Experiment 2 serves both to replicate this selective interference effect and to illustrate how the dynamic visual noise technique can be used to characterize further the nature of interference in the visuo-spatial sketchpad. Experiment 3 confirms the robustness of the dynamic visual noise technique. Additionally, this experiment demonstrates a double dissociation between two types of interference, dynamic visual noise and irrelevant speech, and two types of mediation, rote and visual mnemonic.     

Attentional bias in high- and low-anxious individuals: Evidence for threat-induced effects on engagement and disengagement

Attentional bias to threatening visual stimuli (words or pictures) is commonly present in anxious individuals, but not in non-anxious people. There is evidence to show that attentional bias to threat can be induced in all individuals when threat is imposed by threat not of symbolic nature, but by cues that predict aversive stimulation (loud noise or electric shock). However, it is not known whether attentional bias in such situations is still influenced by individual differences in anxiety. This question was addressed in two experiments using a spatial cuing task in which visual cues predicted the occurrence of an aversive event consisting of a loud human scream. Speeded attentional engagement to threat cues was positively correlated with trait anxiety in Experiment 1. Experiment 2 showed that speeded attentional engagement was present only in participants selected for high anxiety but not in low-anxious participants. In both experiments, slower disengagement from threat cues was found in all participants, irrespective of their trait anxiety levels.     

Attentional bias in high- and low-anxious individuals: evidence for threat-induced effects on engagement and disengagement

Attentional bias to threatening visual stimuli (words or pictures) is commonly present in anxious individuals, but not in non-anxious people. There is evidence to show that attentional bias to threat can be induced in all individuals when threat is imposed by threat not of symbolic nature, but by cues that predict aversive stimulation (loud noise or electric shock). However, it is not known whether attentional bias in such situations is still influenced by individual differences in anxiety. This question was addressed in two experiments using a spatial cuing task in which visual cues predicted the occurrence of an aversive event consisting of a loud human scream. Speeded attentional engagement to threat cues was positively correlated with trait anxiety in Experiment 1. Experiment 2 showed that speeded attentional engagement was present only in participants selected for high anxiety but not in low-anxious participants. In both experiments, slower disengagement from threat cues was found in all participants, irrespective of their trait anxiety levels.     

Hearing speech in noise: seeing a loud talker is better

Seeing the talker improves the intelligibility of speech degraded by noise (a visual speech benefit). Given that talkers exaggerate spoken articulation in noise, this set of two experiments examined whether the visual speech benefit was greater for speech produced in noise than in quiet. We first examined the extent to which spoken articulation was exaggerated in noise by measuring the motion of face markers as four people uttered 10 sentences either in quiet or in babble-speech noise (these renditions were also filmed). The tracking results showed that articulated motion in speech produced in noise was greater than that produced in quiet and was more highly correlated with speech acoustics. Speech intelligibility was tested in a second experiment using a speech-perception-in-noise task under auditory-visual and auditory-only conditions. The results showed that the visual speech benefit was greater for speech recorded in noise than for speech recorded in quiet. Furthermore, the amount of articulatory movement was related to performance on the perception task, indicating that the enhanced gestures made when speaking in noise function to make speech more intelligible.     

Error biases in inner and overt speech: evidence from tongue twisters

To compare the properties of inner and overt speech, Oppenheim and Dell (2008) counted participants' self-reported speech errors when reciting tongue twisters either overtly or silently and found a bias toward substituting phonemes that resulted in words in both conditions, but a bias toward substituting similar phonemes only when speech was overt. Here, we report 3 experiments revisiting their conclusion that inner speech remains underspecified at the subphonemic level, which they simulated within an activation-feedback framework. In 2 experiments, participants recited tongue twisters that could result in the errorful substitutions of similar or dissimilar phonemes to form real words or nonwords. Both experiments included an auditory masking condition, to gauge the possible impact of loss of auditory feedback on the accuracy of self-reporting of speech errors. In Experiment 1, the stimuli were composed entirely from real words, whereas, in Experiment 2, half the tokens used were nonwords. Although masking did not have any effects, participants were more likely to report substitutions of similar phonemes in both experiments, in inner as well as overt speech. This pattern of results was confirmed in a 3rd experiment using the real-word materials from Oppenheim and Dell (in press). In addition to these findings, a lexical bias effect found in Experiments 1 and 3 disappeared in Experiment 2. Our findings support a view in which plans for inner speech are indeed specified at the feature level, even when there is no intention to articulate words overtly, and in which editing of the plan for errors is implicated. (PsycINFO Database Record (c) 2010 APA, all rights reserved).     

Privileged Access by Irrelevant Speech to Short-term Memory: The Role of Changing State

Memory for visually presented items is impaired by speech that is played as an irrelevant background. The paper presents the view that changing state of the auditory material is an important prerequisite for this disruption. Four experiments studied the effects of sounds varying in complexity in an attempt to establish which features of changing state in the auditory signal lead to diminished recall. Simple unvarying or repetitive speech sounds were not sufficient to induce the irrelevant speech effect (Experiment 1): in addition, simple analogues of speech, possessing regular or irregular envelopes and using a range of carriers, failed to imitate the action of speech (Experiment 2). Variability of between-utterance phonology in the irrelevant stream (Experiment 3) emerged as a crucial factor. Moreover, predictability of the syllable sequence did not reduce the degree of disruption (Experiment 4) suggesting that supra-syllabic characteristics of the speech are of little importance. The results broadly support the idea that disruption of short-term memory only occurs when the speech stream changes in state. It is argued that disruption occurs in memory when cues to serial order based on phonological representations of heard material interfere with the phonological codes of visual origin. It is suggested that cues to changing state of the speech input contaminate those associated with items of visual origin, which are already in a phonological store.     

Amodal processing of visual speech as revealed by priming

This study investigated the linguistic processing of visual speech (video of a talker's utterance without audio) by determining if such has the capacity to prime subsequently presented word and nonword targets. The priming procedure is well suited for the investigation of whether speech perception is amodal since visual speech primes can be used with targets presented in different modalities. To this end, a series of priming experiments were conducted using several tasks. It was found that visually spoken words (for which overt identification was poor) acted as reliable primes for repeated target words in the naming, written and auditory lexical decision tasks. These visual speech primes did not produce associative or reliable form priming. The lack of form priming suggests that the repetition priming effect was constrained by lexical level processes. That priming found in all tasks is consistent with the view that similar processes operate in both visual and auditory speech processing.     

Incrementality in naming and reading complex numerals: evidence from eyetracking

Individuals speak incrementally when they interleave planning and articulation. Eyetracking, along with the measurement of speech onset latencies, can be used to gain more insight into the degree of incrementality adopted by speakers. In the current article, two eyetracking experiments are reported in which pairs of complex numerals were named (arabic format, Experiment 1) or read aloud (alphabetic format, Experiment 2) as house numbers and as clock times. We examined whether the degree of incrementality is differentially influenced by the production task (naming vs. reading) and mode (house numbers vs. clock time expressions), by comparing gaze durations and speech onset latencies. In both tasks and modes, dissociations were obtained between speech onset latencies (reflecting articulation) and gaze durations (reflecting planning), indicating incrementality. Furthermore, whereas none of the factors that determined gaze durations were reflected in the reading and naming latencies for the house numbers, the dissociation between gaze durations and response latencies for the clock times concerned mainly numeral length in both tasks. These results suggest that the degree of incrementality is influenced by the type of utterance (house number vs. clock time) rather than by task (reading vs. naming). The results highlight the importance of the utterance structure in determining the degree of incrementality.     

Incrementality in naming and reading complex numerals: evidence from eyetracking

Individuals speak incrementally when they interleave planning and articulation. Eyetracking, along with the measurement of speech onset latencies, can be used to gain more insight into the degree of incrementality adopted by speakers. In the current article, two eyetracking experiments are reported in which pairs of complex numerals were named (arabic format, Experiment 1) or read aloud (alphabetic format, Experiment 2) as house numbers and as clock times. We examined whether the degree of incrementality is differentially influenced by the production task (naming vs. reading) and mode (house numbers vs. clock time expressions), by comparing gaze durations and speech onset latencies. In both tasks and modes, dissociations were obtained between speech onset latencies (reflecting articulation) and gaze durations (reflecting planning), indicating incrementality. Furthermore, whereas none of the factors that determined gaze durations were reflected in the reading and naming latencies for the house numbers, the dissociation between gaze durations and response latencies for the clock times concerned mainly numeral length in both tasks. These results suggest that the degree of incrementality is influenced by the type of utterance (house number vs. clock time) rather than by task (reading vs. naming). The results highlight the importance of the utterance structure in determining the degree of incrementality.     

Use of speech-modulated noise adds strong “bottom-up” cues for phonemic restoration

When deleted segments of speech are replaced by extraneous sounds rather than silence, the missing speech fragments may be perceptually restored and intelligibility improved. This phonemic restoration (PhR) effect has been used to measure various aspects of speech processing, with deleted portions of speech typically being replaced by stochastic noise. However, several recent studies of PhR have used speech-modulated noise, which may provide amplitude-envelope cues concerning the replaced speech. The present study compared the effects upon intelligibility of replacing regularly spaced portions of speech with stochastic (white) noise versus speech-modulated noise. In Experiment 1, filling periodic gaps in sentences with noise modulated by the amplitude envelope of the deleted speech fragments produced twice the intelligibility increase obtained with interpolated stochastic noise. Moreover, when lists of isolated monosyllables were interrupted in Experiment 2, interpolation of speech-modulated noise increased intelligibility whereas stochastic noise reduced intelligibility. The augmentation of PhR produced by modulated noise appeared without practice, suggesting that speech processing normally involves not only a narrowband analysis of spectral information but also a wideband integration of amplitude levels across critical bands. This is of considerable theoretical interest, but it also suggests that since PhRs produced by speech-modulated noise utilize potent bottom-up cues provided by the noise, they differ from the PhRs produced by extraneous sounds, such as coughs and stochastic noise.     

Effects of background music on phonological short-term memory

Immediate memory for visually presented verbal material is disrupted by concurrent speech, even when the speech is unattended and in a foreign language. Unattended noise does not produce a reliable decrement. These results have been interpreted in terms of a phonological short-term store that excludes non-speechlike sounds. The characteristics of this exclusion process were explored by studying the effects of music on the serial recall of sequences of nine digits presented visually. Experiment 1 compared the effects of unattended vocal or instrumental music with quiet and showed that both types of music disrupted STM performance, with vocal music being more disruptive than instrumental music. Experiment 2 attempted to replicate this result using more highly trained subjects. Vocal music caused significantly more disruption than instrumental music, which was not significantly worse than the silent control condition. Experiment 3 compared instrumental music with unattended speech and with noise modulated in amplitude, the degree of modulation being the same as in speech. The results showed that the noise condition did not differ from silence; both of these proved less disruptive than instrumental music, which was in turn less disruptive than the unattended speech condition. Theoretical interpretation of these results and their potential practical implications for the disruption of cognitive performance by background music are discussed.