Temporal integration of speech prosody is shaped by language experience: an fMRI study

Differences in hemispheric functions underlying speech perception may be related to the size of temporal integration windows over which prosodic features (e.g., pitch) span in the speech signal. Chinese tone and intonation, both signaled by variations in pitch contours, span over shorter (local) and longer (global) temporal domains, respectively. This cross-linguistic (Chinese and English) study uses functional magnetic resonance imaging to show that pitch contours associated with tones are processed in the left hemisphere by Chinese listeners only, whereas pitch contours associated with intonation are processed predominantly in the right hemisphere. These findings argue against the view that all aspects of speech prosody are lateralized to the right hemisphere, and promote the idea that varying-sized temporal integration windows reflect a neurobiological adaptation to meet the 'prosodic needs' of a particular language.     

Probabilistic constraint satisfaction at the lexical/phonetic interface: evidence for gradient effects of within-category VOT on lexical access

Research in speech perception has been dominated by a search for invariant properties of the signal that correlate with lexical and sublexical categories. We argue that this search for invariance has led researchers to ignore the perceptual consequences of systematic variation within such categories and that sensitivity to this variation may provide an important source of information for integrating information over time in speech perception. Data from a study manipulating VOT continua in words using an eye-movement paradigm indicate that lexical access shows graded sensitivity to within-category variation in VOT and that this sensitivity has a duration sufficient to be useful for information integration. These data support a model in which the perceptual system integrates information from multiple sources and from the surrounding temporal context using probabilistic cue-weighting mechanisms.     

Temporal processing capabilities in repetition conduction aphasia

This study investigates the temporal resolution capacities of the central-auditory system in a subject (NP) suffering from repetition conduction aphasia. More specifically, the patient was asked to detect brief gaps between two stretches of broadband noise (gap detection task) and to evaluate the duration of two biphasic (WN-3) continuous noise elements, starting with white noise (WN) followed by 3 kHz bandpass-filtered noise (duration discrimination task). During the gap detection task, the two portions of each stimulus were either identical (“intra-channel condition”) or differed (“inter-channel condition”) in the spectral characteristics of the leading and trailing acoustic segments. NP did not exhibit any deficits in the intra-channel condition of the gap detection task, indicating intact auditory temporal resolution across intervals of 1–3 ms. By contrast, the inter-channel condition yielded increased threshold values. Based upon the “multiple-looks” model of central-auditory processing, this profile points at a defective integration window operating across a few tens of milliseconds – a temporal range associated with critical features of the acoustic speech signal such as voice onset time and formant transitions. Additionally, NP was found impaired during a duration discrimination task addressing longer integration windows (ca. 150 ms). Concerning speech, this latter time domain approximately corresponds to the duration of stationary segmental units such as fricatives and long vowels. On the basis of our results we suggest, that the patient’s auditory timing deficits in non-speech tasks may account, at least partially, for his impairments in speech processing.     

Temporal binding of auditory spatial information across dynamic binaural events

Simultaneity judgments were used to measure temporal binding windows (TBW) for brief binaural events (changes in interaural time and/or level differences [ITD and ILD]) and test the hypothesis that ITD and ILD contribute to perception via separate sensory dimensions subject to binding via slow (100+ ms)—presumably cortical—mechanisms as in multisensory TBW. Stimuli were continuous low-frequency noises that included two brief shifts of either type (ITD or ILD), both of which are heard as lateral position changes. TBW for judgments within a single cue dimension were narrower for ITD (mean = 444 ms) than ILD (807 ms). TBW for judgments across cue dimensions (i.e., one ITD shift and one ILD shift) were similar to within-cue ILD (778 ms). The results contradict the original hypothesis, in that cross-cue comparisons were no slower than within-cue ILD comparisons. Rather, the wide TBW values—consistent with previous estimates of multisensory TBW—suggest slow integrative processing for both types of judgments. Narrower TBW for ITD than ILD judgments suggests important cue-specific differences in the neural mechanisms or the perceptual correlates of integration across binaural-cue dimensions.     

What Are You Waiting For? Real‐Time Integration of Cues for Fricatives Suggests Encapsulated Auditory Memory

Speech unfolds over time, and the cues for even a single phoneme are rarely available simultaneously. Consequently, to recognize a single phoneme, listeners must integrate material over several hundred milliseconds. Prior work contrasts two accounts: (a) a memory buffer account in which listeners accumulate auditory information in memory and only access higher level representations (i.e., lexical representations) when sufficient information has arrived; and (b) an immediate integration scheme in which lexical representations can be partially activated on the basis of early cues and then updated when more information arises. These studies have uniformly shown evidence for immediate integration for a variety of phonetic distinctions. We attempted to extend this to fricatives, a class of speech sounds which requires not only temporal integration of asynchronous cues (the frication, followed by the formant transitions 150–350 ms later), but also integration across different frequency bands and compensation for contextual factors like coarticulation. Eye movements in the visual world paradigm showed clear evidence for a memory buffer. Results were replicated in five experiments, ruling out methodological factors and tying the release of the buffer to the onset of the vowel. These findings support a general auditory account for speech by suggesting that the acoustic nature of particular speech sounds may have large effects on how they are processed. It also has major implications for theories of auditory and speech perception by raising the possibility of an encapsulated memory buffer in early auditory processing.     

Temporal predictive mechanisms modulate motor reaction time during initiation and inhibition of speech and hand movement

Skilled movement is mediated by motor commands executed with extremely fine temporal precision. The question of how the brain incorporates temporal information to perform motor actions has remained unanswered. This study investigated the effect of stimulus temporal predictability on response timing of speech and hand movement. Subjects performed a randomized vowel vocalization or button press task in two counterbalanced blocks in response to temporally-predictable and unpredictable visual cues. Results indicated that speech and hand reaction time was decreased for predictable compared with unpredictable stimuli. This finding suggests that a temporal predictive code is established to capture temporal dynamics of sensory cues in order to produce faster movements in responses to predictable stimuli. In addition, results revealed a main effect of modality, indicating faster hand movement compared with speech. We suggest that this effect is accounted for by the inherent complexity of speech production compared with hand movement. Lastly, we found that movement inhibition was faster than initiation for both hand and speech, suggesting that movement initiation requires a longer processing time to coordinate activities across multiple regions in the brain. These findings provide new insights into the mechanisms of temporal information processing during initiation and inhibition of speech and hand movement.     

Temporal structure coding with and without awareness

In order to interpret a constantly changing environment, visual events far apart in space and time must be integrated into a unified percept. While spatial properties of invisible signals are known to be encoded without awareness, the fate of temporal properties remains largely unknown. Here, we probed temporal integration for two distinct motion stimuli that were either visible or rendered invisible using continuous flash suppression. We found that when invisible, both the direction of apparent motion and the gender of point-light walkers were processed only when defined across short time periods (i.e., respectively 100 ms and 1000 ms). This limitation was not observed under full visibility. These similar findings at two different hierarchical levels of processing suggest that temporal integration windows shrink in the absence of perceptual awareness. We discuss this phenomenon as a key prediction of the global neuronal workspace and the information integration theories of consciousness.     

时间分层加工研究述评

为了探讨数秒内不同层级时间加工的特性,研究者分别从“时间信息加工”和“信息加工的计时特性”角度开展了一系列研究。Münsterberg (1889)、Michon(1985) 、Lewis 和 Miall(2003) 及Vierodt(1868)从前一角度,分别指出1/3秒、1/2秒、1秒及3秒可能是数秒以内时距加工机制的分界点,分界点以下与以上的加工机制存在差异。P?ppel(1997, 2009)则从后一角度指出限制信息加工过程的两类时间窗,一类时间窗是以20-60毫秒振荡周期运行的高频系统,属于初级整合单元;另一类时间窗主要是处理2-3秒以内事件系列的低频系统,属于高级整合单元。前一类时间窗可以为信息加工整合基本的心理事件,后一类时间窗则是把2-3秒内的心理事件整合为基本知觉单元。基于以往研究的剖析,我们认为1/3秒、1/2秒及1秒等分界点的真伪性尚需进一步验证,并进一步假设40毫秒以内时间不能觉察为时距;40毫秒至3秒之间,随着长度增加,自动化加工减弱,控制性加工增强;3秒以上主要为控制性加工,涉及记忆过程。     

Functional neuroanatomy of gesture–speech integration in children varies with individual differences in gesture processing

Gesture is an integral part of children's communicative repertoire. However, little is known about the neurobiology of speech and gesture integration in the developing brain. We investigated how 8‐ to 10‐year‐old children processed gesture that was essential to understanding a set of narratives. We asked whether the functional neuroanatomy of gesture–speech integration varies as a function of (1) the content of speech, and/or (2) individual differences in how gesture is processed. When gestures provided missing information not present in the speech (i.e., disambiguating gesture; e.g., “pet” + flapping palms = bird), the presence of gesture led to increased activity in inferior frontal gyri, the right middle temporal gyrus, and the left superior temporal gyrus, compared to when gesture provided redundant information (i.e., reinforcing gesture; e.g., “bird” + flapping palms = bird). This pattern of activation was found only in children who were able to successfully integrate gesture and speech behaviorally, as indicated by their performance on post‐test story comprehension questions. Children who did not glean meaning from gesture did not show differential activation across the two conditions. Our results suggest that the brain activation pattern for gesture–speech integration in children overlaps with—but is broader than—the pattern in adults performing the same task. Overall, our results provide a possible neurobiological mechanism that could underlie children's increasing ability to integrate gesture and speech over childhood, and account for individual differences in that integration.     

Interplay between acoustic/phonetic and semantic processes during spoken sentence comprehension: an ERP study

When listening to speech in everyday-life situations, our cognitive system must often cope with signal instabilities such as sudden breaks, mispronunciations, interfering noises or reverberations potentially causing disruptions at the acoustic/phonetic interface and preventing efficient lexical access and semantic integration. The physiological mechanisms allowing listeners to react instantaneously to such fast and unexpected perturbations in order to maintain intelligibility of the delivered message are still partly unknown. The present electroencephalography (EEG) study aimed at investigating the cortical responses to real-time detection of a sudden acoustic/phonetic change occurring in connected speech and how these mechanisms interfere with semantic integration. Participants listened to sentences in which final words could contain signal reversals along the temporal dimension (time-reversed speech) of varying durations and could have either a low- or high-cloze probability within sentence context. Results revealed that early detection of the acoustic/phonetic change elicited a fronto-central negativity shortly after the onset of the manipulation that matched the spatio-temporal features of the Mismatch Negativity (MMN) recorded in the same participants during an oddball paradigm. Time reversal also affected late event-related potentials (ERPs) reflecting semantic expectancies (N400) differently when words were predictable or not from the sentence context. These findings are discussed in the context of brain signatures to transient acoustic/phonetic variations in speech. They contribute to a better understanding of natural speech comprehension as they show that acoustic/phonetic information and semantic knowledge strongly interact under adverse conditions.     

Estimating the growth of internal evidence guiding perceptual decisions

Perceptual decision-making is thought to involve a gradual accrual of noisy evidence. Temporal integration of the evidence reduces the relative contribution of dynamic internal noise to the decision variable, thereby boosting its signal-to-noise ratio. We aimed to estimate the internal evidence guiding perceptual decisions over time, using a novel combination of external noise and the response signal methods. Observers performed orientation discrimination of patterns presented in external noise. We varied the contrast of the patterns and the delay at which observers were forced to signal their decision. Each test stimulus (patterns and noise sample) was presented twice. Across two experiments we varied the availability of the visual stimulus for processing. Observer model analyses of discrimination accuracy and response consistency to two passes of the same stimulus, suggested that there was very little growth in the internal evidence. The improvement in accuracy over time characterised by the speed-accuracy trade-off function predominantly reflected a decreasing proportion of non-visual decisions, or pure guesses. There was no advantage to having the visual patterns visible for longer than 80 ms, indicating that only the visual information in a short window after display onset was used to drive the decisions. The remarkable constancy of the internal evidence over time suggests that temporal integration of the sensory information was very limited. Alternatively, more extended integration of the evidence from memory could have taken place, provided that the dominant source of internal noise limiting performance occurs between-trials, which cannot be reduced by prolonged evidence integration.     

The role of temporal speech cues in facilitating the fluency of adults who stutter

PurposeAdults who stutter speak more fluently during choral speech contexts than they do during solo speech contexts. The underlying mechanisms for this effect remain unclear, however. In this study, we examined the extent to which the choral speech effect depended on presentation of intact temporal speech cues. We also examined whether speakers who stutter followed choral signals more closely than typical speakers did.Method8 adults who stuttered and 8 adults who did not stutter read 60 sentences aloud during a solo speaking condition and three choral speaking conditions (240 total sentences), two of which featured either temporally altered or indeterminate word duration patterns. Effects of these manipulations on speech fluency, rate, and temporal entrainment with the choral speech signal were assessed.ResultsAdults who stutter spoke more fluently in all choral speaking conditions than they did when speaking solo. They also spoke slower and exhibited closer temporal entrainment with the choral signal during the mid- to late-stages of sentence production than the adults who did not stutter. Both groups entrained more closely with unaltered choral signals than they did with altered choral signals.ConclusionsFindings suggest that adults who stutter make greater use of speech-related information in choral signals when talking than adults with typical fluency do. The presence of fluency facilitation during temporally altered choral speech and conversation babble, however, suggests that temporal/gestural cueing alone cannot account for fluency facilitation in speakers who stutter. Other potential fluency enhancing mechanisms are discussed.Educational Objectives: The reader will be able to (a) summarize competing views on stuttering as a speech timing disorder, (b) describe the extent to which adults who stutter depend on an accurate rendering of temporal information in order to benefit from choral speech, and (c) discuss possible explanations for fluency facilitation in the presence of inaccurate or indeterminate temporal cues.     

The role of audiovisual asynchrony in person recognition

Audiovisual integration (AVI) has been demonstrated to play a major role in speech comprehension. Previous research suggests that AVI in speech comprehension tolerates a temporal window of audiovisual asynchrony. However, few studies have employed audiovisual presentation to investigate AVI in person recognition. Here, participants completed an audiovisual voice familiarity task in which the synchrony of the auditory and visual stimuli was manipulated, and in which visual speaker identity could be corresponding or noncorresponding to the voice. Recognition of personally familiar voices systematically improved when corresponding visual speakers were presented near synchrony or with slight auditory lag. Moreover, when faces of different familiarity were presented with a voice, recognition accuracy suffered at near synchrony to slight auditory lag only. These results provide the first evidence for a temporal window for AVI in person recognition between approximately 100 ms auditory lead and 300 ms auditory lag.     

Hearing by eye

Recent work on integration of auditory and visual information during speech perception has indicated that adults are surprisingly good at, and rely extensively on, lip reading. The conceptual status of lip read information is of interest: such information is at the same time both visual and phonological. Three experiments investigated the nature of short term coding of lip read information in hearing subjects. The first experiment used asynchronous visual and auditory information and showed that a subject's ability to repeat words, when heard speech lagged lip movements, was unaffected by the lag duration, both quantitatively and qualitatively. This suggests that lip read information is immediately recoded into a durable code. An experiment on serial recall of lip read items showed a serial position curve containing a recency effect (characteristic of auditory but not visual input). It was then shown that an auditory suffix diminishes the recency effect obtained with lip read stimuli. These results are consistent with the hypothesis that seen speech, that is not heard, is encoded into a durable code which has some shared properties with heard speech. The results of the serial recall experiments are inconsistent with interpretations of the recency and suffix effects in terms of precategorical acoustic storage, for they demonstrate that recency and suffix effects can be supra-modal.     

A sensorimotor theory of temporal tracking and beat induction

In this paper, we develop a theory of the neurobiological basis of temporal tracking and beat induction as a form of sensory-guided action. We propose three principal components for the neurological architecture of temporal tracking: (1) the central auditory system, which represents the temporal information in the input signal in the form of a modulation power spectrum; (2) the musculoskeletal system, which carries out the action and (3) a controller, in the form of a parieto-cerebellar-frontal loop, which carries out the synchronisation between input and output by means of an internal model of the musculoskeletal dynamics. The theory is implemented in the form of a computational algorithm which takes sound samples as input and synchronises a simple linear mass-spring-damper system to simulate audio-motor synchronisation. The model may be applied to both the tracking of isochronous click sequences and beat induction in rhythmic music or speech, and also accounts for the approximate Weberian property of timing.     

时间信息加工与信息加工时间特性双视角下的重要时间参数及其证据

时间既是人类信息加工的对象, 也是(非时间)信息加工的制约因素。数十毫秒至数秒之间的时间加工与人类日常生活关联紧密, 譬如主观计时、演奏及言语等活动。根据以往文献分析可知, 在该时间区域内, 20~ 60 ms、1/3~1 s、2~3 s是研究者关注的重要时间参数, 但是支持这些参数的证据尚存分歧。首先从“时间信息加工”和“信息加工的时间特性”的视角介绍时间参数的基本观点及其提出背景, 然后基于“时间信息加工”视角从行为学研究、脑损伤研究、神经药理学研究, 脑电研究、脑成像研究、经颅磁刺激研究、经颅直流电刺激研究等领域介评了1/3~1 s和2~3 s分界区域的证据, 接着基于“信息加工的时间特性”视角从时序知觉阈限研究、感觉运动同步研究、主观节奏研究、言语行为研究、知觉逆转研究、返回抑制研究及失匹配负波研究等领域介评了20~60 ms和2~3 s时间窗口的证据。未来研究既要注意构建基于分界区域与时间窗口的更强解释力的理论假说, 也要厘清分界区域与时间窗口的联系与区别。     

ERP evidence on the interaction between information structure and emotional salience of words

Both emotional words and words focused by information structure can capture attention. This study examined the interplay between emotional salience and information structure in modulating attentional resources in the service of integrating emotional words into sentence context. Event-related potentials (ERPs) to affectively negative, neutral, and positive words, which were either focused or nonfocused in question_–answer pairs, were evaluated during sentence comprehension. The results revealed an early negative effect (90–200 ms), a P2 effect, as well as an effect in the N400 time window, for both emotional salience and information structure. Moreover, an interaction between emotional salience and information structure occurred within the N400 time window over right posterior electrodes, showing that information structure influences the semantic integration only for neutral words, but not for emotional words. This might reflect the fact that the linguistic salience of emotional words can override the effect of information structure on the integration of words into context. The interaction provides evidence for attention–emotion interactions at a later stage of processing. In addition, the absence of interaction in the early time window suggests that the processing of emotional information is highly automatic and independent of context. The results suggest independent attention capture systems of emotional salience and information structure at the early stage but an interaction between them at a later stage, during the semantic integration of words.     

How the visual system detects changes in the direction of moving targets

To determine how the visual system represents information about change in target direction, we studied the detection of such change under conditions of varying stimulus certainty. Target direction was either held constant over trials or was allowed to vary randomly. When target direction was constant the observer could be certain about that stimulus characteristic; randomizing the target direction rendered the observer uncertain. We measured response times (RTs) to changes in target direction following initial trajectories of varying time and distance. In different conditions, the observer was uncertain about either the direction of the initial trajectory, or the direction of change or both. With brief initial trajectories in random directions, uncertainty about initial direction elevated RTs by 50 ms or more. When the initial trajectories were at least 500 ms, this directional uncertainty ceased to affect RTs; then, only uncertainty about the direction of change affected RTs. We discuss the implications of these results for (i) schemes by which the visual system might code directional change; (ii) the visual integration time for directional information; and (iii) adaptational processes in motion perception.     

Adaptation of the relative onset time of two-component tones

The results of three selective adaptation experiments employing nonspeech signals that differed in temporal onset are reported. In one experiment, adaptation effects were observed when both the adapting and test stimuli were selected from the same nonspeech test continuum. This result was interpreted as evidence for selective processing of temporal order information in nonspeech signals. Two additional experiments tested for the presence of cross-series adaptation effects from speech to nonspeech and then from nonspeech to speech. Both experiments failed to show any evidence of cross-series adaptation effects, implying a possible dissociation between perceptual classes of speech and nonspeech signals in processing temporal order information. Despite the absence of cross-series effects, it is argued that the ability of the auditory system to process temporal order information may still provide a possible basis for explaining the perception of voicing in stops that differ in VOT. The results of the present experiments, taken together with earlier findings on the perception of temporal onset in nonspeech signals, were viewed as an example of the way spoken language has exploited the basic sensory capabilities of the auditory system to signal phonetic differences.     

The influence of selective attention to auditory and visual speech on the integration of audiovisual speech information

Conflicting visual speech information can influence the perception of acoustic speech, causing an illusory percept of a sound not present in the actual acoustic speech (the McGurk effect). We examined whether participants can voluntarily selectively attend to either the auditory or visual modality by instructing participants to pay attention to the information in one modality and to ignore competing information from the other modality. We also examined how performance under these instructions was affected by weakening the influence of the visual information by manipulating the temporal offset between the audio and video channels (experiment 1), and the spatial frequency information present in the video (experiment 2). Gaze behaviour was also monitored to examine whether attentional instructions influenced the gathering of visual information. While task instructions did have an influence on the observed integration of auditory and visual speech information, participants were unable to completely ignore conflicting information, particularly information from the visual stream. Manipulating temporal offset had a more pronounced interaction with task instructions than manipulating the amount of visual information. Participants' gaze behaviour suggests that the attended modality influences the gathering of visual information in audiovisual speech perception.