It's the way that you, er, say it: hesitations in speech affect language comprehension

Everyday speech is littered with disfluency, often correlated with the production of less predictable words (e.g., Beattie & Butterworth [Beattie, G., & Butterworth, B. (1979). Contextual probability and word frequency as determinants of pauses in spontaneous speech. Language and Speech, 22, 201-211.]). But what are the effects of disfluency on listeners? In an ERP experiment which compared fluent to disfluent utterances, we established an N400 effect for unpredictable compared to predictable words. This effect, reflecting the difference in ease of integrating words into their contexts, was reduced in cases where the target words were preceded by a hesitation marked by the word er. Moreover, a subsequent recognition memory test showed that words preceded by disfluency were more likely to be remembered. The study demonstrates that hesitation affects the way in which listeners process spoken language, and that these changes are associated with longer-term consequences for the representation of the message.     

Language comprehension vs. language production: age effects on fMRI activation

Normal language acquisition is a process that unfolds with amazing speed primarily in the first years of life. However, the refinement of linguistic proficiency is an ongoing process, extending well into childhood and adolescence. An increase in lateralization and a more focussed productive language network have been suggested to be the neural correlates of this process. However, the processes underlying the refinement of language comprehension are less clear. Using a language comprehension (Beep Stories) and a language production (Vowel Identification) task in fMRI, we studied language representation and lateralization in 36 children, adolescents, and young adults (age 6-24 years). For the language comprehension network, we found a more focal activation with age in the bilateral superior temporal gyri. No significant increase of lateralization with age could be observed, so the neural basis of language comprehension as assessed with the Beep Stories task seems to be established in a bilateral network by late childhood. For the productive network, however, we could confirm an increase with age both in focus and lateralization. Only in the language comprehension task did verbal IQ correlate with lateralization, with higher verbal IQ being associated with more right-hemispheric involvement. In some subjects (24%), language comprehension and language production were lateralized to opposite hemispheres.     

Time in language: event duration in language comprehension

This work investigates how we process and represent event duration in on-line language comprehension. Specifically, it examines how events of different duration are processed and what type of knowledge underlies their representations. Studies 1–4 examined verbs and phrases in different contexts. They showed that durative events took longer to process than non-durative events and that the duration attributed to the stimulus events correlated with on-line processing times. Studies 5 and 6 indicated that durative events occur in semantically more diverse contexts and elicit semantically more diverse associations than non-durative events. Semantic and contextual diversity also correlated with attributed durations and processing times. Results indicate that (a) event-specific durations are computed on-line from multiple unfolding cues, (b) processing cost and duration representations emerge from semantic and contextual diversity reflecting our experience, and (c) key components of duration representations may be situation-specific knowledge of causal and contingency relations between events.     

The effects of late acquisition of L2 and the consequences of immigration on L1 for semantic and morpho-syntactic language aspects

It has been hypothesized that some aspects of a second language (L2) might be learned easier than others if a language is learned late. On the other hand, non-use might result in a loss of language skills in one's native, i.e. one's first language (L1) (language attrition). To study which, if any, aspects of language are affected by either late acquisition or non-use, long-term German immigrants to the US and English native speakers who are long-term immigrants to Germany as well as two additional control groups of native German speakers were tested with an auditory semantic and morpho-syntactic priming paradigm. German adjectives correctly or incorrectly inflected for gender and semantically associated or not associated with the target noun served as primes. Participants made a lexical decision on the target word. All groups of native German speakers gained from semantically and morpho-syntactically congruent primes. Evidence for language attrition was neither found for semantic nor morpho-syntactic priming effects in the German immigrants. In contrast, English native speakers did not gain from a morpho-syntactic congruent prime, whereas semantic priming effects were similar as for the remaining groups. The present data suggest that the full acquisition of at least some syntactic functions may be restricted to limited periods in life while semantic and morpho-syntactic functions seem to be relatively inured to loss due to non-use.     

Effects of language comprehension on visual processing - MEG dissociates early perceptual and late N400 effects

We investigated whether and when information conveyed by spoken language impacts on the processing of visually presented objects. In contrast to traditional views, grounded-cognition posits direct links between language comprehension and perceptual processing. We used a magnetoencephalographic cross-modal priming paradigm to disentangle these views. In a sentence-picture verification task, pictures (e.g. of a flying duck) were paired with three sentence conditions: A feature-matching sentence about a duck in the air, a feature-mismatching sentence about a duck in a lake, and an unrelated sentence. Brain responses to pictures showed enhanced activity in the N400 time-window for the unrelated compared to both related conditions in the left temporal lobe. The M1 time-window revealed more activation for the feature-matching than for the other two conditions in the occipital cortex. These dissociable effects on early visual processing and semantic integration support models in which language comprehension engages two complementary systems, a perceptual and an abstract one.     

On the universality of language comprehension strategies: evidence from Turkish

A fundamental question in psycholinguistic research concerns the universality of comprehension strategies. We investigated this issue by examining the so-called "subject preference" in Turkish, a language which allows for a natural (unmarked) object reading of an initial ambiguous argument. Using event-related brain potentials (ERPs), we observed increased processing difficulty in the form of a broadly distributed positivity when an initial ambiguous argument was disambiguated towards an object reading. This effect was independent of the animacy (i.e. semantic subject prototypicality) of the ambiguous argument. Our results therefore speak in favour of a universal tendency to interpret the first argument encountered as the "subject" of the clause, even in languages providing no obvious structural motivation for such a strategy. However, we argue that the underlying explanation for this preference must be modified in accordance with cross-linguistic considerations.     

On the parity of structural persistence in language production and comprehension

Structural priming creates structural persistence. That is, differences in experience with syntax can change subsequent language performance, and the changes can be observed in both language production and comprehension. However, the effects in comprehension and production appear to differ. In comprehension, persistence is typically found when the verbs are the same in primes and targets; in production, persistence occurs without verb overlap. The contrast suggests a theoretically important hypothesis: parsing in comprehension is lexically driven while formulation in production is structurally driven. A major weakness in this hypothesis about comprehension-production differences is that its empirical motivation rests on the outcomes of experiments in which the priming manipulations differ, the primed sentence structures differ, and the measures of priming differ. To sharpen the comparison, we examined structural persistence with and without verb overlap in both reading comprehension and spoken production, using the same prime presentation procedure, the same syntactic structures, the same sentences, and the same participants. These methods yielded abstract structural persistence in comprehension as well as production. A measure of the strength of persistence revealed significant effects of priming and verb overlap without significant comprehension—production differences. This argues for uniformity in the structural mechanisms of language processing.     

The impact of early maternal verbal scaffolding and child language abilities on later decoding and reading comprehension skills

This study examined, via structural equation modeling, early predictors of children's 8 year reading decoding and 10 year comprehension at later school age. Maternal verbal scaffolding indirectly influenced both decoding and comprehension, through its support of children's language abilities at 3 and 4 years of age. Additionally, there was a trend for a direct effect of 4 year child language on reading comprehension at 10 years. As maternal verbal scaffolding was assessed during every day routines, this suggests that rich language input in the broad social context of the home promotes language and, in turn, later reading skills. Given that maternal verbal scaffolding can be enhanced through interventions, these results have important educational implications.     

Not all errors are created equal: metacognition and changing answers on multiple-choice tests.

Two experiments investigated the role of metacognition in changing answers to multiple-choice, general-knowledge questions. Both experiments revealed qualitatively different errors produced by speeded responding versus confusability amongst the alternatives; revision completely corrected the former, but had no effect on the latter. Experiment 2 also demonstrated that a pretest, designed to make participants' actual experience with answer changing either positive or negative, affected the tendency to correct errors. However, this effect was not apparent in the proportion of correct responses; it was only discovered when the metacognitive component to answer changing was isolated with a Type 2 signal-detection measure of discrimination. Overall, the results suggest that future research on answer changing should more closely consider the metacognitive factors underlying answer changing, using Type 2 signal-detection theory to isolate these aspects of performance.     

Dissociable patterns of brain activity during comprehension of rapid and syntactically complex speech: evidence from fMRI

Sentence comprehension is a complex task that involves both language-specific processing components and general cognitive resources. Comprehension can be made more difficult by increasing the syntactic complexity or the presentation rate of a sentence, but it is unclear whether the same neural mechanism underlies both of these effects. In the current study, we used event-related functional magnetic resonance imaging (fMRI) to monitor neural activity while participants heard sentences containing a subject-relative or object-relative center-embedded clause presented at three different speech rates. Syntactically complex object-relative sentences activated left inferior frontal cortex across presentation rates, whereas sentences presented at a rapid rate recruited frontal brain regions such as anterior cingulate and premotor cortex, regardless of syntactic complexity. These results suggest that dissociable components of a large-scale neural network support the processing of syntactic complexity and speech presented at a rapid rate during auditory sentence processing.     

The temporal dynamics of first versus second language production

In this study we explored the temporal origin of processing differences between first and second language production. Forty highly proficient bilinguals named objects of high and low lexical frequency aloud for both L1 and L2 separately while event-related brain potentials (ERPs) were recorded. The first electrophysiological differences elicited by response language occurred at the same early P2 peak (∼140–220 ms) where we observed the onset of the lexical frequency effect, but only for those bilinguals who started naming in an L1 context and afterwards switched to an L2 naming context. The bilinguals who named objects in the reverse direction did not display a language effect in the ERPs. Taken together, the data show that the L2 naming disadvantage originates during the onset of lexical access and seems to be driven by both representational strength, which is lower for L2 words, and language control demands, which are higher for L2 words.     

Eye movements and spoken language comprehension: effects of visual context on syntactic ambiguity resolution

When participants follow spoken instructions to pick up and move objects in a visual workspace, their eye movements to the objects are closely time-locked to referential expressions in the instructions. Two experiments used this methodology to investigate the processing of the temporary ambiguities that arise because spoken language unfolds over time. Experiment 1 examined the processing of sentences with a temporarily ambiguous prepositional phrase (e.g., "Put the apple on the towel in the box") using visual contexts that supported either the normally preferred initial interpretation (the apple should be put on the towel) or the less-preferred interpretation (the apple is already on the towel and should be put in the box). Eye movement patterns clearly established that the initial interpretation of the ambiguous phrase was the one consistent with the context. Experiment 2 replicated these results using prerecorded digitized speech to eliminate any possibility of prosodic differences across conditions or experimenter demand. Overall, the findings are consistent with a broad theoretical framework in which real-time language comprehension immediately takes into account a rich array of relevant nonlinguistic context.     

Not all ambiguous words are created equal: An EEG investigation of homonymy and polysemy

Event-related potentials (ERPs) were used to investigate the time-course of meaning activation of different types of ambiguous words. Unbalanced homonymous ("pen"), balanced homonymous ("panel"), metaphorically polysemous ("lip"), and metonymically polysemous words ("rabbit") were used in a visual single-word priming delayed lexical decision task. The theoretical distinction between homonymy and polysemy was reflected in the N400 component. Homonymous words (balanced and unbalanced) showed effects of dominance/frequency with reduced N400 effects predominantly observed for dominant meanings. Polysemous words (metaphors and metonymies) showed effects of core meaning representation with both dominant and subordinate meanings showing reduced N400 effects. Furthermore, the division within polysemy, into metaphor and metonymy, was supported. Differences emerged in meaning activation patterns with the subordinate meanings of metaphor inducing differentially reduced N400 effects moving from left hemisphere electrode sites to right hemisphere electrode sites, potentially suggesting increased involvement of the right hemisphere in the processing of figurative meaning.     

I see what you're saying: the integration of complex speech and scenes during language comprehension

The effect of language-driven eye movements in a visual scene with concurrent speech was examined using complex linguistic stimuli and complex scenes. The processing demands were manipulated using speech rate and the temporal distance between mentioned objects. This experiment differs from previous research by using complex photographic scenes, three-sentence utterances and mentioning four target objects. The main finding was that objects that are more slowly mentioned, more evenly placed and isolated in the speech stream are more likely to be fixated after having been mentioned and are fixated faster. Surprisingly, even objects mentioned in the most demanding conditions still show an effect of language-driven eye-movements. This supports research using concurrent speech and visual scenes, and shows that the behavior of matching visual and linguistic information is likely to generalize to language situations of high information load.     

Not all implicit measures of attitudes are created equal: Evidence from an embodiment perspective

We hypothesize that while evaluative priming involves proprioceptive cues, the IAT is representational due to its structural features and the specific algorithm upon which the IAT-effect rests. As predicted, evaluative priming is shown to rely on differential facial muscle activity while the IAT as a measurement instrument is not influenced by proprioceptive information. Evaluative priming does not yield differential responsiveness for congruent and incongruent trials when facial muscle activity is inhibited whereas the IAT-effect is shown to be impervious to such inhibition. Implications for the underlying mechanisms of implicit measures are discussed.     

The influence of speaker gaze on listener comprehension: Contrasting visual versus intentional accounts

Previous research has shown that listeners follow speaker gaze to mentioned objects in a shared environment to ground referring expressions, both for human and robot speakers. What is less clear is whether the benefit of speaker gaze is due to the inference of referential intentions (Staudte and Crocker, 2011) or simply the (reflexive) shifts in visual attention. That is, is gaze special in how it affects simultaneous utterance comprehension? In four eye-tracking studies we directly contrast speech-aligned speaker gaze of a virtual agent with a non-gaze visual cue (arrow). Our findings show that both cues similarly direct listeners’ attention and that listeners can benefit in utterance comprehension from both cues. Only when they are similarly precise, however, does this equality extend to incongruent cueing sequences: that is, even when the cue sequence does not match the concurrent sequence of spoken referents can listeners benefit from gaze as well as arrows. The results suggest that listeners are able to learn a counter-predictive mapping of both cues to the sequence of referents. Thus, gaze and arrows can in principle be applied with equal flexibility and efficiency during language comprehension.     

Listening comprehension and working memory are impaired in attention-deficit hyperactivity disorder irrespective of language impairment

This study investigated listening comprehension and working memory abilities in children with attention-deficit hyperactivity disorder (ADHD), presenting with and without language impairments (LI). A 4-group design classified a community sample (n = 77) of boys aged 9–12 into ADHD, ADHD + LI, LI, and Normal groups. Children completed tests of basic language and cognitive skills, verbal and spatial working memory, and passage-level listening comprehension. Multivariate analyses and post hoc comparisons indicated that ADHD children who did not have co-occurring LI comprehended factual information from spoken passages as well as normal children, but were poorer at comprehending inferences and monitoring comprehension of instructions. ADHD children did not differ from normal children in verbal span, but showed significantly poorer verbal working memory, spatial span, and spatial working memory. The ADHD + LI and LI groups were most impaired in listening comprehension and working memory performance, but did not differ from each other. Listening comprehension skills were significantly correlated with both verbal and spatial working memory, and parent–teacher ratings of inattention and hyperactivity/impulsivity. Findings that children with ADHD but no LI showed subtle higher-level listening comprehension deficits have implications for both current diagnostic practices and conceptualizations of ADHD.     

Neural network processing of natural language: II. Towards a unified model of corticostriatal function in learning sentence comprehension and non-linguistic sequencing

A central issue in cognitive neuroscience today concerns how distributed neural networks in the brain that are used in language learning and processing can be involved in non-linguistic cognitive sequence learning. This issue is informed by a wealth of functional neurophysiology studies of sentence comprehension, along with a number of recent studies that examined the brain processes involved in learning non-linguistic sequences, or artificial grammar learning (AGL). The current research attempts to reconcile these data with several current neurophysiologically based models of sentence processing, through the specification of a neural network model whose architecture is constrained by the known cortico-striato-thalamo-cortical (CSTC) neuroanatomy of the human language system. The challenge is to develop simulation models that take into account constraints both from neuranatomical connectivity, and from functional imaging data, and that can actually learn and perform the same kind of language and artificial syntax tasks. In our proposed model, structural cues encoded in a recurrent cortical network in BA47 activate a CSTC circuit to modulate the flow of lexical semantic information from BA45 to an integrated representation of meaning at the sentence level in BA44/6. During language acquisition, corticostriatal plasticity is employed to allow closed class structure to drive thematic role assignment. From the AGL perspective, repetitive internal structure in the AGL strings is encoded in BA47, and activates the CSTC circuit to predict the next element in the sequence. Simulation results from Caplan's [Caplan, D., Baker, C., & Dehaut, F. (1985). Syntactic determinants of sentence comprehension in aphasia. Cognition, 21, 117-175] test of syntactic comprehension, and from Gomez and Schvaneveldts' [Gomez, R. L., & Schvaneveldt, R. W. (1994). What is learned from artificial grammars?. Transfer tests of simple association. Journal of Experimental Psychology: Learning, Memory and Cognition, 20, 396-410] artificial grammar learning experiments are presented. These results are discussed in the context of a brain architecture for learning grammatical structure for multiple natural languages, and non-linguistic sequences.     

Learning to attend: a connectionist model of situated language comprehension

Evidence from numerous studies using the visual world paradigm has revealed both that spoken language can rapidly guide attention in a related visual scene and that scene information can immediately influence comprehension processes. These findings motivated the coordinated interplay account ( Knoeferle & Crocker, 2006 ) of situated comprehension, which claims that utterance-mediated attention crucially underlies this closely coordinated interaction of language and scene processing. We present a recurrent sigma-pi neural network that models the rapid use of scene information, exploiting an utterance-mediated attentional mechanism that directly instantiates the CIA. The model is shown to achieve high levels of performance (both with and without scene contexts), while also exhibiting hallmark behaviors of situated comprehension, such as incremental processing, anticipation of appropriate role fillers, as well as the immediate use, and priority, of depicted event information through the coordinated use of utterance-mediated attention to the scene.     

Not all same-different discriminations are created equal: Evidence contrary to a unidimensional account of same-different learning

In Experiment 1, we trained four pigeons to concurrently discriminate displays of 16 same icons (16S) from displays of 16 different icons (16D) as well as between displays of same icons (16S) from displays that contained 15 same icons and one different icon (15S:1D). The birds rapidly learned to discriminate 16S vs. 16D displays, but they failed to learn to discriminate 16S vs. 15S:1D displays. In Experiment 2, the same pigeons acquired the 16S vs. 15S:1D task after being required to locate and peck at the odd-item in the 15S:1D displays. Acquisition of the 16S vs. 15S:1D task had little effect on discriminative performance in the concurrent 16S:16D task, suggesting that a unidimensional entropy explanation for mastery of these two same-different tasks is not viable. During testing, the birds transferred discriminative performance in both tasks to displays composed of different visual stimuli. Such concurrent discrimination learning, performance, and transfer suggest that pigeons are flexible in the way they process the displays seen in these two same-different tasks.