Grounding language in action

We report a new phenomenon associated with language comprehension: the action-sentence compatibility effect (ACE). Participants judged whether sentences were sensible by making a response that required moving toward or away from their bodies. When a sentence implied action in one direction (e.g., "Close the drawer" implies action away from the body), the participants had difficulty making a sensibility judgment requiring a response in the opposite direction. The ACE was demonstrated for three sentences types: imperative sentences, sentences describing the transfer of concrete objects, and sentences describing the transfer of abstract entities, such as "Liz told you the story." These dataare inconsistent with theories of language comprehension in which meaning is represented as a set of relations among nodes. Instead, the data support an embodied theory of meaning that relates the meaning of sentences to human action.     

Processing abstract language modulates motor system activity

Embodiment theory proposes that neural systems for perception and action are also engaged during language comprehension. Previous neuroimaging and neurophysiological studies have only been able to demonstrate modulation of action systems during comprehension of concrete language. We provide neurophysiological evidence for modulation of motor system activity during the comprehension of both concrete and abstract language. In Experiment 1, when the described direction of object transfer or information transfer (e.g., away from the reader to another) matched the literal direction of a hand movement used to make a response, speed of responding was faster than when the two directions mismatched (an action-sentence compatibility effect). In Experiment 2, we used single-pulse transcranial magnetic stimulation to study changes in the corticospinal motor pathways to hand muscles while reading the same sentences. Relative to sentences that do not describe transfer, there is greater modulation of activity in the hand muscles when reading sentences describing transfer of both concrete objects and abstract information. These findings are discussed in relation to the human mirror neuron system.     

Temporal dynamics of the action-sentence compatibility effect

A number of recent studies have demonstrated variants of the action-sentence compatibility effect (ACE), wherein the execution of a motor response is facilitated by the comprehension of sentences that describe actions taking place in the same direction as the motor response (e.g., a sentence about action towards one's body facilitates the execution of an arm movement towards the body). This paper presents an experiment that explores how the timing of the motor response during the processing of sentences affects the magnitude of the ACE that is observed. The results show that the ACE occurs when the motor response is executed at an early point in the comprehension of the sentence, disappears for a time, and then reappears when the motor response is executed right before the end of the sentence. These data help to refine our understanding of the temporal dynamics involved in the activation and use of motor information during sentence comprehension.     

The Action-Sentence Compatibility Effect: It's All in the Timing

When participants are asked to make sensibility judgments on sentences that describe action toward the body (i.e., "Mark dealt the cards to you") or away from the body (i.e., "You dealt the cards to Mark"), they are faster to respond when the response requires an arm movement in the same direction as the action described by the sentence. This congruence effect is known as the Action–Sentence Compatibility Effect (ACE). This study reports 4 experiments that extend our understanding of the ACE by exploring how the time at which one prepares the motor response required for the sensibility judgment affects the magnitude of the ACE. Results show that the ACE arises only when participants have the opportunity to plan their motor response while they are processing the sentence.     

Experiential simulations of negated text information

We investigated the question of whether comprehenders mentally simulate a described situation even when this situation is explicitly negated in the sentence. In two experiments, participants read negative sentences such as There was no eagle in the sky, and subsequently responded to pictures of mentioned entities in the context of a recognition task. Participants' responses following negative sentences were faster when the depicted entity matched rather than mismatched the negated situation. These results suggest that comprehenders simulate the negated situation when processing a negated sentence. The results thereby provide further support for the experiential-simulations view of language comprehension.     

Power and spatial relations

Embodied views of language hold that perceptual and motor simulations are involved during comprehension. Study 1 investigated power-related language in a picture recognition task wherein participants judged whether the picture presented at either the upper or lower screen position matched the sentence. In Study 2, participants chose the picture that best described the sentence between two (identical) images that were aligned vertically. Results demonstrated that participants responded faster to pictures presented at the implied location and chose that picture more often. Such results suggest that spatial information implicit in power-related language is involved in linguistic comprehension.     

Experiential simulations of negated text information

We investigated the question of whether comprehenders mentally simulate a described situation even when this situation is explicitly negated in the sentence. In two experiments, participants read negative sentences such as There was no eagle in the sky, and subsequently responded to pictures of mentioned entities in the context of a recognition task. Participants' responses following negative sentences were faster when the depicted entity matched rather than mismatched the negated situation. These results suggest that comprehenders simulate the negated situation when processing a negated sentence. The results thereby provide further support for the experiential-simulations view of language comprehension.     

The action-sentence compatibility effect in Japanese sentences

The action-sentence compatibility effect (ACE) is a phenomenon in which a reader's response to a sentence is made faster when there is congruity between the action described in the sentence and the action that makes up the response. Previous studies showed the ACE occurs in action-related sentences in several languages. However, all these were SVO (verb-object) languages, in which verbs are placed before object nouns; this order is reversed in SOV languages. Moreover, those studies investigated hand responses. This study assessed the existence of the ACE in Japanese, an SOV language, and in foot responses. 24 female participants judged the sensibility of Japanese sentences that described actions and responded with either their foot or hand as an effector. Reaction times were significantly faster when there was congruity between the effector described in the sentences and the effector actually used for the response. However, sentence dependency was also found in the foot responses.     

The sound of motion in spoken language: visual information conveyed by acoustic properties of speech

Language is generally viewed as conveying information through symbols whose form is arbitrarily related to their meaning. This arbitrary relation is often assumed to also characterize the mental representations underlying language comprehension. We explore the idea that visuo-spatial information can be analogically conveyed through acoustic properties of speech and that such information is integrated into an analog perceptual representation as a natural part of comprehension. Listeners heard sentences describing objects, spoken at varying speaking rates. After each sentence, participants saw a picture of an object and judged whether it had been mentioned in the sentence. Participants were faster to recognize the object when motion implied by speaking rate matched the motion implied by the picture. Results suggest that visuo-spatial referential information can be analogically conveyed and represented.     

Sentence Comprehension Deficits in Alzheimer''s Disease: A Comparison of Off-Line vs. On-Line Sentence Processing

Two studies explored whether sentence comprehension impairments in Alzheimer's disease (AD) are due to deficits in syntactic processing or memory. Study 1 used a picture-pointing sentence comprehension task to measure the final outcome of comprehension in an off-line fashion. It showed the comprehension of 30 patients with AD to be impaired, but suggested that the deficits could not be attributed solely to syntactic impairments. Study 2 investigated the effects of memory on sentence comprehension by comparing off-line (grammaticality judgment) with on-line (cross-modal naming) language processing in 11 AD and 9 control subjects. The results revealed impaired performance in the off-line task but normal performance in the on-line task using the same sentences. Performance on the off-line task correlated with independent measures of verbal working memory. These data are used to argue that sentence comprehension impairments are related to verbal working memory deficits in AD.     

Perception of motion affects language processing

Recently developed accounts of language comprehension propose that sentences are understood by constructing a perceptual simulation of the events being described. These simulations involve the re-activation of patterns of brain activation that were formed during the comprehender's interaction with the world. In two experiments we explored the specificity of the processing mechanisms required to construct simulations during language comprehension. Participants listened to (and made judgments on) sentences that described motion in a particular direction (e.g. "The car approached you"). They simultaneously viewed dynamic black-and-white stimuli that produced the perception of movement in the same direction as the action specified in the sentence (i.e. towards you) or in the opposite direction as the action specified in the sentence (i.e. away from you). Responses were faster to sentences presented concurrently with a visual stimulus depicting motion in the opposite direction as the action described in the sentence. This suggests that the processing mechanisms recruited to construct simulations during language comprehension are also used during visual perception, and that these mechanisms can be quite specific.     

Language comprehenders mentally represent the shapes of objects

We examined the prediction that people activate perceptual symbols during language comprehension. Subjects read sentences describing an animal or object in a certain location. The shape of the object or animal changed as a function of its location (e.g., eagle in the sky, eagle in a nest). However, this change was only implied by the sentences. After reading a sentence, subjects were presented with a line drawing of the object in question. They judged whether the object had been mentioned in the sentence (Experiment 1) or simply named the object (Experiment 2). In both cases, responses were faster when the pictured object's shape matched the shape implied by the sentence than when there was a mismatch. These results support the hypothesis that perceptual symbols are routinely activated in language comprehension.     

Embodied Simulations Are Modulated by Sentential Perspective

There is considerable evidence that language comprehenders derive lexical‐semantic meaning by mentally simulating perceptual and motor attributes of described events. However, the nature of these simulations—including the level of detail that is incorporated and contexts under which simulations occur—is not well understood. Here, we examine the effects of first‐ versus third‐person perspective on mental simulations during sentence comprehension. First‐person sentences describing physical transfer towards or away from the body (e.g., “You threw the microphone,” “You caught the microphone”) modulated response latencies when responses were made along a front‐back axis, consistent with the action‐sentence compatibility effect (ACE). This effect was not observed for third‐person sentences (“He threw the microphone,” “He caught the microphone”). The ACE was observed when making responses along a left‐right axis for third‐person, but not first‐person sentences. Abstract sentences (e.g., “He heard the message”) did not show an ACE along either axis. These results show that perspective is a detail that is simulated during action sentence comprehension, and that motoric activations are flexible and affected by the pronominal perspective used in the sentence.     

Encouraging prediction during production facilitates subsequent comprehension: Evidence from interleaved object naming in sentence context and sentence reading

Many studies have shown that a supportive context facilitates language comprehension. A currently influential view is that language production may support prediction in language comprehension. Experimental evidence for this, however, is relatively sparse. Here we explored whether encouraging prediction in a language production task encourages the use of predictive contexts in an interleaved comprehension task. In Experiment 1a, participants listened to the first part of a sentence and provided the final word by naming aloud a picture. The picture name was predictable or not predictable from the sentence context. Pictures were named faster when they could be predicted than when this was not the case. In Experiment 1b the same sentences, augmented by a final spill-over region, were presented in a self-paced reading task. No difference in reading times for predictive versus non-predictive sentences was found. In Experiment 2, reading and naming trials were intermixed. In the naming task, the advantage for predictable picture names was replicated. More importantly, now reading times for the spill-over region were considerable faster for predictive than for non-predictive sentences. We conjecture that these findings fit best with the notion that prediction in the service of language production encourages the use of predictive contexts in comprehension. Further research is required to identify the exact mechanisms by which production exerts its influence on comprehension.     

Moving words: dynamic representations in language comprehension

Eighty-two participants listened to sentences and then judged whether two sequentially presented visual objects were the same. On critical trials, participants heard a sentence describe the motion of a ball toward or away from the observer (e.g., “The pitcher hurled the softball to you”). Seven hundred and fifty milliseconds after the offset of the sentence, a picture of an object was presented for 500 ms, followed by another picture. On critical trials, the two pictures depicted the kind of ball mentioned in the sentence. The second picture was displayed 175 ms after the first. Crucially, it was either slightly larger or smaller than the first picture, thus suggesting movement of the ball toward or away from the observer. Participants responded more quickly when the implied movement of the balls matched the movement described in the sentence. This result provides support for the view that language comprehension involves dynamic perceptual simulations.     

Memory-load interference in syntactic processing

Participants remembered a short set of words while reading syntactically complex sentences (object–extracted clefts) and syntactically simpler sentences (subject–extracted clefts) in a memory–load study. The study also manipulated whether the words in the set and the words in the sentence were of matched or unmatched types (common nouns vs. proper names). Performance in sentence comprehension was worse for complex sentences than for simpler sentences, and this effect was greater when the type of words in the memory load matched the type of words in the sentence. These results indicate that syntactic processing is not modular, instead suggesting that it relies on working memory resources that are used for other nonsyntactic processes. Further, the results indicate that similarity–based interference is an important constraint on information processing that can be overcome to some degree during language comprehension by using the coherence of language to construct integrated representations of meaning.     

Intonational marking of given and new information: Some consequences for comprehension

The role of intonation in conveying discourse relationships in auditory sentence comprehension was investigated in two experiments. Using the simple comprehension time paradigm, Experiment 1 found that sentences with accented new information were understood faster than sentences with a neutral intonation contour and that the presence of accent in context sentences facilitated comprehension of subsequent targets. Both experiments showed faster comprehension times in conditions in which accent placement was appropriate for the information structure of the sentence. In Experiment I, comprehension times were faster when the accent fell on the information focus than when it fell elsewhere in the sentence. In Experiment 2, faster times resulted when new information was accented and given information was not, compared to conditions in which this accent pattern was reversed. This effect held for both active and passive sentences, and whether the new information occurred in the subject or object position.     

Spatial distance effects on incremental semantic interpretation of abstract sentences: Evidence from eye tracking

A large body of evidence has shown that visual context information can rapidly modulate language comprehension for concrete sentences and when it is mediated by a referential or a lexical-semantic link. What has not yet been examined is whether visual context can also modulate comprehension of abstract sentences incrementally when it is neither referenced by, nor lexically associated with, the sentence. Three eye-tracking reading experiments examined the effects of spatial distance between words (Experiment 1) and objects (Experiment 2 and 3) on participants’ reading times for sentences that convey similarity or difference between two abstract nouns (e.g., ‘Peace and war are certainly different...’). Before reading the sentence, participants inspected a visual context with two playing cards that moved either far apart or close together. In Experiment 1, the cards turned and showed the first two nouns of the sentence (e.g., ‘peace’, ‘war’). In Experiments 2 and 3, they turned but remained blank. Participants’ reading times at the adjective (Experiment 1: first-pass reading time; Experiment 2: total times) and at the second noun phrase (Experiment 3: first-pass times) were faster for sentences that expressed similarity when the preceding words/objects were close together (vs. far apart) and for sentences that expressed dissimilarity when the preceding words/objects were far apart (vs. close together). Thus, spatial distance between words or entirely unrelated objects can rapidly and incrementally modulate the semantic interpretation of abstract sentences.     

汉语句子理解中物体隐含颜色心理模拟的ERPs研究

当前研究采用事件相关电位(ERPs)技术,同时沿用经典的句图匹配范式,考察句子理解中物体典型颜色与非典型颜色的心理加工过程在脑电活动上的反映。实验中被试先阅读句子再判断句子后呈现的图片物体是否在句子中出现过,句子中隐含的物体颜色或是关键物体的典型颜色或是非典型颜色。实验结果发现,典型颜色句子版本下句图不匹配条件比句图匹配条件引发了更大的N400效应;而非典型颜色句子版本下两者N400差异不显著。研究结果表明,人们在汉语句子理解过程中会实时对隐含的物体颜色信息进行心理模拟。并且,句子隐含物体颜色的典型性是造成匹配易化或不匹配易化的关键因素之一。     

Representing object colour in language comprehension

Embodied theories of cognition hold that mentally representing something red engages the neural subsystems that respond to environmental perception of that colour. This paper examines whether implicit perceptual information on object colour is represented during sentence comprehension even though doing so does not necessarily facilitate task performance. After reading a sentence that implied a particular colour for a given object, participants were presented with a picture of the object that either matched or mismatched the implied colour. When asked if the pictured object was mentioned in the preceding sentence, people's responses were faster when the colours mismatched than when they matched, suggesting that object colour is represented differently to other object properties such as shape and orientation. A distinction between stable and unstable embodied representations is proposed to allow embodied theories to account for these findings.