Perception of auditory motion affects language processing

Previous reports have demonstrated that the comprehension of sentences describing motion in a particular direction (toward, away, up, or down) is affected by concurrently viewing a stimulus that depicts motion in the same or opposite direction. We report 3 experiments that extend our understanding of the relation between perception and language processing in 2 ways. First, whereas most previous studies of the relation between perception and language processing have focused on visual perception, our data show that sentence processing can be affected by the concurrent processing of auditory stimuli. Second, it is shown that the relation between the processing of auditory stimuli and the processing of sentences depends on whether the sentences are presented in the auditory or visual modality.     

Grounding language in action

We report a new phenomenon associated with language comprehension: the action-sentence compatibility effect (ACE). Participants judged whether sentences were sensible by making a response that required moving toward or away from their bodies. When a sentence implied action in one direction (e.g., "Close the drawer" implies action away from the body), the participants had difficulty making a sensibility judgment requiring a response in the opposite direction. The ACE was demonstrated for three sentences types: imperative sentences, sentences describing the transfer of concrete objects, and sentences describing the transfer of abstract entities, such as "Liz told you the story." These dataare inconsistent with theories of language comprehension in which meaning is represented as a set of relations among nodes. Instead, the data support an embodied theory of meaning that relates the meaning of sentences to human action.     

有关运动的语言理解的知觉模拟

针对阅读理解是建构抽象命题网络的传统理论,知觉符号理论提出：阅读加工是建构语言所述事件的知觉模拟,并由此推论：有关运动的语言理解包括动态的知觉模拟。该动态模拟假设得到一些实证研究的支持。行为学上的研究证明了语言理解中动态知觉模拟的存在,还对这种模拟的机制进行了初步探讨。该假设也得到一些认知神经科学研究的支持。作为一种新兴的观点,动态模拟假设还需要深入的检验,以进一步充实知觉符号理论的语言理解模拟观     

Embodiment Effects and Language Comprehension in Alzheimer's Disease

It has been shown that when participants are asked to make sensibility judgments on sentences that describe a transfer of an object toward or away from their body, they are faster to respond when the response requires a movement in the same direction as the transfer described in the sentence. This phenomenon is known as the action compatibility effect (ACE). This study investigates whether the ACE exists for volunteers with Alzheimer's disease (AD), whether the ACE can facilitate language comprehension, and also whether the ACE can still be produced if the order of the two events is inverted, that is, whether overt movement can prime comprehension of transfer sentences. In Study 1, participants with AD, younger, and older adults were tested on an adaptation of the ACE Paradigm. In Study 2, the same paradigm was modified to include an arm movement that participants had to perform prior to sentence exposure on screen. In Study 1, young, older adults, and individuals with AD were faster to respond when the direction of the response movement matched the directionality implied by the sentence (ACE). In Study 2, no traditional ACE was found; participants were faster when the direction of the movement immediately preceding the sentence matched the directionality of the sentence. It was found that compatibility effects generated a relative advantage, that transfer schemata are easier to process, and that an ACE‐like effect can be the result of mutual priming between language and movement. Results suggested preservation in AD of the neural systems for action engaged during language comprehension, and conditions under which comprehension in AD can be facilitated in real life may be identified.     

语言理解中的动作知觉：基于具身认知的视角

具身认知理论是目前认知科学领域中最新的研究范式和取向, 它认为人的认知过程依赖于感知和动作系统, 强调身体在认知中发挥重要作用。具身语言理解则认为, 身体、动作和知觉系统在语言认知中也起着不可或缺的作用。文章在简要回顾具身语言理解的相关理论如索引假设、浸入式经历者框架、语言神经理论的基础上, 重点从语言理解的四个层面的实证研究, 即音位、单词、句子、语篇, 证实具身语言理解的观点。未来的研究应该着眼于用具身认知观点来解释更高级的抽象语言表征, 具身单词、句子、语篇的具体表征形式, 并利用具身语言认知的观点进行认知的本土化研究。     

动词理解中空间表征的激活过程

探讨动词理解中空间表征的激活过程。被试听以不同空间轴向动词作谓语的句子后辨认视觉刺激的形状。实验1以肯定句为材料探讨语义理解对视知觉任务的影响。实验2以客观原因否定句为材料排除句子表征的影响。实验3以主观原因否定句为材料探讨动词空间元素的激活机制。总的结果表明,动词理解过程中激活其表征中的空间元素,这种激活是自动的、非策略性的,不受情境中客观原因或主观意愿否定的影响。动词理解空间效应反映了语言表征中的知觉运动特征     

Moving words: dynamic representations in language comprehension

Eighty-two participants listened to sentences and then judged whether two sequentially presented visual objects were the same. On critical trials, participants heard a sentence describe the motion of a ball toward or away from the observer (e.g., “The pitcher hurled the softball to you”). Seven hundred and fifty milliseconds after the offset of the sentence, a picture of an object was presented for 500 ms, followed by another picture. On critical trials, the two pictures depicted the kind of ball mentioned in the sentence. The second picture was displayed 175 ms after the first. Crucially, it was either slightly larger or smaller than the first picture, thus suggesting movement of the ball toward or away from the observer. Participants responded more quickly when the implied movement of the balls matched the movement described in the sentence. This result provides support for the view that language comprehension involves dynamic perceptual simulations.     

Spatial and linguistic aspects of visual imagery in sentence comprehension

There is mounting evidence that language comprehension involves the activation of mental imagery of the content of utterances ( Barsalou, 1999 ; Bergen, Chang, & Narayan, 2004 ; Bergen, Narayan, & Feldman, 2003 ; Narayan, Bergen, & Weinberg, 2004 ; Richardson, Spivey, McRae, & Barsalou, 2003 ; Stanfield & Zwaan, 2001 ; Zwaan, Stanfield, & Yaxley, 2002 ). This imagery can have motor or perceptual content. Three main questions about the process remain under‐explored, however. First, are lexical associations with perception or motion sufficient to yield mental simulation, or is the integration of lexical semantics into larger structures, like sentences, necessary? Second, what linguistic elements (e.g., verbs, nouns, etc.) trigger mental simulations? Third, how detailed are the visual simulations that are performed? A series of behavioral experiments address these questions, using a visual object categorization task to investigate whether up‐ or down‐related language selectively interferes with visual processing in the same part of the visual field (following Richardson et al., 2003 ). The results demonstrate that either subject nouns or main verbs can trigger visual imagery, but only when used in literal sentences about real space—metaphorical language does not yield significant effects—which implies that it is the comprehension of the sentence as a whole and not simply lexical associations that yields imagery effects. These studies also show that the evoked imagery contains detail as to the part of the visual field where the described scene would take place.     

The Action-Sentence Compatibility Effect: It's All in the Timing

When participants are asked to make sensibility judgments on sentences that describe action toward the body (i.e., "Mark dealt the cards to you") or away from the body (i.e., "You dealt the cards to Mark"), they are faster to respond when the response requires an arm movement in the same direction as the action described by the sentence. This congruence effect is known as the Action–Sentence Compatibility Effect (ACE). This study reports 4 experiments that extend our understanding of the ACE by exploring how the time at which one prepares the motor response required for the sensibility judgment affects the magnitude of the ACE. Results show that the ACE arises only when participants have the opportunity to plan their motor response while they are processing the sentence.     

Language and age effects in children's processing of word order

We compare the processing of transitive sentences in young learners of a strict word order language (English) and two languages that allow noun omissions and many variant word orders: Turkish, a case-marked language, and Mandarin Chinese, a non case-marked language. Children aged 1–3 years listened to simple transitive sentences in the typical word order of their language, paired with two visual scenes, only one of which matched the sentence. Multiple measures of comprehension (percent of looking to match, latency to look to match, number of switches of attention) revealed a general pattern of early sensitivity to word order, coupled with language and age effects in children's processing efficiency. In particular, English learners showed temporally speedier processing of transitive sentences than Turkish learners, who also displayed more uncertainty about the matching scene. Mandarin learners behaved like Turkish learners in showing slower processing of sentences, and all language groups displayed faster processing by older than younger children. These results demonstrate that sentence processing is sensitive to crosslinguistic features beginning early in language development.     

Embodied Simulations Are Modulated by Sentential Perspective

There is considerable evidence that language comprehenders derive lexical‐semantic meaning by mentally simulating perceptual and motor attributes of described events. However, the nature of these simulations—including the level of detail that is incorporated and contexts under which simulations occur—is not well understood. Here, we examine the effects of first‐ versus third‐person perspective on mental simulations during sentence comprehension. First‐person sentences describing physical transfer towards or away from the body (e.g., “You threw the microphone,” “You caught the microphone”) modulated response latencies when responses were made along a front‐back axis, consistent with the action‐sentence compatibility effect (ACE). This effect was not observed for third‐person sentences (“He threw the microphone,” “He caught the microphone”). The ACE was observed when making responses along a left‐right axis for third‐person, but not first‐person sentences. Abstract sentences (e.g., “He heard the message”) did not show an ACE along either axis. These results show that perspective is a detail that is simulated during action sentence comprehension, and that motoric activations are flexible and affected by the pronominal perspective used in the sentence.     

Perceptual simulation in developing language comprehension

We tested an embodied account of language proposing that comprehenders create perceptual simulations of the events they hear and read about. In Experiment 1, children (ages 7–13 years) performed a picture verification task. Each picture was preceded by a prerecorded spoken sentence describing an entity whose shape or orientation matched or mismatched the depicted object. Responses were faster for matching pictures, suggesting that participants had formed perceptual-like situation models of the sentences. The advantage for matching pictures did not increase with age. Experiment 2 extended these findings to the domain of written language. Participants (ages 7–10 years) of high and low word reading ability verified pictures after reading sentences aloud. The results suggest that even when reading is effortful, children construct a perceptual simulation of the described events. We propose that perceptual simulation plays a more central role in developing language comprehension than was previously thought.     

Processing abstract language modulates motor system activity

Embodiment theory proposes that neural systems for perception and action are also engaged during language comprehension. Previous neuroimaging and neurophysiological studies have only been able to demonstrate modulation of action systems during comprehension of concrete language. We provide neurophysiological evidence for modulation of motor system activity during the comprehension of both concrete and abstract language. In Experiment 1, when the described direction of object transfer or information transfer (e.g., away from the reader to another) matched the literal direction of a hand movement used to make a response, speed of responding was faster than when the two directions mismatched (an action-sentence compatibility effect). In Experiment 2, we used single-pulse transcranial magnetic stimulation to study changes in the corticospinal motor pathways to hand muscles while reading the same sentences. Relative to sentences that do not describe transfer, there is greater modulation of activity in the hand muscles when reading sentences describing transfer of both concrete objects and abstract information. These findings are discussed in relation to the human mirror neuron system.     

Eye Movements Reveal the Dynamic Simulation of Speed in Language

This study investigates how speed of motion is processed in language. In three eye‐tracking experiments, participants were presented with visual scenes and spoken sentences describing fast or slow events (e.g., The lion ambled/dashed to the balloon). Results showed that looking time to relevant objects in the visual scene was affected by the speed of verb of the sentence, speaking rate, and configuration of a supporting visual scene. The results provide novel evidence for the mental simulation of speed in language and show that internal dynamic simulations can be played out via eye movements toward a static visual scene.     

Sentence processing deficits in two Cantonese aphasic patients

This paper reports the performances of two Cantonese aphasics on tasks examining their sentence processing deficits. The data on sentence comprehension show that thematically noncanonical sentences, full passives, and subject-gap sentences present greater difficulty to these patients than canonical sentences, truncated passives, and object-gap sentences, respectively. These patterns are consistent with previous observations on Chinese aphasics and are expected given the structural differences between Chinese and English. In a Cantonese grammaticality judgment test, a set of structures are identified that can elicit clear judgments from normal subjects and aphasics, contrary to the claim that grammaticality judgments in Chinese are probabilistic and fragile. Most interestingly, the patients' overall performance patterns reveal a double dissociation between sentence comprehension and judgment of sentence well-formedness, suggesting that the two tasks are supported by independent processing mechanisms.     

汉语句子可继续性对句子理解加工的即时影响

通过操纵动词与后接词的关系,来产生搭配异常并控制句子的可继续性,研究汉语词汇信息在句子加工中的即时作用。实验一利用跨通道技术,发现动一名词搭配异常对目标名词的加工有即时影响,形容词条件下可继续性对目标形容词的加工有即时影响;实验二采用眼动技术,发现名词条件和形容词条件下可继续性均有显著的即时影响。结果显示,语义等多种词汇信息被通达后立即共同参与了汉语句子加工。     

Memory-load interference in syntactic processing

Participants remembered a short set of words while reading syntactically complex sentences (object–extracted clefts) and syntactically simpler sentences (subject–extracted clefts) in a memory–load study. The study also manipulated whether the words in the set and the words in the sentence were of matched or unmatched types (common nouns vs. proper names). Performance in sentence comprehension was worse for complex sentences than for simpler sentences, and this effect was greater when the type of words in the memory load matched the type of words in the sentence. These results indicate that syntactic processing is not modular, instead suggesting that it relies on working memory resources that are used for other nonsyntactic processes. Further, the results indicate that similarity–based interference is an important constraint on information processing that can be overcome to some degree during language comprehension by using the coherence of language to construct integrated representations of meaning.     

Temporal dynamics of the action-sentence compatibility effect

A number of recent studies have demonstrated variants of the action-sentence compatibility effect (ACE), wherein the execution of a motor response is facilitated by the comprehension of sentences that describe actions taking place in the same direction as the motor response (e.g., a sentence about action towards one's body facilitates the execution of an arm movement towards the body). This paper presents an experiment that explores how the timing of the motor response during the processing of sentences affects the magnitude of the ACE that is observed. The results show that the ACE occurs when the motor response is executed at an early point in the comprehension of the sentence, disappears for a time, and then reappears when the motor response is executed right before the end of the sentence. These data help to refine our understanding of the temporal dynamics involved in the activation and use of motor information during sentence comprehension.     

Seeing, acting, understanding: motor resonance in language comprehension

Observing actions and understanding sentences about actions activates corresponding motor processes in the observer-comprehender. In 5 experiments, the authors addressed 2 novel questions regarding language-based motor resonance. The 1st question asks whether visual motion that is associated with an action produces motor resonance in sentence comprehension. The 2nd question asks whether motor resonance is modulated during sentence comprehension. The authors' experiments provide an affirmative response to both questions. A rotating visual stimulus affects both actual manual rotation and the comprehension of manual rotation sentences. Motor resonance is modulated by the linguistic input and is a rather immediate and localized phenomenon. The results are discussed in the context of theories of action observation and mental simulation.