Perceptual simulation in developing language comprehension

We tested an embodied account of language proposing that comprehenders create perceptual simulations of the events they hear and read about. In Experiment 1, children (ages 7–13 years) performed a picture verification task. Each picture was preceded by a prerecorded spoken sentence describing an entity whose shape or orientation matched or mismatched the depicted object. Responses were faster for matching pictures, suggesting that participants had formed perceptual-like situation models of the sentences. The advantage for matching pictures did not increase with age. Experiment 2 extended these findings to the domain of written language. Participants (ages 7–10 years) of high and low word reading ability verified pictures after reading sentences aloud. The results suggest that even when reading is effortful, children construct a perceptual simulation of the described events. We propose that perceptual simulation plays a more central role in developing language comprehension than was previously thought.     

Language comprehenders mentally represent the shapes of objects

We examined the prediction that people activate perceptual symbols during language comprehension. Subjects read sentences describing an animal or object in a certain location. The shape of the object or animal changed as a function of its location (e.g., eagle in the sky, eagle in a nest). However, this change was only implied by the sentences. After reading a sentence, subjects were presented with a line drawing of the object in question. They judged whether the object had been mentioned in the sentence (Experiment 1) or simply named the object (Experiment 2). In both cases, responses were faster when the pictured object's shape matched the shape implied by the sentence than when there was a mismatch. These results support the hypothesis that perceptual symbols are routinely activated in language comprehension.     

Incidental picture exposure affects later reading: evidence from the N400

Language comprehenders form a mental representation of the implied shape of objects mentioned in the text. In the present study, the influence of prior visual experience on subsequent reading was assessed. In two separate phases, participants saw a picture of an object and read a text about the object, suggesting the same or a different shape. When the shapes in the two phases mismatched, ERPs during reading showed a larger N400 amplitude than when the shapes matched, suggesting that a picture presented incidentally 15 min earlier affected reading. These results further strengthen the case for the interaction of language and visual experience during language comprehension.     

汉语句子理解中物体隐含颜色心理模拟的ERPs研究

当前研究采用事件相关电位(ERPs)技术,同时沿用经典的句图匹配范式,考察句子理解中物体典型颜色与非典型颜色的心理加工过程在脑电活动上的反映。实验中被试先阅读句子再判断句子后呈现的图片物体是否在句子中出现过,句子中隐含的物体颜色或是关键物体的典型颜色或是非典型颜色。实验结果发现,典型颜色句子版本下句图不匹配条件比句图匹配条件引发了更大的N400效应;而非典型颜色句子版本下两者N400差异不显著。研究结果表明,人们在汉语句子理解过程中会实时对隐含的物体颜色信息进行心理模拟。并且,句子隐含物体颜色的典型性是造成匹配易化或不匹配易化的关键因素之一。     

Mental simulation of four visual object properties: similarities and differences as assessed by the sentence–picture verification task

In the sentence–picture verification (SPV) task, people read sentences implying the shape/size/colour/orientation of objects. They then verify whether pictured objects, which either match or mismatch the implied visual information mentioned in the sentence. Faster verification times on matching trials (match advantage) are considered supportive to the notion that readers perform mental simulations during sentence comprehension. This study advances this work by applying a within-subjects design to the SPV-task, enabling us to directly address the strength of and correlation between the match advantages for the properties shape, size, colour, and orientation. Results showed varying match advantages with colour showing the strongest effect, and no match advantage for orientation. Shape, size, and colour were significantly correlated, whereas there were no significant correlations with orientation. These findings suggest that interpretations of match advantages could benefit from a re-evaluation of mental simulation accountsby distinguishing between intrinsic (shape, size, and colour) and extrinsic (orientation) object properties.     

The role of age and perceptual symbols in language comprehension

Older and younger participants read sentences about objects and were then shown a picture of an object that either matched or mismatched the implied shape of the object in the sentence. Participants' response times were recorded when they judged whether the object had been mentioned in the sentence. Responses were faster in the shape-matching condition for all participants, but the mismatch effect was stronger for older than for younger adults, even when the larger variability of the older group's response times was controlled for. These results suggest that older adults may construct stronger situation models than younger adults.     

Is Complex Visual Information Implicated During Language Comprehension? The Case of Cast Shadows

Previous research showed that sensorimotor information affects the perception of properties associated with implied perceptual context during language comprehension. Three experiments addressed a novel question of whether perceptual context may contribute to a simulation of information about such out-of-sight objects as cast shadows. In Experiment 1, participants read a sentence that implied a particular shadow cast on a target (blinds vs. an open window) and then verified the picture of the object onto which a shadow was cast. Responses were faster when the shadow of blinds cast on the object matched that implied by the sentence. However, the data did not show the same matching effect for pictures with cast shadows from an open window. In Experiments 2 and 3, we found that verification times for pictures with no cast shadows were faster when preceded by an “open window” sentence, thus suggesting that reading the sentence does not elicit a visual simulation of any specific shadow. Experiment 3 showed that the objects superimposed with a cast shadow of the blinds and blinds themselves were verified faster after reading a “blinds” sentence. However, the results of an order analysis showed the temporal stability of the “blinds shadows” effect, but the disappearance of the “blinds” effect in the second half of the data. We conclude that the results are compatible, to a lesser or greater extent, with multiple accounts, and discuss our findings in the context of a mental imagery view, a mental simulation view, and an amodal representation view.     

Perceptual representation as a mechanism of lexical ambiguity resolution: An investigation of span and processing time

In 2 experiments, the authors investigated the ability of high- and low-span comprehenders to construe subtle shades of meaning through perceptual representation. High- and low-span comprehenders responded to pictures that either matched or mismatched a target object's shape as implied by the preceding sentence context. At 750 ms after hearing the sentence describing the target object, both high- and low-span comprehenders had activated a contextually appropriate perceptual representation of the target object. However, only high-span comprehenders had perceptually represented the contextually appropriate meaning immediately upon hearing the sentence, whereas low-span comprehenders required more processing time before the perceptual representation was activated. The results are interpreted in a framework of co-occurring lexical representations and perceptual-motor representations.     

Experiential simulations of negated text information

We investigated the question of whether comprehenders mentally simulate a described situation even when this situation is explicitly negated in the sentence. In two experiments, participants read negative sentences such as There was no eagle in the sky, and subsequently responded to pictures of mentioned entities in the context of a recognition task. Participants' responses following negative sentences were faster when the depicted entity matched rather than mismatched the negated situation. These results suggest that comprehenders simulate the negated situation when processing a negated sentence. The results thereby provide further support for the experiential-simulations view of language comprehension.     

Centering on figurative features during the comprehension of sentences describing scenes

Summary Subjects were asked to read sentences which described scenes containing objects. In the scene described in each sentence, a specific part of a particular object was necessarily implied as having an important role. The object was named but none of its parts were. The assumption was that the subjects, while processing the sentence and immediately afterward, would cognitively center on the important part, as a function of the context created by the sentence. Immediately after reading the sentence, the subjects were probed with a picture of either the important part, or of an unimportant part of the object. Judgments of the compatibility of this picture probe with the sentence were faster for pictures of important parts than for pictures of other parts. This was taken as supporting the hypothesis of cognitive centration. In a second experiment, in which words were used as probes instead of pictures, a purely verbal process to account for the results was ruled out. In a third experiment, subjects were given instructions to intentionally form visual images of the scenes described by the sentences. In this case, overall response times to the picture probes were shorter than in the absence of such instructions, but this decrease was greater for pictures of unimportant parts. This finding was interpreted as showing that imagery instructions increase the rate of activation of features to varying degrees as a function of the previous level of activation.     

Experiential simulations of negated text information

We investigated the question of whether comprehenders mentally simulate a described situation even when this situation is explicitly negated in the sentence. In two experiments, participants read negative sentences such as There was no eagle in the sky, and subsequently responded to pictures of mentioned entities in the context of a recognition task. Participants' responses following negative sentences were faster when the depicted entity matched rather than mismatched the negated situation. These results suggest that comprehenders simulate the negated situation when processing a negated sentence. The results thereby provide further support for the experiential-simulations view of language comprehension.     

Representing object colour in language comprehension

Embodied theories of cognition hold that mentally representing something red engages the neural subsystems that respond to environmental perception of that colour. This paper examines whether implicit perceptual information on object colour is represented during sentence comprehension even though doing so does not necessarily facilitate task performance. After reading a sentence that implied a particular colour for a given object, participants were presented with a picture of the object that either matched or mismatched the implied colour. When asked if the pictured object was mentioned in the preceding sentence, people's responses were faster when the colours mismatched than when they matched, suggesting that object colour is represented differently to other object properties such as shape and orientation. A distinction between stable and unstable embodied representations is proposed to allow embodied theories to account for these findings.     

Partial picture effects on children's memory for sentences containing implicit information

Two experiments were conducted examining the effects of partial picture adjuncts on young children's coding of information that was implied in sentences. In the two most critical conditions of these studies, subjects were presented sentences specifying a subject, an action, and a direct object with the instrument used to carry out the action not specified in the sentence (e.g., The workman dug a hole in the ground). Implicit-sentence-only subjects received only the sentences, whereas the implicit sentence + partial picture subjects also viewed a partial picture depicting the action in the sentence minus the implied instrument. The main hypothesis was that subsequent recall of the sentences given the implied instrument as a cue would be facilitated by the partial pictures provided at study, since they would lead the children to infer the instrument. That occurred with 6- to 7-year-old children, but not with preschool children. Consistent with the conclusion that the partial pictures prompted 6- to 7-year-olds to infer the instruments, implicit sentence + partial picture subjects recalled as much as subjects in two other conditions, one in which subjects were explicitly told the instruments at study and one in which subjects saw the instruments depicted in pictures at study. In contrast, preschool subjects who heard explicit sentences containing the instruments outperformed subjects who heard implicit sentences even when the implicit sentences were accompanied by pictures depicting the instruments. This failure of complete pictures to facilitate preschoolers' recall of information implied in sentences contrasts with the many demonstrations of prose learning facilitation when picture and sentence contents explicitly and completely overlap. In summary, there were developmental differences in whether (a) partial pictures significantly facilitated inferencing (and subsequent cued recall) and (b) complete pictures containing information not explicitly stated in sentences promoted cued recall of the sentences.     

理解能力不同的个体抑制干扰信息的效率

根据多媒语言理解测试 ,筛选出汉语综合理解能力高、低不同的两组被试 ,采用固定速移动窗口阅读与选择性再认相结合的实验范式 ,在两个实验中考察了两组被试在理解加工汉语双字词和两从句复合句的过程中对外在干扰信息的抑制效率。词汇加工与句子加工两个实验的结果一致表明 :不同语言理解能力的个体在抑制干扰信息的效率上存在差异 ,低理解能力者的抑制效率也较低 ;抑制效率受到干扰材料意义性的影响 ,越有意义的干扰越难以被抑制     

Moving words: dynamic representations in language comprehension

Eighty-two participants listened to sentences and then judged whether two sequentially presented visual objects were the same. On critical trials, participants heard a sentence describe the motion of a ball toward or away from the observer (e.g., “The pitcher hurled the softball to you”). Seven hundred and fifty milliseconds after the offset of the sentence, a picture of an object was presented for 500 ms, followed by another picture. On critical trials, the two pictures depicted the kind of ball mentioned in the sentence. The second picture was displayed 175 ms after the first. Crucially, it was either slightly larger or smaller than the first picture, thus suggesting movement of the ball toward or away from the observer. Participants responded more quickly when the implied movement of the balls matched the movement described in the sentence. This result provides support for the view that language comprehension involves dynamic perceptual simulations.     

Processing relative clauses varying on syntactic and semantic dimensions: An analysis with event-related potentials

Event-related potentials were used to study how parsing of German relative clauses is influenced by semantic information. Subjects read well-formed sentences containing either a subject or an object relative clause and answered questions concerning the thematic roles expressed in those sentences. Half of the sentences contained past participles that on grounds of semantic plausibility biased either a subject or an object relative reading; the other half contained past participles that provided no semantic information favoring either reading. The past participle elicited an N400 component, larger in amplitude for neutral than for semantically biased verbs, but this occurred only in the case of subject relative clauses. More specific effects were obtained only for a subgroup of subjects, when these were grouped into fast and slow comprehenders on the basis of their questionanswering reaction times. Fast comprehenders showed larger N400 amplitudes for neutral than for semantically biased past participles in general and larger N400s for the latter when there was a bias for an object relative reading as opposed to a subject relative reading. Syntactic ambiguity resolution, indicated by an auxiliary in sentence final position, was associated in this subgroup with a positive component (P345), larger in amplitude for auxiliaries indicating an object relative reading than for those indicating a subject relative reading. The latter component was independent of semantically biasing information given by a preceding past participle. Implications of these findings for models of language comprehension are considered.     

The sound of motion in spoken language: visual information conveyed by acoustic properties of speech

Language is generally viewed as conveying information through symbols whose form is arbitrarily related to their meaning. This arbitrary relation is often assumed to also characterize the mental representations underlying language comprehension. We explore the idea that visuo-spatial information can be analogically conveyed through acoustic properties of speech and that such information is integrated into an analog perceptual representation as a natural part of comprehension. Listeners heard sentences describing objects, spoken at varying speaking rates. After each sentence, participants saw a picture of an object and judged whether it had been mentioned in the sentence. Participants were faster to recognize the object when motion implied by speaking rate matched the motion implied by the picture. Results suggest that visuo-spatial referential information can be analogically conveyed and represented.     

Musical metaphors: Evidence for a spatial grounding of non-literal sentences describing auditory events

This study investigated whether the spatial terms high and low, when used in sentence contexts implying a non-literal interpretation, trigger similar spatial associations as would have been expected from the literal meaning of the words. In three experiments, participants read sentences describing either a high or a low auditory event (e.g., The soprano sings a high aria vs. The pianist plays a low note). In all Experiments, participants were asked to judge (yes/no) whether the sentences were meaningful by means of up/down (Experiments 1 and 2) or left/right (Experiment 3) key press responses. Contrary to previous studies reporting that metaphorical language understanding differs from literal language understanding with regard to simulation effects, the results show compatibility effects between sentence implied pitch height and response location. The results are in line with grounded models of language comprehension proposing that sensory motor experiences are being elicited when processing literal as well as non-literal sentences.     

Investigating individual differences in children's real-time sentence comprehension using language-mediated eye movements

Individual differences in children's online language processing were explored by monitoring their eye movements to objects in a visual scene as they listened to spoken sentences. Eleven skilled and 11 less-skilled comprehenders were presented with sentences containing verbs that were either neutral with respect to the visual context (e.g., Jane watched her mother choose the cake, where all of the objects in the scene were choosable) or supportive (e.g., Jane watched her mother eat the cake, where the cake was the only edible object). On hearing the supportive verb, the children made fast anticipatory eye movements to the target object (e.g., the cake), suggesting that children extract information from the language they hear and use this to direct ongoing processing. Less-skilled comprehenders did not differ from controls in the speed of their anticipatory eye movements, suggesting normal sensitivity to linguistic constraints. However, less-skilled comprehenders made a greater number of fixations to target objects, and these fixations were of a duration shorter than those observed in the skilled comprehenders, especially in the supportive condition. This pattern of results is discussed in terms of possible processing limitations, including difficulties with memory, attention, or suppressing irrelevant information.