The impact of continuity editing in narrative film on event segmentation

Filmmakers use continuity editing to engender a sense of situational continuity or discontinuity at editing boundaries. The goal of this study was to assess the impact of continuity editing on how people perceive the structure of events in a narrative film and to identify brain networks that are associated with the processing of different types of continuity editing boundaries. Participants viewed a commercially produced film and segmented it into meaningful events, while brain activity was recorded with functional magnetic resonance imaging (MRI). We identified three degrees of continuity that can occur at editing locations: edits that are continuous in space, time, and action; edits that are discontinuous in space or time but continuous in action; and edits that are discontinuous in action as well as space or time. Discontinuities in action had the biggest impact on behavioral event segmentation, and discontinuities in space and time had minor effects. Edits were associated with large transient increases in early visual areas. Spatial-temporal changes and action changes produced strikingly different patterns of transient change, and they provided evidence that specialized mechanisms in higher order perceptual processing regions are engaged to maintain continuity of action in the face of spatiotemporal discontinuities. These results suggest that commercial film editing is shaped to support the comprehension of meaningful events that bridge breaks in low-level visual continuity, and even breaks in continuity of spatial and temporal location.     

Walking through doorways causes forgetting: Further explorations

Previous research using virtual environments has revealed a location-updating effect in which there is a decline in memory when people move from one location to another. Here we assess whether this effect reflects the influence of the experienced context, in terms of the degree of immersion of a person in an environment, as suggested by some work in spatial cognition, or by a shift in context. In Experiment 1, the degree of immersion was reduced by using smaller displays. In comparison, in Experiment 2 an actual, rather than a virtual, environment was used, to maximize immersion. Location-updating effects were observed under both of these conditions. In Experiment 3, the original encoding context was reinstated by having a person return to the original room in which objects were first encoded. However, inconsistent with an encoding specificity account, memory did not improve by reinstating this context. Finally, we did a further analysis of the results of this and previous experiments to assess the differential influence of foregrounding and retrieval interference. Overall, these data are interpreted in terms of the event horizon model of event cognition and memory.     

Situation Models: The Mental Leap Into Imagined Worlds

Situation models are mental representations of the state of affairs described in a text rather than of the text itself. Much of the research on situation models in narrative comprehension suggests that comprehenders behave as though they are in the narrated situation rather than outside of it. This article reviews some of this evidence and provides an outlook on future developments.     

Perception of motion affects language processing

Recently developed accounts of language comprehension propose that sentences are understood by constructing a perceptual simulation of the events being described. These simulations involve the re-activation of patterns of brain activation that were formed during the comprehender's interaction with the world. In two experiments we explored the specificity of the processing mechanisms required to construct simulations during language comprehension. Participants listened to (and made judgments on) sentences that described motion in a particular direction (e.g. "The car approached you"). They simultaneously viewed dynamic black-and-white stimuli that produced the perception of movement in the same direction as the action specified in the sentence (i.e. towards you) or in the opposite direction as the action specified in the sentence (i.e. away from you). Responses were faster to sentences presented concurrently with a visual stimulus depicting motion in the opposite direction as the action described in the sentence. This suggests that the processing mechanisms recruited to construct simulations during language comprehension are also used during visual perception, and that these mechanisms can be quite specific.     

Walking through doorways causes forgetting: Event structure or updating disruption?

According to event cognition theory, people segment experience into separate event models. One consequence of this segmentation is that when people transport objects from one location to another, memory is worse than if people move across a large location. In two experiments participants navigated through a virtual environment, and recognition memory was tested in either the presence or the absence of a location shift for objects that were recently interacted with (i.e., just picked up or set down). Of particular concern here is whether this location updating effect is due to (a) differences in retention intervals as a result of the navigation process, (b) a temporary disruption in cognitive processing that may occur as a result of the updating processes, or (c) a need to manage multiple event models, as has been suggested in prior research. Experiment 1 explored whether retention interval is driving this effect by recording travel times from the acquisition of an object and the probe time. The results revealed that travel times were similar, thereby rejecting a retention interval explanation. Experiment 2 explored whether a temporary disruption in processing is producing the effect by introducing a 3-second delay prior to the presentation of a memory probe. The pattern of results was not affected by adding a delay, thereby rejecting a temporary disruption account. These results are interpreted in the context of the event horizon model, which suggests that when there are multiple event models that contain common elements there is interference at retrieval, which compromises performance.     

Language knowledge and event knowledge in language use

This paper examines how semantic knowledge is used in language comprehension and in making judgments about events in the world. We contrast knowledge gleaned from prior language experience (“language knowledge”) and knowledge coming from prior experience with the world (“world knowledge”). In two corpus analyses, we show that previous research linking verb aspect and event representations have confounded language and world knowledge. Then, using carefully chosen stimuli that remove this confound, we performed four experiments that manipulated the degree to which language knowledge or world knowledge should be salient and relevant to performing a task, finding in each case that participants use the type of knowledge most appropriate to the task. These results provide evidence for a highly context-sensitive and interactionist perspective on how semantic knowledge is represented and used during language processing.     

Perceptual simulation in developing language comprehension

We tested an embodied account of language proposing that comprehenders create perceptual simulations of the events they hear and read about. In Experiment 1, children (ages 7–13 years) performed a picture verification task. Each picture was preceded by a prerecorded spoken sentence describing an entity whose shape or orientation matched or mismatched the depicted object. Responses were faster for matching pictures, suggesting that participants had formed perceptual-like situation models of the sentences. The advantage for matching pictures did not increase with age. Experiment 2 extended these findings to the domain of written language. Participants (ages 7–10 years) of high and low word reading ability verified pictures after reading sentences aloud. The results suggest that even when reading is effortful, children construct a perceptual simulation of the described events. We propose that perceptual simulation plays a more central role in developing language comprehension than was previously thought.     

Time in language: event duration in language comprehension

This work investigates how we process and represent event duration in on-line language comprehension. Specifically, it examines how events of different duration are processed and what type of knowledge underlies their representations. Studies 1–4 examined verbs and phrases in different contexts. They showed that durative events took longer to process than non-durative events and that the duration attributed to the stimulus events correlated with on-line processing times. Studies 5 and 6 indicated that durative events occur in semantically more diverse contexts and elicit semantically more diverse associations than non-durative events. Semantic and contextual diversity also correlated with attributed durations and processing times. Results indicate that (a) event-specific durations are computed on-line from multiple unfolding cues, (b) processing cost and duration representations emerge from semantic and contextual diversity reflecting our experience, and (c) key components of duration representations may be situation-specific knowledge of causal and contingency relations between events.     

The Wind Chilled the Spectators, but the Wine Just Chilled: Sense, Structure, and Sentence Comprehension

Anticipation plays a role in language comprehension. In this article, we explore the extent to which verb sense influences expectations about upcoming structure. We focus on change of state verbs like shatter , which have different senses that are expressed in either transitive or intransitive structures, depending on the sense that is used. In two experiments we influence the interpretation of verb sense by manipulating the thematic fit of the grammatical subject as cause or affected entity for the verb, and test whether readers' expectations for a transitive or intransitive structure change as a result. This sense-biasing context influenced reading times in the postverbal regions. Reading times for transitive sentences were faster following good-cause than good-theme subjects, but the opposite pattern was found for intransitive sentences. We conclude that readers use sense-contingent subcategorization preferences during on-line comprehension.     

Typhoid Pollyanna: The effect of category valence on retrieval order of positive and negative category members

Previous research suggests that people process pleasant information more efficiently than unpleasant information. This phenomenon has been illustrated in a variety of contexts and paradigms, and is frequently referred to as the Pollyanna principle. One important aspect of the Pollyanna principle is that people tend to retrieve pleasant members of a category from semantic memory prior to unpleasant members of the same category. However, we propose that the retrieval advantage of favourably viewed category members holds only for positive categories and that prototypicality of category members might be a better predictor of ease of retrieval than favourability. These possibilities were tested in two studies. In Study 1, favourability effects on retrieval order were observed for positive categories but not for neutral or negative categories. In Study 2, prototypical category members were retrieved before less prototypical category members for both positive and negative categories. These results suggest that the Pollyanna principle might be the result of a confound between favourability and prototypicality for members of favourable categories.     

Discourse-mediation of the mapping between language and the visual world: Eye movements and mental representation

Two experiments explored the mapping between language and mental representations of visual scenes. In both experiments, participants viewed, for example, a scene depicting a woman, a wine glass and bottle on the floor, an empty table, and various other objects. In Experiment 1, participants concurrently heard either ‘The woman will put the glass on the table’ or ‘The woman is too lazy to put the glass on the table’. Subsequently, with the scene unchanged, participants heard that the woman ‘will pick up the bottle, and pour the wine carefully into the glass.’ Experiment 2 was identical except that the scene was removed before the onset of the spoken language. In both cases, eye movements after ‘pour’ (anticipating the glass) and at ‘glass’ reflected the language-determined position of the glass, as either on the floor, or moved onto the table, even though the concurrent (Experiment 1) or prior (Experiment 2) scene showed the glass in its unmoved position on the floor. Language-mediated eye movements thus reflect the real-time mapping of language onto dynamically updateable event-based representations of concurrently or previously seen objects (and their locations).     

Processing of color words activates color representations

Two experiments were conducted to investigate whether color representations are routinely activated when color words are processed. Congruency effects of colors and color words were observed in both directions. Lexical decisions on color words were faster when preceding colors matched the color named by the word. Color-discrimination responses were slowed down when preceding color words mismatched the test color even if no task had to be performed on these words. These findings are consistent with the experiential view of language comprehension according to which color perception and the comprehension of color words are based on overlapping representational resources.     

Language comprehension vs. language production: age effects on fMRI activation

Normal language acquisition is a process that unfolds with amazing speed primarily in the first years of life. However, the refinement of linguistic proficiency is an ongoing process, extending well into childhood and adolescence. An increase in lateralization and a more focussed productive language network have been suggested to be the neural correlates of this process. However, the processes underlying the refinement of language comprehension are less clear. Using a language comprehension (Beep Stories) and a language production (Vowel Identification) task in fMRI, we studied language representation and lateralization in 36 children, adolescents, and young adults (age 6-24 years). For the language comprehension network, we found a more focal activation with age in the bilateral superior temporal gyri. No significant increase of lateralization with age could be observed, so the neural basis of language comprehension as assessed with the Beep Stories task seems to be established in a bilateral network by late childhood. For the productive network, however, we could confirm an increase with age both in focus and lateralization. Only in the language comprehension task did verbal IQ correlate with lateralization, with higher verbal IQ being associated with more right-hemispheric involvement. In some subjects (24%), language comprehension and language production were lateralized to opposite hemispheres.     

Activating event knowledge

An increasing number of results in sentence and discourse processing demonstrate that comprehension relies on rich pragmatic knowledge about real-world events, and that incoming words incrementally activate such knowledge. If so, then even outside of any larger context, nouns should activate knowledge of the generalized events that they denote or typically play a role in. We used short stimulus onset asynchrony priming to demonstrate that (1) event nouns prime people (sale-shopper) and objects (trip-luggage) commonly found at those events; (2) location nouns prime people/animals (hospital-doctor) and objects (barn-hay) commonly found at those locations; and (3) instrument nouns prime things on which those instruments are commonly used (key-door), but not the types of people who tend to use them (hose-gardener). The priming effects are not due to normative word association. On our account, facilitation results from event knowledge relating primes and targets. This has much in common with computational models like LSA or BEAGLE in which one word primes another if they frequently occur in similar contexts. LSA predicts priming for all six experiments, whereas BEAGLE correctly predicted that priming should not occur for the instrument-people relation but should occur for the other five. We conclude that event-based relations are encoded in semantic memory and computed as part of word meaning, and have a strong influence on language comprehension.     

Representing object colour in language comprehension

Embodied theories of cognition hold that mentally representing something red engages the neural subsystems that respond to environmental perception of that colour. This paper examines whether implicit perceptual information on object colour is represented during sentence comprehension even though doing so does not necessarily facilitate task performance. After reading a sentence that implied a particular colour for a given object, participants were presented with a picture of the object that either matched or mismatched the implied colour. When asked if the pictured object was mentioned in the preceding sentence, people's responses were faster when the colours mismatched than when they matched, suggesting that object colour is represented differently to other object properties such as shape and orientation. A distinction between stable and unstable embodied representations is proposed to allow embodied theories to account for these findings.     

The role of illustrations in children’s inferential comprehension

Illustrations are a salient source of information in children’s books, yet their effect on children’s reading comprehension has been studied only through literal factual recall. The purpose of the current study was to determine the effect of illustrations on bridging inferences, an important aspect of meaning making in comprehension models. Identical short stories were presented under different illustration conditions with pictures that represented different parts of the story. Participants were 73 7- to 11-year-olds. Illustrations both facilitated and interfered with inferencing depending on the type of information depicted; however, this effect was reduced as grade increased. Additional findings were that the overall ability to make inferences increased with age and working memory was a significant predictor of this skill. Results are discussed in relation to cognitive and developmental models of comprehension.