Constructing visual representations of natural scenes: the roles of short- and long-term visual memory

A "follow-the-dot" method was used to investigate the visual memory systems supporting accumulation of object information in natural scenes. Participants fixated a series of objects in each scene, following a dot cue from object to object. Memory for the visual form of a target object was then tested. Object memory was consistently superior for the two most recently fixated objects, a recency advantage indicating a visual short-term memory component to scene representation. In addition, objects examined earlier were remembered at rates well above chance, with no evidence of further forgetting when 10 objects intervened between target examination and test and only modest forgetting with 402 intervening objects. This robust prerecency performance indicates a visual long-term memory component to scene representation.     

Object files across eye movements: Previous fixations affect the latencies of corrective saccades

One of the factors contributing to a seamless visual experience is object correspondence—that is, the integration of pre- and postsaccadic visual object information into one representation. Previous research had suggested that before the execution of a saccade, a target object is loaded into visual working memory and subsequently is used to locate the target object after the saccade. Until now, studies on object correspondence have not taken previous fixations into account. In the present study, we investigated the influence of previously fixated information on object correspondence. To this end, we adapted a gaze correction paradigm in which a saccade was executed toward either a previously fixated or a novel target. During the saccade, the stimuli were displaced such that the participant’s gaze landed between the target stimulus and a distractor. Participants then executed a corrective saccade to the target. The results indicated that these corrective saccades had lower latencies toward previously fixated than toward nonfixated targets, indicating object-specific facilitation. In two follow-up experiments, we showed that presaccadic spatial and object (surface feature) information can contribute separately to the execution of a corrective saccade, as well as in conjunction. Whereas the execution of a corrective saccade to a previously fixated target object at a previously fixated location is slowed down (i.e., inhibition of return), corrective saccades toward either a previously fixated target object or a previously fixated location are facilitated. We concluded that corrective saccades are executed on the basis of object files rather than of unintegrated feature information.     

The effect of spatial and nonspatial contextual information on visual object memory

Recent research has found visual object memory can be stored as part of a larger scene representation rather than independently of scene context. The present study examined how spatial and nonspatial contextual information modulate visual object memory. Two experiments tested participants’ visual memory by using a change detection task in which a target object's orientation was either the same as it appeared during initial viewing or changed. In addition, we examined the effect of spatial and nonspatial contextual manipulations on change detection performance. The results revealed that visual object representations can be maintained reliably after viewing arrays of objects. Moreover, change detection performance was significantly higher when either spatial or nonspatial contextual information remained the same in the test image. We concluded that while processing complex visual stimuli such as object arrays, visual object memory can be stored as part of a comprehensive scene representation, and both spatial and nonspatial contextual changes modulate visual memory retrieval and comparison.     

Failures of retrieval and comparison constrain change detection in natural scenes

In a change detection paradigm, a target object in a natural scene either rotated in depth, was replaced by another object token, or remained the same. Change detection performance was reliably higher when a target postcue allowed participants to restrict retrieval and comparison processes to the target object (Experiment 1). Change detection performance remained excellent when the target object was not attended at change (Experiment 2) and when a concurrent verbal working memory load minimized the possibility of verbal encoding (Experiment 3). Together, these data demonstrate that visual representations accumulate in memory from attended objects as the eyes and attention are oriented within a scene and that change blindness derives, at least in part, from retrieval and comparison failure.     

Object appearance,disappearance, and attention prioritization in real-world scenes

We examined the prioritization of abruptly appearing and disappearing objects in real-world scenes. These scene changes occurred either during a fixation (transient appearance/disappearance) or during a saccade (nontransient appearance/disappearance). Prioritization was measured by the eyes’ propensity to be directed to the region of the scene change. Object additions and deletions were fixated at rates greater than chance, suggesting that both types of scene change are cues used by the visual system to guide attention during scene exploration, although appearances were fixated twice as often as disappearances, indicating that new objects are more salient than deleted objects. New and deleted objects were prioritized sooner and more frequently if they occurred during a fixation, as compared with during a saccade, indicating an important role of the transient signal that often accompanies sudden changes in scenes. New objects were prioritized regardless of whether they appeared during a fixation or a saccade, whereas prioritization of a deleted object occurred only if (1) a transient signal was present or (2) the removal of the object revealed previously occluded objects.     

To see and remember: visually specific information is retained in memory from previously attended objects in natural scenes

What is the nature of the representation formed during the viewing of natural scenes? We tested two competing hypotheses regarding the accumulation of visual information during scene viewing. The first holds that coherent visual representations disintegrate as soon as attention is withdrawn from an object and thus that the visual representation of a scene is exceedingly impoverished. The second holds that visual representations do not necessarily decay upon the withdrawal of attention, but instead can be accumulated in memory from previously attended regions. Target objects in line drawings of natural scenes were changed during a saccadic eye movement away from those objects. Three findings support the second hypothesis. First, changes to the visual form of target objects (token substitution) were successfully detected, as indicated by both explicit and implicit measures, even though the target object was not attended when the change occurred. Second, these detections were often delayed until well after the change. Third, changes to semantically inconsistent target objects were detected better than changes to semantically consistent objects.     

The relationship between online visual representation of a scene and long-term scene memory

In 3 experiments the author investigated the relationship between the online visual representation of natural scenes and long-term visual memory. In a change detection task, a target object either changed or remained the same from an initial image of a natural scene to a test image. Two types of changes were possible: rotation in depth, or replacement by another object from the same basic-level category. Change detection during online scene viewing was compared with change detection after delay of 1 trial (Experiments 2A and 2B) until the end of the study session (Experiment 1) or 24 hr (Experiment 3). There was little or no decline in change detection performance from online viewing to a delay of 1 trial or delay until the end of the session, and change detection remained well above chance after 24 hr. These results demonstrate that long-term memory for visual detail in a scene is robust.     

Eye movements and picture processing during recognition

Eye movements were monitored during a recognition memory test of previously studied pictures of full-color scenes. The test scenes were identical to the originals, had an object deleted from them, or included a new object substituted for an original object. In contrast to a prior report (Parker, 1978), we found no evidence that object deletions or substitutions could be recognized on the basis of information acquired from the visual periphery. Deletions were difficult to recognize peripherally, and the eyes were not attracted to them. Overall, the amplitude of the average saccade to the critical object during the memory test was less than 4.5 degrees of visual angle in all conditions and averaged 4.1 degrees across conditions. We conclude that information about object presence and identity in a scene is limited to a relatively small region around the current fixation point.     

Object-position binding in visual memory for natural scenes and object arrays

Nine experiments examined the means by which visual memory for individual objects is structured into a larger representation of a scene. Participants viewed images of natural scenes or object arrays in a change detection task requiring memory for the visual form of a single target object. In the test image, 2 properties of the stimulus were independently manipulated: the position of the target object and the spatial properties of the larger scene or array context. Memory performance was higher when the target object position remained the same from study to test. This same-position advantage was reduced or eliminated following contextual changes that disrupted the relative spatial relationships among contextual objects (context deletion, scrambling, and binding change) but was preserved following contextual change that did not disrupt relative spatial relationships (translation). Thus, episodic scene representations are formed through the binding of objects to scene locations, and object position is defined relative to a larger spatial representation coding the relative locations of contextual objects.     

Scene and position specificity in visual memory for objects

This study investigated whether and how visual representations of individual objects are bound in memory to scene context. Participants viewed a series of naturalistic scenes, and memory for the visual form of a target object in each scene was examined in a 2-alternative forced-choice test, with the distractor object either a different object token or the target object rotated in depth. In Experiments 1 and 2, object memory performance was more accurate when the test object alternatives were displayed within the original scene than when they were displayed in isolation, demonstrating object-to-scene binding. Experiment 3 tested the hypothesis that episodic scene representations are formed through the binding of object representations to scene locations. Consistent with this hypothesis, memory performance was more accurate when the test alternatives were displayed within the scene at the same position originally occupied by the target than when they were displayed at a different position.     

The Role of Fixation Position in Detecting Scene Changes Across Saccades

Target objects presented within color images of naturalistic scenes were deleted or rotated during a saccade to or from the target object or to a control region of the scene. Despite instructions to memorize the details of the scenes and to monitor for object changes, viewers frequently failed to notice the changes. However, the failure to detect change was mediated by three other important factors: First, accuracy generally increased as the distance between the changing region and the fixation immediately before or after the change decreased. Second, changes were sometimes initially missed, but subsequently noticed when the changed region was later refixated. Third, when an object disappeared from a scene, detection of that disappearance was greatly improved when the deletion occurred during the saccade toward that object. These results suggest that fixation position and saccade direction play an important role in determining whether changes will be detected. It appears that more information can be retained across views than has been suggested by previous studies.     

Saccades elicit obligatory allocation of visual working memory

In daily life, visual working memory (VWM) typically works in contexts in which people make frequent saccades. Here, we investigated whether people can effectively control the allocation of VWM when making a saccade. Subjects were required to make an intervening saccade in the process of a memory task. The saccade target was either a to-be-remembered item or an extra, not-to-be-remembered item. The results showed that memory performance was poorer when a saccade was made to the extra, not-to-be-remembered item, regardless of its similarity to the memory item(s). In contrast, when memorizing the items while remaining fixated, subjects had similar memory performance whether an extra, not-to-be-remembered item was present or not. Taken together, these results demonstrated that volitional control over VWM allocation is greatly impaired when a saccade is made, indicating that VWM contains an automatic part that cooperates with eye movements and is allocated to a saccade target obligatorily.     

The properties of object representations constructed during visual search in natural scenes

To investigate properties of object representations constructed during a visual search task, we manipulated the proportion of trials/task within a block: In a search-frequent block, 80% of trials were search tasks; remaining trials presented a memory task; in a memory-frequent block, this proportion was reversed. In the search task, participants searched for a toy car (Experiments 1 and 2) or a T-shape object (Experiment 3). In the memory task, participants had to memorize objects in a scene. Memory performance was worse in the search-frequent block than in the memory-frequent block in Experiments 1 and 3, but not in Experiment 2 (token change in Experiment 1; type change in Experiments 2 and 3). Experiment 4 demonstrated that lower performance in the search-frequent block was not due to eye-movement behaviour. Results suggest that object representations constructed during visual search are different from those constructed during memorization and they are modulated by type of target.     

The role of the saccade target object in the perception of a visually stable world

Although the proximal stimulus shifts position on our retinae with each saccade, we perceive our world as stable and continuous. Most theories of visual stability implicitly assume a mechanism that spatially adjusts perceived locations associated with the retinal array by using, as a parameter, extra-retinal eye position information, a signal that encodes the size and direction of the saccade. The results from the experiment reported in this article challenge this idea. During a participant's saccade to a target object, one of the following was displaced: the entire scene, the target object, or the background behind the target object. Participants detected the displacement of the target object twice as frequently as the displacement of the entire background. The direction of displacement relative to the saccade also affected detectability. We use a new theory, the saccade target theory (McConkie & Currie, 1996), to interpret these results. This theory proposes that retinal (as opposed to extra-retinal) factors, primarily those concerning the saccade target object, are critical for the detection of intrasaccadic stimulus shifts.     

Does conspicuity enhance distraction? Saliency and eye landing position when searching for objects

While visual saliency may sometimes capture attention, the guidance of eye movements in search is often dominated by knowledge of the target. How is the search for an object influenced by the saliency of an adjacent distractor? Participants searched for a target amongst an array of objects, with distractor saliency having an effect on response time and on the speed at which targets were found. Saliency did not predict the order in which objects in target-absent trials were fixated. The within-target landing position was distributed around a modal position close to the centre of the object. Saliency did not affect this position, the latency of the initial saccade, or the likelihood of the distractor being fixated, suggesting that saliency affects the allocation of covert attention and not just eye movements.     

Testing a conceptual locus

Changes to objects that are inconsistent with the scene in which they appear are detected more accurately than changes to consistent objects. In three experiments, we tested whether this inconsistent object advantage derives from the differential retention of conceptual codes generated from a brief view of a real-world scene in accordance with a conceptual short-term memory (CSTM) hypothesis. A scene was presented for 250 msec, followed by a brief mask and a test scene in which a target object was either changed or not changed. In Experiment 1, changes that altered conceptual content (object deletion) were contrasted with visual changes (left-right orientation changes). In Experiment 2, the duration of the mask was manipulated to vary the amount of time available for conceptual consolidation of the initial scene. In Experiment 3, the type of mask was manipulated: Either a meaningless pattern mask or a meaningful, and thus conceptually disruptive, scene was shown. The inconsistent object advantage was obtained in each experiment, yet in none was it modulated in the direction predicted by the CSTM hypothesis. Instead, the inconsistent object advantage is likely to be caused by contextual influence on memory for visual object representations.     

The role of saccade targeting in the transsaccadic integration of object types and tokens

The presence of location-dependent and location-independent benefits on object identification in an eye movement contingent preview paradigm has been taken as support for the transsaccadic integration of object types and object tokens (J. M. Henderson, 1994). A recent study, however, suggests a critical role for saccade targeting in the generation of the 2 preview effects (F. Germeys, De Graef, & Verfaillie, 2002). In the present study, eye movements were monitored in a preview paradigm, and both location-independent and location-dependent preview benefits were observed regardless of the saccade target status of the preview object. The findings support the view that type and token representational systems contribute independently to the integration of object information across eye movements.     

Long-lasting modifications of saccadic eye movements following adaptation induced in the double-step target paradigm

The adaptation of saccadic eye movements to environmental changes occurring throughout life is a good model of motor learning and motor memory. Numerous studies have analyzed the behavioral properties and neural substrate of oculomotor learning in short-term saccadic adaptation protocols, but to our knowledge, none have tested the persistence of the oculomotor memory. In the present study, the double-step target protocol was used in five human subjects to adaptively decrease the amplitude of reactive saccades triggered by a horizontally-stepping visual target. We tested the amplitude of visually guided saccades just before and at different times (up to 19 days) after the adaptation session. The results revealed that immediately after the adaptation session, saccade amplitude was significantly reduced by 22% on average. Although progressively recovering over days, this change in saccade gain was still statistically significant on days 1 and 5, with an average retention rate of 36% and 19%, respectively. On day 11, saccade amplitude no longer differed from the pre-adaptation value. Adaptation was more effective and more resistant to recovery for leftward saccades than for rightward ones. Lastly, modifications of saccade gain related to adaptation were accompanied by a decrease of both saccade duration and peak velocity. A control experiment indicated that all these findings were specifically related to the adaptation protocol, and further revealed that no change in the main sequence relationships could be specifically related to adaptation. We conclude that in humans, the modifications of saccade amplitude that quickly develop during a double-step target adaptation protocol can remain in memory for a much longer period of time, reflecting enduring plastic changes in the brain.     

How Are Different Properties of a Scene Encoded in Visual Memory?

Recent studies in scene perception suggest that much of what observers believe they see is not retained in visual memory. Depending on the roles they play in organizing the perception of a scene, different visual properties may require different amounts of attention to be incorporated into a mental representation of the scene. The goal of this study was to compare how three visual properties of scenes, colour, object position, and object presence, are encoded in visual memory. We used a variation on the change detection “flicker” task and measured the time to detect scene changes when: (1) a cue was provided regarding the type of change; and, (2) no cue was provided. We hypothesized that cueing would enhance the processing of visual properties that require more attention to be encoded into scene representations, whereas cueing would not have an effect for properties that are readily or automatically encoded in visual memory. In Experiment 1, we found that there was a cueing advantage for colour changes, but not for position or presence changes. In Experiment 2, we found the same cueing effect regardless of whether the colour change altered the configuration of the scene or not. These results are consistent with the idea that properties that typically help determine the configuration of the scene, for example, position and presence, are better encoded in scene representations than are surface properties such as colour.     

Effects of foveal priming and extrafoveal preview on object identification

The results of three different experiments suggested that the relation between an object in the fovea on fixation n and an object subsequently brought into the fovea on fixation n + 1 affects the time to identify the second object. In Experiment 1 we extended previous work by demonstrating that a previously seen related priming object speeded the time to name a target object even when a saccade intervened between the two objects. In Experiment 2 we replicated this result and further showed that the benefit on naming time was due to facilitation from the related object rather than inhibition from the unrelated object. In addition, naming of the target object was much slower in both experiments when there was not a peripheral preview of the target object on fixation n. However, because the effect of the foveal priming object was greater when the target was not present than when it was present, priming did not appear to make extraction of the extrafoveal information more efficient. In Experiment 3, fixation times were recorded while subjects looked at four objects in order to identify them. Fixation time on an object was shorter when a related object was fixated immediately before it, even though the four objects did not form a scene. The size of the facilitation was roughly comparable to that in several analogous experiments where scenes were used. The results suggest that the effects of a predictive scene context on object identification may be explainable in terms of an object-to-object or "intralevel" priming mechanism.