The persistence of object file representations

Coherent visual experience of dynamic scenes requires not only that the visual system segment scenes into component objects but that these object representations persist, so that an object can be identified as the same object from an earlier time. Object files (OFs) are visual representations thought to mediate such abilities: OFs lie between lower level sensory processing and higher level recognition, and they track salient objects over time and motion. OFs have traditionally been studied via object-specific preview benefits (OSPBs), in which discriminations of an object's features are speeded when an earlier preview of those features occurred on the same object, as opposed to on a different object, beyond general displaywide priming. Despite its popularity, many fundamental aspects of the OF framework remain unexplored. For example, although OFs are thought to be involved primarily in online visual processing, we do not know how long such representations persist; previous studies found OSPBs for up to 1500 msec but did not test for longer durations. We explored this issue using a modified object reviewing paradigm and found that robust OSPBs persist for more than five times longer than has previously been tested-for at least 8 sec, and possibly for much longer. Object files may be the "glue" that makes visual experience coherent not just in online moment-by-moment processing, but on the scale of seconds that characterizes our everyday perceptual experiences. These findings also bear on research in infant cognition, where OFs are thought to explain infants' abilities to track and enumerate small sets of objects over longer durations.     

Cohesion as a constraint on object persistence in infancy

A critical challenge for visual perception is to represent objects as the same persisting individuals over time and motion. Across several areas of cognitive science, researchers have identified cohesion as among the most important theoretical principles of object persistence: An object must maintain a single bounded contour over time. Drawing inspiration from recent work in adult visual cognition, the present study tested the power of cohesion as a constraint as it operates early in development. In particular, we tested whether the most minimal cohesion violation - a single object splitting into two - would destroy infants' ability to represent a quantity of objects over occlusion. In a forced-choice crawling paradigm, 10- and 12-month-old infants witnessed crackers being sequentially placed into containers, and typically crawled toward the container with the greater cracker quantity. When one of the crackers was visibly split in half, however, infants failed to represent the relative quantities, despite controls for the overall quantities and the motions involved. This result helps to characterize the fidelity and specificity of cohesion as a fundamental principle of object persistence, suggesting that even the simplest possible cohesion violation can dramatically impair infants' object representations and influence their overt behavior.     

See an object,hear an object file: Object correspondence transcends sensory modality

An important task of perceptual processing is to parse incoming information into distinct units and to keep track of those units over time as the same, persisting representations. Within the study of visual perception, maintaining such persisting object representations is helped by “object files”—episodic representations that store (and update) information about objects' properties and track objects over time and motion via spatiotemporal information. Although object files are typically discussed as visual, here we demonstrate that object–file correspondence can be computed across sensory modalities. An object file can be initially formed with visual input and later accessed with corresponding auditory information, suggesting that object files may be able to operate at a multimodal level of perceptual processing.     

Attentive tracking of objects versus substances

Recent research in vision science, infant cognition, and word learning suggests a special role for the processing of discrete objects. But what counts as an object? Answers to this question often depend on contrasting object-based processing with the processing of spatial areas or unbound visual features. In infant cognition and word learning, though, another salient contrast has been between rigid cohesive objects and nonsolid substances. Whereas objects may move from one location to another, a nonsolid substance must pour from one location to another. In the study reported here, we explored whether attentive tracking processes are sensitive to dynamic information of this type. Using a multiple-object tracking task, we found that subjects could easily track four items in a display of eight identical unpredictably moving entities that moved as discrete objects from one location to another, but could not track similar entities that noncohesively "poured" from one location to another-even when the items in both conditions followed the same trajectories at the same speeds. Other conditions revealed that this inability to track multiple "substances" stemmed not from violations of rigidity or cohesiveness per se, because subjects were able to track multiple noncohesive collections and multiple nonrigid deforming objects. Rather, the impairment was due to the dynamic extension and contraction during the substancelike motion, which rendered the location of the entity ambiguous. These results demonstrate a convergence between processes of midlevel adult vision and infant cognition, and in general help to clarify what can count as a persisting dynamic object of attention.     

Object files can be purely episodic

Our ability to track an object as the same persisting entity over time and motion may primarily rely on spatiotemporal representations which encode some, but not all, of an object's features. Previous researchers using the 'object reviewing' paradigm have demonstrated that such representations can store featural information of well-learned stimuli such as letters and words at a highly abstract level. However, it is unknown whether these representations can also store purely episodic information (i.e. information obtained from a single, novel encounter) that does not correspond to pre-existing type-representations in long-term memory. Here, in an object-reviewing experiment with novel face images as stimuli, observers still produced reliable object-specific preview benefits in dynamic displays: a preview of a novel face on a specific object speeded the recognition of that particular face at a later point when it appeared again on the same object compared to when it reappeared on a different object (beyond display-wide priming), even when all objects moved to new positions in the intervening delay. This case study demonstrates that the mid-level visual representations which keep track of persisting identity over time--e.g. 'object files', in one popular framework can store not only abstract types from long-term memory, but also specific tokens from online visual experience.     

Are objects required for object-files? Roles of segmentation and spatiotemporal continuity in computing object persistence

Two central tasks of visual processing are (1) to segment undifferentiated retinal images into discrete objects, and (2) to represent those objects as the same persisting individuals over time and motion. Here we explore the interaction of these two types of processing in the context of object files—mid-level visual representations that “stick” to moving objects on the basis of spatiotemporal properties. Object files can be revealed by object-specific preview benefits (OSPBs), wherein a “preview” of information on a moving object speeds the recognition of that information at a later point when it appears again on the same object (compared to when it reappears on a different moving object), beyond display-wide priming. Here we explore the degree of segmentation required to establish object files in the first place. Surprisingly, we find that no explicit segmentation is required until after the previews disappear, when using purely motion-defined objects (consisting of random elements on a random background). Moreover, OSPBs are observed in such displays even after moderate (but not long) delays between the offset of the preview information and the onset of the motion. These effects indicate that object files can be established without initial static segmentation cues, so long as there is spatiotemporal continuity between the previews and the eventual appearance of the objects. We also find that top-down strategies can sometimes mimic OSPBs, but that these strategies can be eliminated by novel manipulations. We discuss how these results alter our understanding of the nature of object files, and also why researchers must take care to distinguish “true OSPBs” from “illusory OSPBs”.     

Object-based attention: strength of object representation and attentional guidance

Two or more features belonging to a single object are identified more quickly and more accurately than are features belonging to different objects--a finding attributed to sensory enhancement of all features belonging to an attended or selected object. However, several recent studies have suggested that this "single-object advantage" may be a product of probabilistic and configural strategic prioritizations rather than of object-based perceptual enhancement per se, challenging the underlying mechanism that is thought to give rise to object-based attention. In the present article, we further explore constraints on the mechanisms of object-based selection by examining the contribution of the strength of object representations to the single-object advantage. We manipulated factors such as exposure duration (i.e., preview time) and salience of configuration (i.e., objects). Varying preview time changes the magnitude of the object-based effect, so that if there is ample time to establish an object representation (i.e., preview time of 1,000 msec), then both probability and configuration (i.e., objects) guide attentional selection. If, however, insufficient time is provided to establish a robust object-based representation, then only probabilities guide attentional selection. Interestingly, at a short preview time of 200 msec, when the two objects were sufficiently different from each other (i.e., different colors), both configuration and probability guided attention selection. These results suggest that object-based effects can be explained both in terms of strength of object representations (established at longer exposure durations and by pictorial cues) and probabilistic contingencies in the visual environment.     

Interrupting infants’ persisting object representations: an object‐based limit?

Making sense of the visual world requires keeping track of objects as the same persisting individuals over time and occlusion. Here we implement a new paradigm using 10-month-old infants to explore the processes and representations that support this ability in two ways. First, we demonstrate that persisting object representations can be maintained over brief interruptions from additional independent events - just as a memory of a traffic scene may be maintained through a brief glance in the rearview mirror. Second, we demonstrate that this ability is nevertheless subject to an object-based limit: if an interrupting event involves enough objects (carefully controlling for overall salience), then it will impair the maintenance of other persisting object representations even though it is an independent event. These experiments demonstrate how object representations can be studied via their 'interruptibility', and the results are consistent with the idea that infants' persisting object representations are constructed and maintained by capacity-limited mid-level 'object-files'.     

The experience of force: the role of haptic experience of forces in visual perception of object motion and interactions, mental simulation, and motion-related judgments

Forces are experienced in actions on objects. The mechanoreceptor system is stimulated by proximal forces in interactions with objects, and experiences of force occur in a context of information yielded by other sensory modalities, principally vision. These experiences are registered and stored as episodic traces in the brain. These stored representations are involved in generating visual impressions of forces and causality in object motion and interactions. Kinematic information provided by vision is matched to kinematic features of stored representations, and the information about forces and causality in those representations then forms part of the perceptual interpretation. I apply this account to the perception of interactions between objects and to motions of objects that do not have perceived external causes, in which motion tends to be perceptually interpreted as biological or internally caused. I also apply it to internal simulations of events involving mental imagery, such as mental rotation, trajectory extrapolation and judgment, visual memory for the location of moving objects, and the learning of perceptual judgments and motor skills. Simulations support more accurate judgments when they represent the underlying dynamics of the event simulated. Mechanoreception gives us whatever limited ability we have to perceive interactions and object motions in terms of forces and resistances; it supports our practical interventions on objects by enabling us to generate simulations that are guided by inferences about forces and resistances, and it helps us learn novel, visually based judgments about object behavior.     

A temporal same-object advantage in the tunnel effect: facilitated change detection for persisting objects

Meaningful visual experience requires computations that identify objects as the same persisting individuals over time, motion, occlusion, and featural change. This article explores these computations in the tunnel effect: When an object moves behind an occluder, and then an object later emerges following a consistent trajectory, observers irresistibly perceive a persisting object, even when the pre- and postocclusion views contrast featurally. This article introduces a new change detection method for quantifying percepts of the tunnel effect. Observers had to detect color changes in displays where several objects oscillated behind occluders and occasionally changed color. Across comparisons with several types of spatiotemporal gaps, as well as manipulations of occlusion versus implosion, performance was better when objects' kinematics gave the impression of a persisting individual. The results reveal a temporal same-object advantage: better change detection across temporal scene fragments bound into the same persisting object representations. This suggests that persisting objects are the underlying units of visual memory.     

Features,as well as space and time,guide object persistence

What role do surface features (e.g., color) play in the establishment and maintenance of episodic representations of objects (object files)? Mitroff and Alvarez (2007) showed that stimuli that were linked by a continuous spatiotemporal history yielded object-specific preview benefits—a standard index of object files—whereas stimuli linked only by shared surface features did not. Here, it is shown that abruptly changing the features of an object that has been established on the basis of spatiotemporal history can disrupt object-specific preview benefits (Experiments 1 and 2). Moreover, under some conditions, feature match alone can give rise to the preview benefits (Experiment 3). These results indicate that surface features, as well as spatiotemporal factors, play an important role in establishing and maintaining episodic object representations.     

The relationship between apparent motion and object files

Object files (OFs) play an important role in theories of mid-level vision. On some influential views, OFs operate and persist only via spatiotemporal continuity. One open question concerns what occurs when direct spatiotemporal continuity is absent: Do OFs move in accordance with any motion correspondence ultimately resolved? Specifically, do OFs accord with apparent motion (AM) correspondences, which arise despite a lack of continuous spatiotemporal stimulation? In Experiment 1, subjects were presented with an AM display consisting of two circles that, across two frames, were seen as moving between two noncontiguous locations. The two objects were primed with two symbols and were then moved in a single step; a third symbol appeared, and could either match the symbol from the closer or the further object. We found a robust object specific preview benefit (OSPB) for the shorter path, in other words, the path along which AM was perceived. In order to control for the possibility that priming occurs at any nearby object, in Experiment 2, the original two objects never disappeared, but two new objects appeared in the would-be AM locations. No AM was perceived, and no OSPB obtained. In the third experiment the OSPB effect persisted even when motion along the shorter path included an unlikely featural transformation (circles turning into squares). In Experiment 4, which was nearly identical to Experiment 2, no OSPB obtained despite unique featural matches between the initial and new objects, seemingly because no AM was perceived. In Experiment 5, we failed to find an effect of featural priming, even when distance between the objects was equated. Finally, we extended the OSPB effect to two additional kinds of AM—line motion (Experiment 6) and phi motion (Experiment 7), supplying strong evidence that AM correspondences and OF correspondences are controlled by the same basic rules.     

Visual working memory for dynamic objects: Impaired binding between object feature and location

In daily life, people frequently need to observe dynamic objects and temporarily maintain their representations in visual working memory (VWM). The present study explored the mechanism underlying the binding between perceptual features and locations of dynamic objects in VWM. In three experiments, we measured and compared the memory performance for feature-location binding of multiple dynamic and static objects. The results showed that the feature-location binding was impaired for the dynamic objects compared with the static objects. The impairment persisted when the global spatial configuration of the objects remained intact during the motion, as well as when the binding task was relatively easy, such as binding between single-feature objects and coarse locations. The results indicate that object features and locations are not maintained in VWM as well-integrated object files; rather, the formation of feature-location binding may require additional processes, which are disrupted by the constant change of locations in dynamic circumstances. We propose a consolidation process as possible underlying mechanism, and discuss factors that may influence the strength of feature-location binding in dynamic circumstances.     

Episodic representation of diagnostic and nondiagnostic object colour

In three experiments, we investigated transsaccadic object file representations. In each experiment, participants moved their eyes from a central fixation cross to a saccade target located between two peripheral objects. During the saccade, this preview display was replaced with a target display containing a single object to be named. On trials in which the target identity matched one of the preview objects, its colour either matched or did not match the previewed object colour. The results indicated that colour changes disrupt perceptual continuity, but only for the class of objects for which colour is diagnostic of object identity. When the colour is not integral to identifying an object (for example, when the object is a letter or an object without a characteristic colour), object continuity is preserved regardless of changes to the object's colour. These results suggest that object features that are important for defining the object are incorporated into its episodic representation. Furthermore, the results are consistent with previous work showing that the quality of a feature's representation determines its importance in preserving continuity.     

The relationship between object files and conscious perception

Object files (OFs) are hypothesized mid-level representations which mediate our conscious perception of persisting objects-e.g. telling us 'which went where'. Despite the appeal of the OF framework, not previous research has directly explored whether OFs do indeed correspond to conscious percepts. Here we present at least one case wherein conscious percepts of 'which went where' in dynamic ambiguous displays diverge from the analogous correspondence computed by the OF system. Observers viewed a 'bouncing/streaming' display in which two identical objects moved such that they could have either bounced off or streamed past each other. We measured two dependent variables: (1) an explicit report of perceived bouncing or streaming; and (2) an implicit 'object-specific preview benefit' (OSPB), wherein a 'preview' of information on a specific object speeds the recognition of that information at a later point when it appears again on the same object (compared to when it reappears on a different object), beyond display-wide priming. When the displays were manipulated such that observers had a strong bias to perceive streaming (on over 95% of the trials), there was nevertheless a strong OSPB in the opposite direction-such that the object files appeared to have 'bounced' even though the percept 'streamed'. Given that OSPBs have been taken as a hallmark of the operation of object files, the five experiments reported here suggest that in at least some specialized (and perhaps ecologically invalid) cases, conscious percepts of 'which went where' in dynamic ambiguous displays can diverge from the mapping computed by the object-file system.     

Spatio-temporal priority revisited: the role of feature identity and similarity for object correspondence in apparent motion

We live in a dynamic environment in which objects change location over time. To maintain stable object representations the visual system must determine how newly sampled information relates to existing object representations, the correspondence problem. Spatiotemporal information is clearly an important factor that the visual system takes into account when solving the correspondence problem, but is feature information irrelevant as some theories suggest? The Ternus display provides a context in which to investigate solutions to the correspondence problem. Two sets of three horizontally aligned disks, shifted by one position, were presented in alternation. Depending on how correspondence is resolved, these displays are perceived either as one disk "jumping" from one end of the row to the other (element motion) or as a set of three disks shifting back and forth together (group motion). We manipulated a feature (e.g., color) of the disks such that, if features were taken into account by the correspondence process, it would bias the resolution of correspondence toward one version or the other. Features determined correspondence, whether they were luminance-defined or not. Moreover, correspondence could be established on the basis of similarity, when features were not identical between alternations. Finally, the stronger the feature information supported a certain correspondence solution the more it dominated spatiotemporal information.     

Transsaccadic perception of saccade target and flanker objects

To account for location-dependent and location-independent preview benefits in transsaccadic object perception, J. M. Henderson (1994) and J. M. Henderson and M. D. Anes (1994) proposed a dual-route model in which both episodic object representations and long-term memory representations store information across a saccade. Four experiments are reported in which the dual-route model was assessed. Preview benefits for saccade target objects were found to be location independent, whereas preview benefits for flanker objects were location dependent. These results support a single-route, 2-stage model of transsaccadic object perception. First, preattentive object files are set up to parse a set of attentional and/or saccade targets from peripheral vision, causing location-dependent preview benefits. Second, 1 object is attentionally selected for further processing, activating long-term memory representations and resulting in location-independent preview benefits.     

Attention to memory: orienting attention to sound object representations

Despite a growing acceptance that attention and memory interact, and that attention can be focused on an active internal mental representation (i.e., reflective attention), there has been a paucity of work focusing on reflective attention to ‘sound objects’ (i.e., mental representations of actual sound sources in the environment). Further research on the dynamic interactions between auditory attention and memory, as well as its degree of neuroplasticity, is important for understanding how sound objects are represented, maintained, and accessed in the brain. This knowledge can then guide the development of training programs to help individuals with attention and memory problems. This review article focuses on attention to memory with an emphasis on behavioral and neuroimaging studies that have begun to explore the mechanisms that mediate reflective attentional orienting in vision and more recently, in audition. Reflective attention refers to situations in which attention is oriented toward internal representations rather than focused on external stimuli. We propose four general principles underlying attention to short-term memory. Furthermore, we suggest that mechanisms involved in orienting attention to visual object representations may also apply for orienting attention to sound object representations.     

Space and time,not surface features,guide object persistence

Successful visual perception relies on the ability to keep track of distinct entities as the same persisting objects from one moment to the next. This is a computationally difficult process and its underlying nature remains unclear. Here we use the object file framework to explore whether surface feature information (e.g., color, shape) can be used to compute such object persistence. From six experiments we find that spatiotemporal information (location as a function of time) easily determines object files, but surface features do not. The results suggest an unexpectedly strong constraint on the visual system’s ability to compute online object persistence.     

A theory of dynamic occluded and illusory object perception

Humans see whole objects from input fragmented in space and time, yet spatiotemporal object perception is poorly understood. The authors propose the theory of spatiotemporal relatability (STR), which describes the visual information and processes that allow visible fragments revealed at different times and places, due to motion and occlusion, to be assembled into unitary perceived objects. They present a formalization of STR that specifies spatial and temporal relations for object formation. Predictions from the theory regarding conditions that lead to unit formation were tested and confirmed in experiments with dynamic and static, occluded and illusory objects. Moreover, the results support the identity hypothesis of a common process for amodal and modal contour interpolation and provide new evidence regarding the relative efficiency of static and dynamic object formation. STR postulates a mental representation, the dynamic visual icon, that briefly maintains shapes and updates positions of occluded fragments to connect them with visible regions. The theory offers a unified account of interpolation processes for static, dynamic, occluded, and illusory objects.