Encoding color and shape from different parts of an object in visual short-term memory

Can we find an object-based encoding benefit in visual short-term memory (VSTM) when the features to be remembered are from different parts of an object? Using object parts defined by either figure-ground separation or negative minima of curvature, results from five experiments in which the visual change detection paradigm was used showed that the object-based encoding benefit in VSTM is modulated by how features are assigned to parts of an object: Features are best retained when the color and shape features to be remembered belong to the same part of an object. Although less well retained in comparison, features from different parts of an object are still better remembered than features from spatially separated objects. An object-based feature binding therefore exists even when the color and shape features to be remembered are from different parts of an object.     

Surface assignment modulates object formation for visual short-term memory

Recent results have suggested that the operational units of visual short-term memory (VSTM) are whole objects, rather than features or the total amount of information to be remembered. Here, for the first time, the influence of surface assignment on object formation for VSTM was investigated. The observers had to memorize the features of four briefly presented (300 ms) two-part objects, followed by a mask and a cue indicating which object to report on. The experiments contrasted whether there were any apparent depth differences between the two parts of each object, and whether observers had to report on only one or both features of the post-cued target object. Depth differences induced with stereoscopic disparity, and with a pictorial depth cue (simple interposition of object features), interfered strongly with performance when both features of an object needed to be memorized, but aided performance when only a single feature needed to be remembered. Furthermore, there was considerable within-feature interference consistent with some previous findings, but contradicting others. The potential implications for conceptions of VSTM are discussed in the light of two hypothesized stages: an early feature-based stage, as well as a higher-level object-based stage where the depth manipulations exert their effects. The results argue for a strong modulatory influence of surface assignment on object formation for a VSTM task.     

The capacity of visual short-term memory within and between hemifields

Visual short-term memory (VSTM) and attention are both thought to have a capacity limit of four items [e.g. Luck, S. J., & Vogel, E. K. (1997). The capacity of visual working memory for features and conjunctions. Nature, 309, 279-281; Pylyshyn, Z. W., & Storm, R. W. (1988). Tracking multiple independent targets: evidence for a parallel tracking mechanism. Spatial Vision, 3, 179-197.]. Using the multiple object visual tracking paradigm (MOT), it has recently been shown that twice as many items can be simultaneously attended when they are separated between two visual fields compared to when they are all presented within the same hemifield [Alvarez, G. A., & Cavanagh, P. (2004). Independent attention resources for the left and right visual hemifields (Abstract). Journal of Vision, 4(8), 29a.]. Does VSTM capacity also increase when the items to be remembered are distributed between the two visual fields? The current paper investigated this central issue in two different tasks, namely a color and spatial location change detection task, in which the items were displayed either in the two visual fields or in the same hemifield. The data revealed that only memory capacity for spatial locations and not colors increased when the items were separated between the two visual fields. These findings support the view of VSTM as a chain of capacity limited operations where the spatial selection of stimuli, which dominates in both spatial location VSTM and MOT, occupies the first place and shows independence between the two fields.     

Object-based representations govern both the storage of information in visual short-term memory and the retrieval of information from it

Interest is growing in how information is retained in visual short-term memory (VSTM). We describe an experiment that assesses VSTM within the context of multidimensional signal detection theory. On every trial, participants were presented with a 250-ms display containing four colored shapes. They were then probed 900 ms later with a colored shape and made separate old/new judgments about the color and the shape. In any particular trial, one, both, or neither of the probed features had been presented. Performance differed according to whether both probed features belonged to a single object or to two different objects. When both probed features belonged to the same object, featural retrieval was better than would be predicted by independent feature storage. When both probed features belonged to two different objects, featural retrieval was worse than would be predicted by independent feature storage. We conclude that storage in and retrieval from VSTM operate on the basis of object-based representations.     

Improving visual short-term memory by sequencing the stimulus array

When multiple objects are presented briefly and simultaneously in a visual array, visual short-term memory (VSTM) can maintain only a limited number of these items. The present research report reveals that splitting the to-be-remembered items into two sequential arrays significantly increases VSTM performance relative to the simultaneous presentation of the same items. A memory benefit also emerges when the full object array is flashed twice (repeated) rather than being presented continuously for the same duration. Moreover, the sequential and repetition benefits are specifically pronounced for individuals with low performance for simultaneously presented items. Our results suggest that the conventional, simultaneous presentation mode may underestimate VSTM performance due to attentional limitations and/or competition between stimulus representations. In contrast, temporal segregation of the stimulus input may help participants maximize their performance and utilize their full VSTM capacity.     

Internal attention to features in visual short-term memory guides object learning

Attending to objects in the world affects how we perceive and remember them. What are the consequences of attending to an object in mind? In particular, how does reporting the features of a recently seen object guide visual learning? In three experiments, observers were presented with abstract shapes in a particular color, orientation, and location. After viewing each object, observers were cued to report one feature from visual short-term memory (VSTM). In a subsequent test, observers were cued to report features of the same objects from visual long-term memory (VLTM). We tested whether reporting a feature from VSTM: (1) enhances VLTM for just that feature (practice-benefit hypothesis), (2) enhances VLTM for all features (object-based hypothesis), or (3) simultaneously enhances VLTM for that feature and suppresses VLTM for unreported features (feature-competition hypothesis). The results provided support for the feature-competition hypothesis, whereby the representation of an object in VLTM was biased towards features reported from VSTM and away from unreported features (Experiment 1). This bias could not be explained by the amount of sensory exposure or response learning (Experiment 2) and was amplified by the reporting of multiple features (Experiment 3). Taken together, these results suggest that selective internal attention induces competitive dynamics among features during visual learning, flexibly tuning object representations to align with prior mnemonic goals.     

Rapid development of feature binding in visual short-term memory

The binding of object identity (color) and location in visual short-term memory (VSTM) was examined in 6.5- to 12.5-month-old infants (N= 144). Although we previously found that by age 6.5 months, infants can represent both color and location in VSTM, in the present study we observed that 6.5-month-old infants could not remember trivially simple color-location combinations across a 300-ms delay. However, 7.5-month-old infants could bind color and location as effectively as 12.5-month-old infants. Control conditions confirmed that the failure of 6.5-month-old infants was not a result of perceptual or attentional limitations. This rapid development of VSTM binding between 6.5 and 7.5 months occurs during a period of rapid increase in VSTM storage capacity and just after a period of dramatic neuroanatomical changes in parietal cortex. Thus, the ability to bind features and the ability to store multiple objects may both depend on a process that is mediated by posterior parietal cortex and is perhaps related to focused attention.     

Lack of color integration in visual short-term memory binding

Bicolored objects are retained in visual short-term memory (VSTM) less efficiently than unicolored objects. This is unlike shape–color combinations, whose retention in VSTM does not differ from that observed for shapes only. It is debated whether this is due to a lack of color integration and whether this may reflect the function of separate memory mechanisms. Participants judged whether the colors of bicolored objects (each with an external and an internalcolor) were the same or different across two consecutive screens. Colors had to be remembered either individually or in combination. In Experiment 1, external colors in the combined colors condition were remembered better than the internal colors, and performance for both was worse than that in the individual colors condition. The lack of color integration observed in Experiment 1 was further supported by a reduced capacity of VSTM to retain color combinations, relative to individual colors (Experiment 2). An additional account was found in Experiment 3, which showed spared color–color binding in the presence of impaired shape–color binding in a brain-damaged patient, thus suggesting that these two memory mechanisms are different.     

Visual short-term memory benefit for objects on different 3-D surfaces

Visual short-term memory (VSTM) plays an important role in visual cognition. Although objects are located on different 3-dimensional (3-D) surfaces in the real world, how VSTM capacity may be influenced by the presence of multiple 3-D surfaces has never been examined. By manipulating binocular disparities of visual displays, the authors found that more colored objects could be held in VSTM when they were placed on 2 rather than on 1 planar 3-D surfaces. This between-surface benefit in VSTM was present only when binding of objects' colors to their 3-D locations was required (i.e., when observers needed to remember which color appeared where). When binding was not required, no between-surface benefit in VSTM was observed. This benefit in VSTM could not be attributed to the number of spatial locations attended within a given surface. It was not due to a general perceptual grouping effect either, because grouping by motion and grouping by different regions of the same surface did not yield the same benefit. This increment in capacity indicates that VSTM benefits from the placement of objects in a 3-D scene.     

Is visual short-term memory object based? Rejection of the “strong-object” hypothesis

Is the capacity of visual short-term memory (VSTM) limited by the number of objects or by the number of features? VSTM for objects with either one feature or two color features was tested. Results show that capacity was limited primarily by the number of colors to be memorized, not by the number of objects. This result held up with variations in color saturation, blocked or mixed conditions, duration of memory image, and absence or presence of verbal load. However, conjoining features into objects improved VSTM capacity when size-orientation and color-orientation conjunctions were tested. Nevertheless, the number of features still mattered. When feature heterogeneity was controlled, VSTM for conjoined objects was worse than VSTM for objects made of single features. Our results support a weak-object hypothesis of VSTM capacity that suggests that VSTM is limited by both the number of objects and the feature composition of those objects.     

Representation of dynamic spatial configurations in visual short-term memory

Locations of multiple stationary objects are represented on the basis of their global spatial configuration in visual short-term memory (VSTM). Once objects move individually, they form a global spatial configuration with varying spatial inter-object relations over time. The representation of such dynamic spatial configurations in VSTM was investigated in six experiments. Participants memorized a scene with six moving and/or stationary objects and performed a location change detection task for one object specified during the probing phase. The spatial configuration of the objects was manipulated between memory phase and probing phase. Full spatial configurations showing all objects caused higher change detection performance than did no or partial spatial configurations for static and dynamic scenes. The representation of dynamic scenes in VSTM is therefore also based on their global spatial configuration. The variation of the spatiotemporal features of the objects demonstrated that spatiotemporal features of dynamic spatial configurations are represented in VSTM. The presentation of conflicting spatiotemporal cues interfered with memory retrieval. However, missing or conforming spatiotemporal cues triggered memory retrieval of dynamic spatial configurations. The configurational representation of stationary and moving objects was based on a single spatial configuration, indicating that static spatial configurations are a special case of dynamic spatial configurations.     

Is visual short-term memory object based? Rejection of the "strong-object" hypothesis

Is the capacity of visual short-term memory (VSTM) limited by the number of objects or by the number of features? VSTM for objects with either one feature or two color features was tested. Results show that capacity was limited primarily by the number of colors to be memorized, not by the number of objects. This result held up with variations in color saturation, blocked or mixed conditions, duration of memory image, and absence or presence of verbal load. However, conjoining features into objects improved VSTM capacity when size-orientation and color-orientation conjunctions were tested. Nevertheless, the number of features still mattered. When feature heterogeneity was controlled, VSTM for conjoined objects was worse than VSTM for objects made of single features. Our results support a weak-object hypothesis of VSTM capacity that suggests that VSTM is limited by both the number of objects and the feature composition of those objects.     

The capacity of visual short-term memory is not a fixed number of objects

Luck and Vogel (1997) have reported several striking results in support of the view that visual short-term memory (VSTM) has a fixed capacity of four objects, irrespective of how many relevant features those objects comprise. However, more recent studies have challenged this account, indicating only a weak effect of the number of objects once other factors are more evenly equated across conditions. Here, we employed a symmetry manipulation to verify object segmentation in our displays, to demonstrate that when spatial and masking factors are held constant, the number of objects per se has no effect on VSTM. Instead, VSTM capacity may reflect the number of object "parts" or feature conjunctions in a given display.     

Visual cognition influences early vision: the role of visual short-term memory in amodal completion

A partly occluded visual object is perceptually filled in behind the occluding surface, a process known as amodal completion or visual interpolation. Previous research focused on the image-based properties that lead to amodal completion. In the present experiments, we examined the role of a higher-level visual process-visual short-term memory (VSTM)-in amodal completion. We measured the degree of amodal completion by asking participants to perform an object-based attention task on occluded objects while maintaining either zero or four items in visual working memory. When no items were stored in VSTM, participants completed the occluded objects; when four items were stored in VSTM, amodal completion was halted (Experiment 1). These results were not caused by the influence of VSTM on object-based attention per se (Experiment 2) or by the specific location of to-be-remembered items (Experiment 3). Items held in VSTM interfere with amodal completion, which suggests that amodal completion may not be an informationally encapsulated process, but rather can be affected by high-level visual processes.     

Testing visual short-term memory of pigeons (Columba livia) and a rhesus monkey (Macaca mulatta) with a location change detection task

Change detection is commonly used to assess capacity (number of objects) of human visual short-term memory (VSTM). Comparisons with the performance of non-human animals completing similar tasks have shown similarities and differences in object-based VSTM, which is only one aspect (“what”) of memory. Another important aspect of memory, which has received less attention, is spatial short-term memory for “where” an object is in space. In this article, we show for the first time that a monkey and pigeons can be accurately trained to identify location changes, much as humans do, in change detection tasks similar to those used to test object capacity of VSTM. The subject’s task was to identify (touch/peck) an item that changed location across a brief delay. Both the monkey and pigeons showed transfer to delays longer than the training delay, to greater and smaller distance changes than in training, and to novel colors. These results are the first to demonstrate location-change detection in any non-human species and encourage comparative investigations into the nature of spatial and visual short-term memory.     

The benefit of surface uniformity for encoding boundary features in visual working memory

Using a change detection paradigm, the present study examined an object-based encoding benefit in visual working memory (VWM) for two boundary features (two orientations in Experiments 1-2 and two shapes in Experiments 3-4) assigned to a single object. Participants remembered more boundary features when they were conjoined into a single object of uniform color than when they were not bound together into an object. However, such an object-based benefit diminished or disappeared when those features occurred on different parts of an object, as defined by surface discontinuity or by negative minima of curvature. These results suggest that the number of boundary features that can be stored in VWM is influenced by the hierarchical structure of objects: surface uniformity facilitates the integration of two boundary features into a single object representation, but this integration is weakened when the features occur on different parts of an object.     

Why does visual working memory ability improve with age: More objects,more feature detail,or both? A registered report

We investigated how visual working memory (WM) develops with age across the early elementary school period (6–7 years), early adolescence (11–13 years), and early adulthood (18–25 years). The work focuses on changes in two parameters: the number of objects retained at least in part, and the amount of feature-detail remembered for such objects. Some evidence suggests that, while infants can remember up to three objects, much like adults, young children only remember around two objects. This curious, nonmonotonic trajectory might be explained by differences in the level of feature-detail required for successful performance in infant versus child/adult memory paradigms. Here, we examined if changes in one of two parameters (the number of objects, and the amount of detail retained for each object) or both of them together can explain the development of visual WM ability as children grow older. To test it, we varied the amount of feature-detail participants need to retain. In the baseline condition, participants saw an array of objects and simply were to indicate whether an object was present in a probed location or not. This phase begun with a titration procedure to adjust each individual's array size to yield about 80% correct. In other conditions, we tested memory of not only location but also additional features of the objects (color, and sometimes also orientation). Our results suggest that capacity growth across ages is expressed by both improved location-memory (whether there was an object in a location) and feature completeness of object representations.     

Hemifield asymmetries differentiate VSTM for single- and multiple-feature objects

Visual short-term memory (VSTM) is a capacity-limited system for maintaining visual information across brief durations. Limits in the amount of information held in memory reflect processing constraints in the intraparietal sulcus (IPS), a region of the frontoparietal network also involved in visual attention. During VSTM and visual attention, areas of IPS demonstrate hemispheric asymmetries. Whereas the left hemisphere represents information in only the right hemifield, the right hemisphere represents information across the visual field. In visual attention, hemispheric asymmetries are associated with differences in behavioral performance across the visual field. In order to assess the degree of hemifield asymmetries in VSTM, we measured memory performance across the visual field for both single- and two-feature objects. Consistent with theories of right-hemisphere dominance, there was a memory benefit for single-feature items in the left visual hemifield. However, when the number of features increased, the behavioral bias reversed, demonstrating a benefit for remembering two-feature objects in the right hemifield. On an individual basis, the cost of remembering an additional feature in the hemifields was correlated, suggesting that the shift in hemifield biases reflected a redistribution of resources across the visual field. Furthermore, we demonstrate that these results cannot be explained by differences in perceptual or decision-making load. Our results are consistent with a flexible resource model of VSTM in which attention and/or working memory demands result in representation of items in the right hemifield by both the left and right hemispheres.     

Organization of visual short-term memory

The authors examined the organization of visual short-term memory (VSTM). Using a change-detection task, they reported that VSTM stores relational information between individual items. This relational processing is mediated by the organization of items into spatial configurations. The spatial configuration of visual objects is important for VSTM of spatial locations, colors, and shapes. When color VSTM is compared with location VSTM, spatial configuration plays an integral role because configuration is important for color VSTM, whereas color is not important for location VSTM. The authors also examined the role of attention and found that the formation of configuration is modulated by both top-down and bottom-up attentional factors. In summary, the authors proposed that VSTM stores the relational information of individual visual items on the basis of global spatial configuration.     

Feature binding in visual short term memory: A General Recognition Theory analysis

Creating and maintaining accurate bindings of elementary features (e.g., color and shape) in visual short-term memory (VSTM) is fundamental for veridical perception. How are low-level features bound in memory? The present work harnessed a multivariate model of perception – the General Recognition Theory (GRT) – to unravel the internal representations underlying feature binding in VSTM. On each trial, preview and target colored shapes were presented in succession, appearing in either repeated or altered spatial locations. Participants gave two same/different responses: one with respect to color and one with respect to shape. Converging GRT analyses on the accuracy confusion matrices provided substantial evidence for binding in the form of violations of perceptual independence at the level of the individual stimulus, such that positive correlations were obtained when both features repeated or alternated together, while negative correlations were obtained when one feature repeated and the other alternated. This “cloverleaf” GRT pattern of binding was similar whether the spatial location of the preview and target repeated or altered. The current results are consistent with: (a) the discrete memory “slots” model of VSTM, and (b) the notion that spatial location is not necessary for the formation of “object files.” The GRT approach presented here offers a viable quantitative model for testing various questions regarding feature binding in VSTM.