Feature binding in visual short term memory: A General Recognition Theory analysis

Creating and maintaining accurate bindings of elementary features (e.g., color and shape) in visual short-term memory (VSTM) is fundamental for veridical perception. How are low-level features bound in memory? The present work harnessed a multivariate model of perception – the General Recognition Theory (GRT) – to unravel the internal representations underlying feature binding in VSTM. On each trial, preview and target colored shapes were presented in succession, appearing in either repeated or altered spatial locations. Participants gave two same/different responses: one with respect to color and one with respect to shape. Converging GRT analyses on the accuracy confusion matrices provided substantial evidence for binding in the form of violations of perceptual independence at the level of the individual stimulus, such that positive correlations were obtained when both features repeated or alternated together, while negative correlations were obtained when one feature repeated and the other alternated. This “cloverleaf” GRT pattern of binding was similar whether the spatial location of the preview and target repeated or altered. The current results are consistent with: (a) the discrete memory “slots” model of VSTM, and (b) the notion that spatial location is not necessary for the formation of “object files.” The GRT approach presented here offers a viable quantitative model for testing various questions regarding feature binding in VSTM.     

Grouping and binding in visual short-term memory

Findings of 2 experiments are reported that challenge the current understanding of visual short-term memory (VSTM). In both experiments, a single study display, containing 6 colored shapes, was presented briefly and then probed with a single colored shape. At stake is how VSTM retains a record of different objects that share common features: In the 1st experiment, 2 study items sometimes shared a common feature (either a shape or a color). The data revealed a color sharing effect, in which memory was much better for items that shared a common color than for items that did not. The 2nd experiment showed that the size of the color sharing effect depended on whether a single pair of items shared a common color or whether 2 pairs of items were so defined-memory for all items improved when 2 color groups were presented. In explaining performance, an account is advanced in which items compete for a fixed number of slots, but then memory recall for any given stored item is prone to error. A critical assumption is that items that share a common color are stored together in a slot as a chunk. The evidence provides further support for the idea that principles of perceptual organization may determine the manner in which items are stored in VSTM. (PsycINFO Database Record (c) 2012 APA, all rights reserved).     

Encoding color and shape from different parts of an object in visual short-term memory

Can we find an object-based encoding benefit in visual short-term memory (VSTM) when the features to be remembered are from different parts of an object? Using object parts defined by either figure-ground separation or negative minima of curvature, results from five experiments in which the visual change detection paradigm was used showed that the object-based encoding benefit in VSTM is modulated by how features are assigned to parts of an object: Features are best retained when the color and shape features to be remembered belong to the same part of an object. Although less well retained in comparison, features from different parts of an object are still better remembered than features from spatially separated objects. An object-based feature binding therefore exists even when the color and shape features to be remembered are from different parts of an object.     

The privileged role of location in visual working memory

Reports have conflicted about the possible special role of location in visual working memory (WM). One important question is: Do we maintain the locations of objects in WM even when they are irrelevant to the task at hand? Here we used a continuous response scale to study the types of reporting errors that participants make when objects are presented at the same or at different locations in space. When several objects successively shared the same location, participants exhibited a higher tendency to report features of the wrong object in memory; that is, they responded with features that belonged to objects retained in memory but not probed at retrieval. On the other hand, a similar effect was not observed when objects shared a nonspatial feature, such as color. Furthermore, the effect of location on reporting errors was present even when its manipulation was orthogonal to the task at hand. These findings are consistent with the view that binding together different nonspatial features of an object in memory might be mediated through an object’s location. Hence, spatial location may have a privileged role in WM. The relevance of these findings to conceptual models, as well as to neural accounts of visual WM, is discussed.     

Understanding the object benefit in visual short-term memory: the roles of feature proximity and connectedness

Past research has identified visual objects as the units of information processing in visual short-term memory (VSTM) and has shown that two features from the same object can be remembered in VSTM as well (or almost as well) as one feature of that object and are much better remembered than the same two features from two spatially separated objects. It is not clear, however, what drives this object benefit in VSTM. Is it the shared spatial location (proximity), the connectedness among features of an object, or both? In six change detection experiments, both location/proximity and connectedness were found to be crucial in determining the magnitude of the object benefit in VSTM. Together, these results indicate that location/proximity and connectedness are essential elements in defining a coherent visual object representation in VSTM.     

Individuation of visual objects over time

How does an observer decide that a particular object viewed at one time is actually the same object as one viewed at a different time? We explored this question using an experimental task in which an observer views two objects as they simultaneously approach an occluder, disappear behind the occluder, and re-emerge from behind the occluder, having switched paths. In this situation the observer either sees both objects continue straight behind the occluder (called "streaming") or sees them collide with each other and switch directions ("bouncing"). This task has been studied in the literature on motion perception, where interest has centered on manipulating spatiotemporal aspects of the motion paths (e.g. velocity, acceleration). Here we instead focus on featural properties (size, luminance, and shape) of the objects. We studied the way degrees and types of featural dissimilarity between the two objects influence the percept of bouncing vs. streaming. When there is no featural difference, the preference for straight motion paths dominates, and streaming is usually seen. But when featural differences increase, the preponderance of bounce responses increases. That is, subjects prefer the motion trajectory in which each continuously existing individual object trajectory contains minimal featural change. Under this model, the data reveal in detail exactly what magnitudes of each type of featural change subjects implicitly regard as reasonably consistent with a continuously existing object. This suggests a simple mathematical definition of "individual object:" an object is a path through feature-trajectory space that minimizes feature change, or, more succinctly, an object is a geodesic in Mahalanobis feature space.     

Is visual short-term memory object based? Rejection of the “strong-object” hypothesis

Is the capacity of visual short-term memory (VSTM) limited by the number of objects or by the number of features? VSTM for objects with either one feature or two color features was tested. Results show that capacity was limited primarily by the number of colors to be memorized, not by the number of objects. This result held up with variations in color saturation, blocked or mixed conditions, duration of memory image, and absence or presence of verbal load. However, conjoining features into objects improved VSTM capacity when size-orientation and color-orientation conjunctions were tested. Nevertheless, the number of features still mattered. When feature heterogeneity was controlled, VSTM for conjoined objects was worse than VSTM for objects made of single features. Our results support a weak-object hypothesis of VSTM capacity that suggests that VSTM is limited by both the number of objects and the feature composition of those objects.     

Is visual short-term memory object based? Rejection of the "strong-object" hypothesis

Is the capacity of visual short-term memory (VSTM) limited by the number of objects or by the number of features? VSTM for objects with either one feature or two color features was tested. Results show that capacity was limited primarily by the number of colors to be memorized, not by the number of objects. This result held up with variations in color saturation, blocked or mixed conditions, duration of memory image, and absence or presence of verbal load. However, conjoining features into objects improved VSTM capacity when size-orientation and color-orientation conjunctions were tested. Nevertheless, the number of features still mattered. When feature heterogeneity was controlled, VSTM for conjoined objects was worse than VSTM for objects made of single features. Our results support a weak-object hypothesis of VSTM capacity that suggests that VSTM is limited by both the number of objects and the feature composition of those objects.     

Binding feature dimensions in visual short-term memory

We explored several possible influences on binding in visual short-term memory (VSTM) performance. The task was to report whether a test object was the same (“old” trials) or different (“new” trials) from any of the sample objects seen a second ago. The objects were composed of two features that varied from continuous to discrete shapes and colors. In “old” trials the test object appeared either in the same or different position. In “new” trials the test object differed along both features, requiring storage of only one feature per object; along one feature, requiring no binding but storage of all features; or it was created by recombining features from the sample, which requires binding. Existing storage hypotheses are unable to explain the similar sensitivity (d′) obtained in the two last conditions when position remained the same and may suggest that links are created between positions and features. Highest sensitivity occurred when the test object remained at the same position, required no binding, and discrete features were used. Object-type × position, and feature combination × position interactions occurred, suggesting different storage modes depending on whether objects change position during retention.     

What’s the object of object working memory in infancy? Unraveling ‘what’ and ‘how many’

Infants have a bandwidth-limited object working memory (WM) that can both individuate and identify objects in a scene, (answering ‘how many?’ or ‘what?’, respectively). Studies of infants’ WM for objects have typically looked for limits on either ‘how many’ or ‘what’, yielding different estimates of infant capacity. Infants can keep track of about three individuals (regardless of identity), but appear to be much more limited in the number of specific identities they can recall. Why are the limits on ‘how many’ and ‘what’ different? Are the limits entirely separate, do they interact, or are they simply two different aspects of the same underlying limit?We sought to unravel these limits in a series of experiments which tested 9- and 12-month-olds’ WM for object identities under varying degrees of difficulty. In a violation-of-expectation looking-time task, we hid objects one at a time behind separate screens, and then probed infants’ WM for the shape identity of the penultimate object in the sequence. We manipulated the difficulty of the task by varying both the number of objects in hiding locations and the number of means by which infants could detect a shape change to the probed object. We found that 9-month-olds’ WM for identities was limited by the number of hiding locations: when the probed object was one of two objects hidden (one in each of two locations), 9-month-olds succeeded, and they did so even though they were given only one means to detect the change. However, when the probed object was one of three objects hidden (one in each of three locations), they failed, even when they were given two means to detect the shape change. Twelve-month-olds, by contrast, succeeded at the most difficult task level.Results show that WM for ‘how many’ and for ‘what’ are not entirely separate. Individuated objects are tracked relatively cheaply. Maintaining bindings between indexed objects and identifying featural information incurs a greater attentional/memory cost. This cost reduces with development. We conclude that infant WM supports a small number of featureless object representations that index the current locations of objects. These can have featural information bound to them, but only at substantial cost.     

Retrospective cues based on object features improve visual working memory performance in older adults

Research with younger adults has shown that retrospective cues can be used to orient top-down attention toward relevant items in working memory. We examined whether older adults could take advantage of these cues to improve memory performance. Younger and older adults were presented with visual arrays of five colored shapes; during maintenance, participants were presented either with an informative cue based on an object feature (here, object shape or color) that would be probed, or with an uninformative, neutral cue. Although older adults were less accurate overall, both age groups benefited from the presentation of an informative, feature-based cue relative to a neutral cue. Surprisingly, we also observed differences in the effectiveness of shape versus color cues and their effects upon post-cue memory load. These results suggest that older adults can use top-down attention to remove irrelevant items from visual working memory, provided that task-relevant features function as cues.     

Internal attention to features in visual short-term memory guides object learning

Attending to objects in the world affects how we perceive and remember them. What are the consequences of attending to an object in mind? In particular, how does reporting the features of a recently seen object guide visual learning? In three experiments, observers were presented with abstract shapes in a particular color, orientation, and location. After viewing each object, observers were cued to report one feature from visual short-term memory (VSTM). In a subsequent test, observers were cued to report features of the same objects from visual long-term memory (VLTM). We tested whether reporting a feature from VSTM: (1) enhances VLTM for just that feature (practice-benefit hypothesis), (2) enhances VLTM for all features (object-based hypothesis), or (3) simultaneously enhances VLTM for that feature and suppresses VLTM for unreported features (feature-competition hypothesis). The results provided support for the feature-competition hypothesis, whereby the representation of an object in VLTM was biased towards features reported from VSTM and away from unreported features (Experiment 1). This bias could not be explained by the amount of sensory exposure or response learning (Experiment 2) and was amplified by the reporting of multiple features (Experiment 3). Taken together, these results suggest that selective internal attention induces competitive dynamics among features during visual learning, flexibly tuning object representations to align with prior mnemonic goals.     

Updating objects in visual short-term memory is feature selective

The purpose of this study was to examine whether the process of updating information in visual short-term memory (VSTM) is object based. We investigated whether modifying the memory of one feature of an object would automatically promote refreshing the memory of all of its other features. The results showed that the facilitative effect of updating was specific to the updated feature of an object and did not spread to its nonupdated features. This feature-selective effect suggests that updating VSTM is not object based (Experiment 1), even though storage was object based (Experiment 2). Control experiments ruled out strategy-based (Experiment 3) and stimulus-related (Experiments 4–6) accounts. Feature-selective updating may indicate that the mechanism used to modify the contents of memory may have a different basis than that used to encode or store information in memory.     

Surface assignment modulates object formation for visual short-term memory

Recent results have suggested that the operational units of visual short-term memory (VSTM) are whole objects, rather than features or the total amount of information to be remembered. Here, for the first time, the influence of surface assignment on object formation for VSTM was investigated. The observers had to memorize the features of four briefly presented (300 ms) two-part objects, followed by a mask and a cue indicating which object to report on. The experiments contrasted whether there were any apparent depth differences between the two parts of each object, and whether observers had to report on only one or both features of the post-cued target object. Depth differences induced with stereoscopic disparity, and with a pictorial depth cue (simple interposition of object features), interfered strongly with performance when both features of an object needed to be memorized, but aided performance when only a single feature needed to be remembered. Furthermore, there was considerable within-feature interference consistent with some previous findings, but contradicting others. The potential implications for conceptions of VSTM are discussed in the light of two hypothesized stages: an early feature-based stage, as well as a higher-level object-based stage where the depth manipulations exert their effects. The results argue for a strong modulatory influence of surface assignment on object formation for a VSTM task.     

Independent encoding of colors and shapes from two stimuli

Observers were presented with brief exposures of pairs of colored objects (letters) and asked to report both the color and the shape of each object. Several observers showed strikingly clear evidence of nearly perfect stochastic independence between reports of the four features (two colors and two shapes). For instance, the probability that the shape of a given object could be reported seemed independent of (a) whether the color of the object could be reported and (b) whether features of the other object could be reported. Such stochastic independence is predicted by many parallel-processing models (e.g., Bundesen, 1990). However, the results are difficult to reconcile with simple serial models in which the encoding of one object is completed before the encoding of another object is begun.     

Setting and changing feature priorities in visual short-term memory

Many everyday tasks require prioritizing some visual features over competing ones, both during the selection from the rich sensory input and while maintaining information in visual short-term memory (VSTM). Here, we show that observers can change priorities in VSTM when, initially, they attended to a different feature. Observers reported from memory the orientation of one of two spatially interspersed groups of black and white gratings. Using colored pre-cues (presented before stimulus onset) and retro-cues (presented after stimulus offset) predicting the to-be-reported group, we manipulated observers’ feature priorities independently during stimulus encoding and maintenance, respectively. Valid pre-cues reliably increased observers’ performance (reduced guessing, increased report precision) as compared to neutral ones; invalid pre-cues had the opposite effect. Valid retro-cues also consistently improved performance (by reducing random guesses), even if the unexpected group suddenly became relevant (invalid-valid condition). Thus, feature-based attention can reshape priorities in VSTM protecting information that would otherwise be forgotten.     

Perceptual completion and object-based representations in short-term visual memory

Object-based representations in visual short-term memory (VSTM) were examined using a change detection memory task. A display comprising two rows of four differently colored elements was followed by a probe display in which only one of the rows reappeared. On same trials, the probed row was identical to the corresponding row in the memory display. On different trials, two of the elements in the probed row had their colors exchanged. In each memory display, a task-irrelevant visual element appeared between the two rows, with the potential to function as an occluder. Performance was enhanced when perceptual completion meant that four, rather than eight, objects were perceived in the memory display and when the probe display revealed that the occluded elements continued behind the occluder. It appears that several forms of representation can co-occur to support VSTM, one of which is object based.     

Memory load affects object individuation in 18-month-old infants

Accurate representation of a changing environment requires individuation-the ability to determine how many numerically distinct objects are present in a scene. Much research has characterized early individuation abilities by identifying which object features infants can use to individuate throughout development. However, despite the fact that without memory featural individuation would be impossible, little is known about how memory constrains object individuation. Here, we investigated infants' ability to individuate multiple objects at once and asked whether individuation performance changes as a function of memory load. In three experiments, 18-month-old infants saw one, two, or three objects hidden and always saw the correct number of objects retrieved. On some trials, one or more of these objects surreptitiously switched identity prior to retrieval. We asked whether infants would use this identity mismatch to individuate and, hence, continue searching for the missing object(s). We found that infants were less likely to individuate objects as memory load grew, but that infants individuated more successfully when the featural contrast between the hidden and retrieved objects increased. These results suggest that remembering more objects may result in a loss of representational precision, thereby decreasing the likelihood of successful individuation. We close by discussing possible links between our results and findings from adult working memory.     

Environment size and the use of feature and geometric cues for reorientation

We tested associative-based accounts of orientation by investigating the influence of environment size on the use of feature and geometric cues for reorientation. Two groups of participants were trained in dynamic three-dimensional virtual rectangular environments that differed in size to find a distinctly colored bin located at one of the four corners. Subsequently, we probed the reliance on feature and geometric cues for reorientation during test trials by presenting six trial types: Small Geometry Only, Large Geometry Only, Small Cue Conflict, Large Cue Conflict, Small Distal, and Large Distal. During Geometry Only test trials, all bins were black; thus, all distinctive featural information was removed leaving only geometric cues. For Cue Conflict test trials, all colored bins were shifted counter-clockwise one corner; thus, the geometric cues from the trained corner and the trained color were in direct conflict. During Distal test trials, the bin in the geometrically incorrect corner farthest from the trained corner was colored the same as during training; the remaining three bins were black. Thus, only this distant feature cue could be used to determine the location of the goal bin. Results suggested that geometric cues were used across changes in environment size, featural cues exerted greater influence when in conflict with geometric cues, and the far featural cue was used to disambiguate the correct from the rotationally equivalent location. In short, both feature and geometric cues were used for reorientation, and environment size influenced the relative use of feature and geometric cues. Collectively, our results provide evidence against associative-based accounts of orientation.     

Infant object segregation implies information integration

Researchers, including Needham (2001, this issue), have found that infants as young as 4.5 months of age have the ability to use featural information to segregate objects. However, considerable research on infants' perception of color, shape, size, orientation, and so on has shown that infants younger than 4.5 months are capable of using these featural cues to discriminate between objects or other test items. Infants as young as 2 months of age also can perceive a moving object as unified. In this article, we argue for an information processing explanation of these results, which centers on the development of infants' ability to integrate both featural and object information. The proposed explanation is based upon L. B. Cohen's (1991, 1998) information processing propositions and is consistent with the evidence on object segregation as well as evidence from our laboratory and others' on infant perception and cognition.