Feature binding in visual short term memory: A General Recognition Theory analysis

Creating and maintaining accurate bindings of elementary features (e.g., color and shape) in visual short-term memory (VSTM) is fundamental for veridical perception. How are low-level features bound in memory? The present work harnessed a multivariate model of perception – the General Recognition Theory (GRT) – to unravel the internal representations underlying feature binding in VSTM. On each trial, preview and target colored shapes were presented in succession, appearing in either repeated or altered spatial locations. Participants gave two same/different responses: one with respect to color and one with respect to shape. Converging GRT analyses on the accuracy confusion matrices provided substantial evidence for binding in the form of violations of perceptual independence at the level of the individual stimulus, such that positive correlations were obtained when both features repeated or alternated together, while negative correlations were obtained when one feature repeated and the other alternated. This “cloverleaf” GRT pattern of binding was similar whether the spatial location of the preview and target repeated or altered. The current results are consistent with: (a) the discrete memory “slots” model of VSTM, and (b) the notion that spatial location is not necessary for the formation of “object files.” The GRT approach presented here offers a viable quantitative model for testing various questions regarding feature binding in VSTM.     

Are multiple visual short-term memory storages necessary to explain the retro-cue effect?

Recent research has shown that change detection performance is enhanced when, during the retention interval, attention is cued to the location of the upcoming test item. This retro-cue advantage has led some researchers to suggest that visual short-term memory (VSTM) is divided into a durable, limited-capacity storage and a more fragile, high-capacity storage. Consequently, performance is poor on the no-cue trials because fragile VSTM is overwritten by the test display and only durable VSTM is accessible under these conditions. In contrast, performance is improved in the retro-cue condition because attention keeps fragile VSTM accessible. The aim of the present study was to test the assumptions underlying this two-storage account. Participants were asked to encode an array of colors for a change detection task involving no-cue and retro-cue trials. A retro-cue advantage was found even when the cue was presented after a visual (Experiment 1) or a central (Experiment 2) interference. Furthermore, the magnitude of the interference was comparable between the no-cue and retro-cue trials. These data undermine the main empirical support for the two-storage account and suggest that the presence of a retro-cue benefit cannot be used to differentiate between different VSTM storages.     

Mechanisms of priming of pop-out: Stored representations or feature-gain modulations?

Previous research has shown that repetition of a task-relevant attention-capturing feature facilitates popout search. This priming of pop-out effect is due to some residual memory from recent trials. We explore two possible mechanisms of priming of pop-out: a top-down attentional benefit from a memory of the previous target representation that is stored in visual short-term memory (VSTM) and a bottom-up change of attentional gains from perceptual features of the previously attended target. We manipulated participants’ ability to form a memory trace in VSTM by occupying it with a distractor task and found that occupying VSTM did not interfere with priming of pop-out. We next manipulated attentional gains associated with feature values by inserting an irrelevant task between pop-out searches. We found that the color of the target from the intervening perceptual task influenced pop-out search: The current pop-out search was facilitated when the intervening task’s target matched the target color of the pop-out search. These results suggest that priming of pop-out might not be due to a memory trace of the previous targets in VSTM but, rather, might be due to changes in attentional control based on priming from relatively low-level feature representations of previously attended objects.     

Working memory binding of visual object features in older adults

Accurate mental representation of visual stimuli requires retaining not only the individual features but also the correct relationship between them. This associative process of binding is mediated by working memory (WM) mechanisms. The present study re-examined reports of WM-related binding deficits with aging. In Experiment 1, 31 older and 31 younger adults completed a visual change detection task with feature–location relations presented either simultaneously or sequentially; the paradigm was also designed specifically to minimize the impact of lengthy retention intervals, elaborative rehearsal, and processing demands of multi-stimulus probes. In Experiment 2, 38 older and 42 younger adults completed a modified task containing both feature–location relations and feature–feature conjunctions. In Experiment 1 although feature–location binding was more difficult with sequential compared with simultaneous presentation, the effect was independent of age. In Experiment 2 while older adults were overall slower and less accurate than young adults, there were no age-specific deficits in WM binding. Overall, after controlling for methodological factors, there was no evidence of an age-related visual WM binding deficit for surface or location features. However, unlike younger adults, older adults appeared less able to restrict processing of irrelevant features, consistent with reported declines with age in strategic capacities of WM.     

The influence of location and visual features on visual object memory

In five experiments, we examined the influence of contextual objects’ location and visual features on visual memory. Participants’ visual memory was tested with a change detection task in which they had to judge whether the orientation (Experiments 1A, 1B, and 2) or color (Experiments 3A and 3B) of a target object was the same. Furthermore, contextual objects’ locations and visual features were manipulated in the test image. The results showed that change detection performance was better when contextual objects’ locations remained the same from study to test, demonstrating that the original spatial configuration is important for subsequent visual memory retrieval. The results further showed that changes to contextual objects’ orientation, but not color, reduced orientation change detection performance; and changes to contextual objects’ color, but not orientation, impaired color change detection performance. Therefore, contextual objects’ visual features are capable of affecting visual memory. However, selective attention plays an influential role in modulating such effects.     

Internal attention to features in visual short-term memory guides object learning

Attending to objects in the world affects how we perceive and remember them. What are the consequences of attending to an object in mind? In particular, how does reporting the features of a recently seen object guide visual learning? In three experiments, observers were presented with abstract shapes in a particular color, orientation, and location. After viewing each object, observers were cued to report one feature from visual short-term memory (VSTM). In a subsequent test, observers were cued to report features of the same objects from visual long-term memory (VLTM). We tested whether reporting a feature from VSTM: (1) enhances VLTM for just that feature (practice-benefit hypothesis), (2) enhances VLTM for all features (object-based hypothesis), or (3) simultaneously enhances VLTM for that feature and suppresses VLTM for unreported features (feature-competition hypothesis). The results provided support for the feature-competition hypothesis, whereby the representation of an object in VLTM was biased towards features reported from VSTM and away from unreported features (Experiment 1). This bias could not be explained by the amount of sensory exposure or response learning (Experiment 2) and was amplified by the reporting of multiple features (Experiment 3). Taken together, these results suggest that selective internal attention induces competitive dynamics among features during visual learning, flexibly tuning object representations to align with prior mnemonic goals.     

No evidence for binding of items to task-irrelevant backgrounds in visual working memory

When representing visual features such as color and shape in visual working memory (VWM), participants also represent the locations of those features as a spatial configuration of the locations of those features in the display. In everyday life, we encounter objects against some background, yet it is unclear whether the configural representation in memory obligatorily constitutes the entire display, including that (often task-irrelevant) background information. In three experiments, participants completed a change detection task on color and shape; the memoranda were presented in front of uniform gray backgrounds, a textured background (Exp. 1), or a background containing location placeholders (Exps. 2 and 3). When whole-display probes were presented, changes to the objects’ locations or feature bindings impacted memory performance—implying that the spatial configuration of the probes influenced participants’ change decisions. Furthermore, when only a single item was probed, the effect of changing its location or feature bindings was either diminished or completely extinguished, implying that single probes do not necessarily elicit the entire spatial configuration. Critically, when task-irrelevant backgrounds were also presented that may have provided a spatial configuration for the single probes, the effect of location or bindings was not moderated. These findings suggest that although the spatial configuration of a display guides VWM-based recognition, this information does not necessarily always influence the decision process during change detection.     

Organization of visual short-term memory

The authors examined the organization of visual short-term memory (VSTM). Using a change-detection task, they reported that VSTM stores relational information between individual items. This relational processing is mediated by the organization of items into spatial configurations. The spatial configuration of visual objects is important for VSTM of spatial locations, colors, and shapes. When color VSTM is compared with location VSTM, spatial configuration plays an integral role because configuration is important for color VSTM, whereas color is not important for location VSTM. The authors also examined the role of attention and found that the formation of configuration is modulated by both top-down and bottom-up attentional factors. In summary, the authors proposed that VSTM stores the relational information of individual visual items on the basis of global spatial configuration.     

Updating objects in visual short-term memory is feature selective

The purpose of this study was to examine whether the process of updating information in visual short-term memory (VSTM) is object based. We investigated whether modifying the memory of one feature of an object would automatically promote refreshing the memory of all of its other features. The results showed that the facilitative effect of updating was specific to the updated feature of an object and did not spread to its nonupdated features. This feature-selective effect suggests that updating VSTM is not object based (Experiment 1), even though storage was object based (Experiment 2). Control experiments ruled out strategy-based (Experiment 3) and stimulus-related (Experiments 4–6) accounts. Feature-selective updating may indicate that the mechanism used to modify the contents of memory may have a different basis than that used to encode or store information in memory.     

The reaction-time task-rule congruency effect is not affected by working memory load: further support for the activated long-term memory hypothesis

Previous studies claimed that task representation is carried out by the activated long-term memory portion of working memory (WM; Meiran and Kessler in J Exp Psychol Human Percept Perform 34:137–157, 2008). The present study provides a more direct support for this hypothesis. We used the reaction-time task-rule congruency effect (RT-TRCE) in a task-switching setup, and tested the effects of loading WM with irrelevant task rules on RT-TRCE. Experiment 1 manipulated WM load in a between-subject design. WM participants performed a color/shape task switching, while having 0, 1 or 3 numerical task rules as WM load. Experiment 2 used a similar load manipulation (1 or 3 rules to load WM) in a within-subject design. Experiment 3 extended these results by loading WM with perceptual tasks that were more similar to the shape/color tasks. The results show that RT-TRCE was not affected by WM load supporting the activated long-term memory hypothesis.     

Representation of dynamic spatial configurations in visual short-term memory

Locations of multiple stationary objects are represented on the basis of their global spatial configuration in visual short-term memory (VSTM). Once objects move individually, they form a global spatial configuration with varying spatial inter-object relations over time. The representation of such dynamic spatial configurations in VSTM was investigated in six experiments. Participants memorized a scene with six moving and/or stationary objects and performed a location change detection task for one object specified during the probing phase. The spatial configuration of the objects was manipulated between memory phase and probing phase. Full spatial configurations showing all objects caused higher change detection performance than did no or partial spatial configurations for static and dynamic scenes. The representation of dynamic scenes in VSTM is therefore also based on their global spatial configuration. The variation of the spatiotemporal features of the objects demonstrated that spatiotemporal features of dynamic spatial configurations are represented in VSTM. The presentation of conflicting spatiotemporal cues interfered with memory retrieval. However, missing or conforming spatiotemporal cues triggered memory retrieval of dynamic spatial configurations. The configurational representation of stationary and moving objects was based on a single spatial configuration, indicating that static spatial configurations are a special case of dynamic spatial configurations.     

The capacity of visual short-term memory within and between hemifields

Visual short-term memory (VSTM) and attention are both thought to have a capacity limit of four items [e.g. Luck, S. J., & Vogel, E. K. (1997). The capacity of visual working memory for features and conjunctions. Nature, 309, 279-281; Pylyshyn, Z. W., & Storm, R. W. (1988). Tracking multiple independent targets: evidence for a parallel tracking mechanism. Spatial Vision, 3, 179-197.]. Using the multiple object visual tracking paradigm (MOT), it has recently been shown that twice as many items can be simultaneously attended when they are separated between two visual fields compared to when they are all presented within the same hemifield [Alvarez, G. A., & Cavanagh, P. (2004). Independent attention resources for the left and right visual hemifields (Abstract). Journal of Vision, 4(8), 29a.]. Does VSTM capacity also increase when the items to be remembered are distributed between the two visual fields? The current paper investigated this central issue in two different tasks, namely a color and spatial location change detection task, in which the items were displayed either in the two visual fields or in the same hemifield. The data revealed that only memory capacity for spatial locations and not colors increased when the items were separated between the two visual fields. These findings support the view of VSTM as a chain of capacity limited operations where the spatial selection of stimuli, which dominates in both spatial location VSTM and MOT, occupies the first place and shows independence between the two fields.     

视觉长时记忆激活度对促进视觉短时记忆的影响

短时记忆与长时记忆的关系是记忆领域研究的重要内容。基于此, 研究者们就视觉长时记忆是否能促进视觉短时记忆的问题展开了大量研究, 但所得出的结论并不一致。通过不同程度的学习形成不同激活度的视觉长时记忆, 考察对几何图形的视觉长时记忆是否能对变化检测中的短时记忆起到促进作用。结果发现, 低激活水平的视觉长时记忆不能促进视觉短时记忆, 而事先存在且高度激活的视觉长时记忆对视觉短时记忆却具有促进作用; 同时, 随着视觉长时记忆激活水平的提高, 刺激间间隔的效应值也逐渐减小。本研究说明, 视觉长时记忆能否促进视觉短时记忆取决于视觉长时记忆的激活水平, 高激活的视觉长时记忆对阻止视觉短时记忆痕迹的迅速消退具有重要意义。     

Does high memory load kick task-irrelevant information out of visual working memory?

The limited capacity of visual working memory (VWM) requires the existence of an efficient information selection mechanism. While it has been shown that under low VWM load, an irrelevant simple feature can be processed, its fate under high load (e.g., six objects) remains unclear. We explored this issue by probing the “irrelevant-change distracting effect,” in which the change of a stored irrelevant feature affects performance. Simple colored shapes were used as stimuli, with color as the target. Using a whole-probe method (presenting six objects in both the memory and test arrays), in Experiment 1 we found that a change to one of the six shapes led to a significant distracting effect. Using a partial-probe method (presenting the probe either at the screen center or at a location selected from the memory array), in Experiment 2 we showed the distracting effect again. These results suggest that irrelevant simple features can be stored into VWM, regardless of memory load.     

Visual cognition influences early vision: the role of visual short-term memory in amodal completion

A partly occluded visual object is perceptually filled in behind the occluding surface, a process known as amodal completion or visual interpolation. Previous research focused on the image-based properties that lead to amodal completion. In the present experiments, we examined the role of a higher-level visual process-visual short-term memory (VSTM)-in amodal completion. We measured the degree of amodal completion by asking participants to perform an object-based attention task on occluded objects while maintaining either zero or four items in visual working memory. When no items were stored in VSTM, participants completed the occluded objects; when four items were stored in VSTM, amodal completion was halted (Experiment 1). These results were not caused by the influence of VSTM on object-based attention per se (Experiment 2) or by the specific location of to-be-remembered items (Experiment 3). Items held in VSTM interfere with amodal completion, which suggests that amodal completion may not be an informationally encapsulated process, but rather can be affected by high-level visual processes.     

Exogenous attention influences visual short-term memory in infants

Two experiments examined the hypothesis that developing visual attentional mechanisms influence infants' Visual Short-Term Memory (VSTM) in the context of multiple items. Five- and 10-month-old infants (N = 76) received a change detection task in which arrays of three differently colored squares appeared and disappeared. On each trial one square changed color and one square was cued; sometimes the cued item was the changing item, and sometimes the changing item was not the cued item. Ten-month-old infants exhibited enhanced memory for the cued item when the cue was a spatial pre-cue (Experiment 1) and 5-month-old infants exhibited enhanced memory for the cued item when the cue was relative motion (Experiment 2). These results demonstrate for the first time that infants younger than 6 months can encode information in VSTM about individual items in multiple-object arrays, and that attention-directing cues influence both perceptual and VSTM encoding of stimuli in infants as in adults.     

Lack of color integration in visual short-term memory binding

Bicolored objects are retained in visual short-term memory (VSTM) less efficiently than unicolored objects. This is unlike shape–color combinations, whose retention in VSTM does not differ from that observed for shapes only. It is debated whether this is due to a lack of color integration and whether this may reflect the function of separate memory mechanisms. Participants judged whether the colors of bicolored objects (each with an external and an internalcolor) were the same or different across two consecutive screens. Colors had to be remembered either individually or in combination. In Experiment 1, external colors in the combined colors condition were remembered better than the internal colors, and performance for both was worse than that in the individual colors condition. The lack of color integration observed in Experiment 1 was further supported by a reduced capacity of VSTM to retain color combinations, relative to individual colors (Experiment 2). An additional account was found in Experiment 3, which showed spared color–color binding in the presence of impaired shape–color binding in a brain-damaged patient, thus suggesting that these two memory mechanisms are different.     

Encoding color and shape from different parts of an object in visual short-term memory

Can we find an object-based encoding benefit in visual short-term memory (VSTM) when the features to be remembered are from different parts of an object? Using object parts defined by either figure-ground separation or negative minima of curvature, results from five experiments in which the visual change detection paradigm was used showed that the object-based encoding benefit in VSTM is modulated by how features are assigned to parts of an object: Features are best retained when the color and shape features to be remembered belong to the same part of an object. Although less well retained in comparison, features from different parts of an object are still better remembered than features from spatially separated objects. An object-based feature binding therefore exists even when the color and shape features to be remembered are from different parts of an object.     

两维特征图形的客体和空间工作记忆存储研究

研究采用单探测变化检测范式,考察了两维特征图形在视觉客体和视觉空间工作记忆中的存储机制,并对其容量进行测定。40名被试(平均年龄20.56±1.73岁)随机分为两个等组,分别完成实验一和实验二。实验一的刺激图形由颜色和形状两基本特征组成,实验二的刺激为由不同颜色和开口朝向组成的兰道环。两个实验结果均显示:(1)特征交换变化条件下的记忆成绩与单特征变化条件下最差的记忆成绩差异不显著;(2)空间工作记忆任务的成绩显著优于客体工作记忆任务;(3)被试在视觉工作记忆中能存储2~3个客体和3~4个空间位置。这表明,由两种不同维度特征组成的图形在视觉客体和视觉空间工作记忆中均以整合方式进行存储,空间工作记忆的容量大于客体工作记忆。     

How retaining objects containing multiple features in visual working memory regulates the priority for access to visual awareness

The content of visual working memory influences the access to visual awareness. Thus far, research has focused on retention of a single feature, whereas memoranda in real life typically contain multiple features. Here, we intermixed a delayed match-to-sample task to manipulate VWM content, and a breaking Continuous Flash Suppression (b-CFS) task to measure prioritization for visual awareness. Observers memorized either the color (Exp. 1), the shape (Exp. 2) or both the features (Exp. 3) of an item and indicated the location of a suppressed target. We observed that color-matching targets broke suppression faster than color-mismatching targets both when color was memory relevant or irrelevant. Shape only impacted priority for visual awareness through an interaction with color. We conclude that: (1) VWM can regulate the priority of visual information to access visual awareness along a single feature dimension; (2) different features of a memorandum vary in their potency to impact access to visual awareness, and the more dominant feature may even suppress the effect of the less dominant feature; (3) even stimuli that match an irrelevant feature dimension of the memorandum can be prioritized for visual awareness.