Terms of the debate on the format and structure of visual memory

Our ability to actively maintain information in visual memory is strikingly limited. There is considerable debate about why this is so. As with many questions in psychology, the debate is framed dichotomously: Is visual working memory limited because it is supported by only a small handful of discrete “slots” into which visual representations are placed, or is it because there is an insufficient supply of a “resource” that is flexibly shared among visual representations? Here, we argue that this dichotomous framing obscures a set of at least eight underlying questions. Separately considering each question reveals a rich hypothesis space that will be useful for building a comprehensive model of visual working memory. The questions regard (1) an upper limit on the number of represented items, (2) the quantization of the memory commodity, (3) the relationship between how many items are stored and how well they are stored, (4) whether the number of stored items completely determines the fidelity of a representation (vs. fidelity being stochastic or variable), (5) the flexibility with which the memory commodity can be assigned or reassigned to items, (6) the format of the memory representation, (7) how working memories are formed, and (8) how memory representations are used to make responses in behavioral tasks. We reframe the debate in terms of these eight underlying questions, placing slot and resource models as poles in a more expansive theoretical space.     

Fractionating working memory: consolidation and maintenance are independent processes

In the present study, we required subjects to remember simple objects that were masked to interrupt consolidation and allow us to estimate the rate of information accrual in visual working memory. We compared a consolidation-baseline condition with a consolidation-during-maintenance condition in which subjects needed to remember a set of unmasked items and then were shown to-be-remembered masked items. We hypothesized that if the control processes of consolidation and maintenance are performed by common mechanisms, then consolidation should be less efficient when performed during maintenance than when performed alone. However, we found that an identical amount of information was encoded per unit time in the two conditions. These results indicate that working memory consolidation is not slowed by maintenance and suggest a two-step model of encoding in visual working memory.     

Expertise for upright faces improves the precision but not the capacity of visual working memory

Considerable research has focused on how basic visual features are maintained in working memory, but little is currently known about the precision or capacity of visual working memory for complex objects. How precisely can an object be remembered, and to what extent might familiarity or perceptual expertise contribute to working memory performance? To address these questions, we developed a set of computer-generated face stimuli that varied continuously along the dimensions of age and gender, and we probed participants’ memories using a method-of-adjustment reporting procedure. This paradigm allowed us to separately estimate the precision and capacity of working memory for individual faces, on the basis of the assumptions of a discrete capacity model, and to assess the impact of face inversion on memory performance. We found that observers could maintain up to four to five items on average, with equally good memory capacity for upright and upside-down faces. In contrast, memory precision was significantly impaired by face inversion at every set size tested. Our results demonstrate that the precision of visual working memory for a complex stimulus is not strictly fixed but, instead, can be modified by learning and experience. We find that perceptual expertise for upright faces leads to significant improvements in visual precision, without modifying the capacity of working memory.     

Hierarchical encoding in visual working memory: ensemble statistics bias memory for individual items

Influential models of visual working memory treat each item to be stored as an independent unit and assume that there are no interactions between items. However, real-world displays have structure that provides higher-order constraints on the items to be remembered. Even in the case of a display of simple colored circles, observers can compute statistics, such as mean circle size, to obtain an overall summary of the display. We examined the influence of such an ensemble statistic on visual working memory. We report evidence that the remembered size of each individual item in a display is biased toward the mean size of the set of items in the same color and the mean size of all items in the display. This suggests that visual working memory is constructive, encoding displays at multiple levels of abstraction and integrating across these levels, rather than maintaining a veridical representation of each item independently.     

Searching while loaded: Visual working memory does not interfere with hybrid search efficiency but hybrid search uses working memory capacity

In “hybrid search” tasks, such as finding items on a grocery list, one must search the scene for targets while also searching the list in memory. How is the representation of a visual item compared with the representations of items in the memory set? Predominant theories would propose a role for visual working memory (VWM) either as the site of the comparison or as a conduit between visual and memory systems. In seven experiments, we loaded VWM in different ways and found little or no effect on hybrid search performance. However, the presence of a hybrid search task did reduce the measured capacity of VWM by a constant amount regardless of the size of the memory or visual sets. These data are broadly consistent with an account in which VWM must dedicate a fixed amount of its capacity to passing visual representations to long-term memory for comparison to the items in the memory set. The data cast doubt on models in which the search template resides in VWM or where memory set item representations are moved from LTM through VWM to earlier areas for comparison to visual items.     

Human visual short-term memory precision can be varied at will when the number of retained items is low

It has been debated whether human visual working memory is limited by the number of items or the precision with which they are represented. In the research reported here, we show that the precision of working memory can be flexibly and willfully controlled, but only if the number of retained items is low. Electroencephalographic recordings revealed that a neural marker for visual working memory (contralateral delay activity, or CDA) that is known to increase in amplitude with the number of retained items was also affected by the precision with which items were retained. However, willfully enhanced precision increased CDA amplitude only when the number of retained items was low. These results show that both the number and the (willfully controlled) precision of retained items constrain visual working memory: People can enhance the precision of their visual working memory, but only for a few items.     

The aftermath of memory retrieval for recycling visual working memory representations

We examined the aftermath of accessing and retrieving a subset of information stored in visual working memory (VWM)—namely, whether detection of a mismatch between memory and perception can impair the original memory of an item while triggering recognition-induced forgetting for the remaining, untested items. For this purpose, we devised a consecutive-change detection task wherein two successive testing probes were displayed after a single set of memory items. Across two experiments utilizing different memory-testing methods (whole vs. single probe), we observed a reliable pattern of poor performance in change detection for the second test when the first test had exhibited a color change. The impairment after a color change was evident even when the same memory item was repeatedly probed; this suggests that an attention-driven, salient visual change made it difficult to reinstate the previously remembered item. The second change detection, for memory items untested during the first change detection, was also found to be inaccurate, indicating that recognition-induced forgetting had occurred for the unprobed items in VWM. In a third experiment, we conducted a task that involved change detection plus continuous recall, wherein a memory recall task was presented after the change detection task. The analyses of the distributions of recall errors with a probabilistic mixture model revealed that the memory impairments from both visual changes and recognition-induced forgetting are explained better by the stochastic loss of memory items than by their degraded resolution. These results indicate that attention-driven visual change and recognition-induced forgetting jointly influence the “recycling” of VWM representations.     

Proactive interference from items previously stored in visual working memory

This study investigates the fate of information that was previously stored in visual working memory but that is no longer needed. Previous research has found inconsistent results, with some showing effective release of irrelevant information and others showing proactive interference. Using change detection tasks of colors or shapes, we show that participants tend to falsely classify a changed item as "no change" if it matches one of the memory items on the preceding trial. The interference is spatially specific: Memory for the preceding trial interferes more if it matches the feature value and the location of a test item than if it does not. Interference results from retaining information in visual working memory, since it is absent when items on the preceding trials are passively viewed, or are attended but not memorized. We conclude that people cannot fully eliminate unwanted visual information from current working memory tasks.     

Strategic and automatic effects of visual working memory on attention in visual search

Previous research indicates that visual attention can be automatically captured by sensory inputs that match the contents of visual working memory. However, Woodman and Luck (2007) showed that information in working memory can be used flexibly as a template for either selection or rejection according to task demands. We report two experiments that extend their work. Participants performed a visual search task while maintaining items in visual working memory. Memory items were presented for either a short or long exposure duration immediately prior to the search task. Memory was tested by a change-detection task immediately afterwards. On a random half of trials items in memory matched either one distractor in the search task (Experiment 1) or three (Experiment 2). The main result was that matching distractors speeded or slowed target detection depending on whether memory items were presented for a long or short duration. These effects were more in evidence with three matching distractors than one. We conclude that the influence of visual working memory on visual search is indeed flexible but is not solely a function of task demands. Our results suggest that attentional capture by perceptual inputs matching information in visual working memory involves a fast automatic process that can be overridden by a slower top-down process of attentional avoidance.     

You’re looking for what? Comparing search for familiar,nameable objects to search for unfamiliar,novel objects

Searching for items in one’s environment often includes considerable reliance on semantic knowledge. The present study examines the importance of semantic information in visual and memory search, especially with respect to whether the items reside in long-term or working memory. In Experiment 1, participants engaged in hybrid visual memory search for items that were either highly familiar or novel. Importantly, the relatively large number of targets in this hybrid search task necessitated that targets be stored in some form of long-term memory. We found that search for familiar objects was more efficient than search for novel objects. In Experiment 2, we investigated search for familiar versus novel objects when the number of targets was low enough to be stored in working memory. We also manipulated how often participants in Experiment 2 were required to update their target (every trial vs. every block) in order to control for target templates that were stored in long-term memory as a result of repeated exposure over trials. We found no differences in search efficiency for familiar versus novel objects when templates were stored in working memory. Our results suggest that while semantic information may provide additional individuating features that are useful for object recognition in hybrid search, this information could be irrelevant or even distracting when searching for targets stored in working memory.     

Visual working memory represents a fixed number of items regardless of complexity

Does visual working memory represent a fixed number of objects, or is capacity reduced as object complexity increases? We measured accuracy in detecting changes between sample and test displays and found that capacity estimates dropped as complexity increased. However, these apparent capacity reductions were strongly correlated with increases in sample-test similarity (r= .97), raising the possibility that change detection was limited by errors in comparing the sample and test, rather than by the number of items that were maintained in working memory. Accordingly, when sample-test similarity was low, capacity estimates for even the most complex objects were equivalent to the estimate for the simplest objects (r= .88), suggesting that visual working memory represents a fixed number of items regardless of complexity. Finally, a correlational analysis suggested a two-factor model of working memory ability, in which the number and resolution of representations in working memory correspond to distinct dimensions of memory ability.     

Cross-task repetition amnesia: Impaired recall of RSVP targets held in memory for a secondary task

People often fail to select and encode the second of two targets presented within less than 500ms in rapid serial visual presentation (RSVP), an effect known as the attentional blink. We investigated how report of the two targets is affected when one of them is maintained in working memory for a secondary, memory-search task. The results showed that report of either target was impaired when it was a member of the memory set relative to when it was not. This effect was independent of both the temporal interval separating the RSVP target from the presentation of the memory set and the interval separating the targets. We propose that the deficit in recall occurs because the association between a target and the memory-search task interferes with the formation of a new association between that target and the following RSVP task, with the result that observers may be biased to ascribe the target only to the memory set.     

The number and quality of representations in working memory

Flexible-resource theories characterize working memory as a flexible resource that can store either a large number of low-quality representations or a small number of high-quality representations. In contrast, limited-item theories propose that the number of items that can be stored in working memory is strictly limited and cannot be increased by decreasing the quality of the representations. We tested these fundamentally different conceptualizations of working memory capacity by determining whether observers could trade quality for quantity in working memory when given incentives to do so. We found no evidence that observers could increase the number of representations by decreasing their quality in working memory, but observers could make such a trade-off at earlier processing stages. Our results show that the capacity limit of working memory is best characterized as a limit on the number of items that can be stored and not as a limit on a finely divisible resource that simultaneously determines the number and quality of the representations.     

The contents of visual working memory reduce uncertainty during visual search

Information held in visual working memory (VWM) influences the allocation of attention during visual search, with targets matching the contents of VWM receiving processing benefits over those that do not. Such an effect could arise from multiple mechanisms: First, it is possible that the contents of working memory enhance the perceptual representation of the target. Alternatively, it is possible that when a target is presented among distractor items, the contents of working memory operate postperceptually to reduce uncertainty about the location of the target. In both cases, a match between the contents of VWM and the target should lead to facilitated processing. However, each effect makes distinct predictions regarding set-size manipulations; whereas perceptual enhancement accounts predict processing benefits regardless of set size, uncertainty reduction accounts predict benefits only with set sizes larger than 1, when there is uncertainty regarding the target location. In the present study, in which briefly presented, masked targets were presented in isolation, there was a negligible effect of the information held in VWM on target discrimination. However, in displays containing multiple masked items, information held in VWM strongly affected target discrimination. These results argue that working memory representations act at a postperceptual level to reduce uncertainty during visual search.     

Updating working memory in aircraft noise and speech noise causes different fMRI activations

The present study used fMRI/BOLD neuroimaging to investigate how visual‐verbal working memory is updated when exposed to three different background‐noise conditions: speech noise, aircraft noise and silence. The number‐updating task that was used can distinguish between “substitution processes,” which involve adding new items to the working memory representation and suppressing old items, and “exclusion processes,” which involve rejecting new items and maintaining an intact memory set. The current findings supported the findings of a previous study by showing that substitution activated the dorsolateral prefrontal cortex, the posterior medial frontal cortex and the parietal lobes, whereas exclusion activated the anterior medial frontal cortex. Moreover, the prefrontal cortex was activated more by substitution processes when exposed to background speech than when exposed to aircraft noise. These results indicate that (a) the prefrontal cortex plays a special role when task‐irrelevant materials should be denied access to working memory and (b) that, when compensating for different types of noise, either different cognitive mechanisms are involved or those cognitive mechanisms that are involved are involved to different degrees.     

Hemifield asymmetries differentiate VSTM for single- and multiple-feature objects

Visual short-term memory (VSTM) is a capacity-limited system for maintaining visual information across brief durations. Limits in the amount of information held in memory reflect processing constraints in the intraparietal sulcus (IPS), a region of the frontoparietal network also involved in visual attention. During VSTM and visual attention, areas of IPS demonstrate hemispheric asymmetries. Whereas the left hemisphere represents information in only the right hemifield, the right hemisphere represents information across the visual field. In visual attention, hemispheric asymmetries are associated with differences in behavioral performance across the visual field. In order to assess the degree of hemifield asymmetries in VSTM, we measured memory performance across the visual field for both single- and two-feature objects. Consistent with theories of right-hemisphere dominance, there was a memory benefit for single-feature items in the left visual hemifield. However, when the number of features increased, the behavioral bias reversed, demonstrating a benefit for remembering two-feature objects in the right hemifield. On an individual basis, the cost of remembering an additional feature in the hemifields was correlated, suggesting that the shift in hemifield biases reflected a redistribution of resources across the visual field. Furthermore, we demonstrate that these results cannot be explained by differences in perceptual or decision-making load. Our results are consistent with a flexible resource model of VSTM in which attention and/or working memory demands result in representation of items in the right hemifield by both the left and right hemispheres.     

由两个模型看视觉工作记忆容量机制的研究

在众多的实验基础上, 研究者提出了分析视觉工作记忆的“容量有限模型”和“资源灵活分配模型”, 而两个模型在以下三方面存在着争论：(1)视觉工作记忆中保存的项目精度是否是固定的, (2)资源分配的模式是连续分配的还是以量子化模式分配的, (3)存储单元是以特征为单元还是以客体为单元。未来该领域的研究应加强对视觉工作记忆容量单元的探讨, 探索应用新的研究方法, 控制影响视觉工作记忆容量的空间因素等。     

Guided search through memory

In “hybrid” search, observers search a visual display for any of several targets held in memory. It is known that the contents of the memory set can guide visual search (e.g., if the memorized targets are all animals, visual attention can be guided away from signs). It is not known if the visual display can guide memory search (e.g., if the memory set is composed of signs and animals, can a visual display of signs restrict memory search to just the signs?). In three hybrid search experiments, participants memorized sets of items that belonged to either one or several categories. Participants were then presented with visual displays containing multiple items, also drawn from one or several categories. Participants were asked to determine if any of the items from their current memory set were present in the visual display. We replicate the finding that visual search can be guided by the contents of memory. We find weaker, novel evidence that memory search can be guided by the contents of the visual display.     

The benefit of forgetting

Recent research using change-detection tasks has shown that a directed-forgetting cue, indicating that a subset of the information stored in memory can be forgotten, significantly benefits the other information stored in visual working memory. How do these directed-forgetting cues aid the memory representations that are retained? We addressed this question in the present study by using a recall paradigm to measure the nature of the retained memory representations. Our results demonstrated that a directed-forgetting cue leads to higher-fidelity representations of the remaining items and a lower probability of dropping these representations from memory. Next, we showed that this is made possible by the to-be-forgotten item being expelled from visual working memory following the cue, allowing maintenance mechanisms to be focused on only the items that remain in visual working memory. Thus, the present findings show that cues to forget benefit the remaining information in visual working memory by fundamentally improving their quality relative to conditions in which just as many items are encoded but no cue is provided.     

Selective working memory disables inhibition of visual features

Recent research suggests that information held in working memory can facilitate subsequent attentional processing. Here, we explore the negative corollary of this conception: Under which circumstances does information in working memory disrupt subsequent processing? Seventy participants performed visual discriminations in a dual-task paradigm. They were asked to judge colors or shapes in an online attention task under three different working-memory conditions: Same, Switch, or Unknown. In the Same condition, participants selectively maintained one visual feature in working memory, from the same dimension as in the online attention task. In the Switch condition, participants selectively maintained one visual feature in working memory, but had to focus on another visual dimension in the online attention task. In the Unknown condition, participants could not predict which visual feature would be relevant for the working-memory task. We found that irrelevant features in the online attention task were particularly difficult to ignore in the Switch condition, that is, when the irrelevant features belong to a visual dimension that is simultaneously prioritized in selective working memory. The findings are consistent with accounts in terms of neural overlap between working-memory and attention circuits, and suggest that mechanisms of selection, rather than resource limitations, critically determine the extent of visual interference.