Attention to memory: orienting attention to sound object representations

Despite a growing acceptance that attention and memory interact, and that attention can be focused on an active internal mental representation (i.e., reflective attention), there has been a paucity of work focusing on reflective attention to ‘sound objects’ (i.e., mental representations of actual sound sources in the environment). Further research on the dynamic interactions between auditory attention and memory, as well as its degree of neuroplasticity, is important for understanding how sound objects are represented, maintained, and accessed in the brain. This knowledge can then guide the development of training programs to help individuals with attention and memory problems. This review article focuses on attention to memory with an emphasis on behavioral and neuroimaging studies that have begun to explore the mechanisms that mediate reflective attentional orienting in vision and more recently, in audition. Reflective attention refers to situations in which attention is oriented toward internal representations rather than focused on external stimuli. We propose four general principles underlying attention to short-term memory. Furthermore, we suggest that mechanisms involved in orienting attention to visual object representations may also apply for orienting attention to sound object representations.     

Memory-based attentional capture by colour and shape contents in visual working memory

Current theories assume that there is substantial overlap between visual working memory (VWM) and visual attention functioning, such that active representations in VWM automatically act as an attentional set, resulting in attentional biases towards objects that match the mnemonic content. Most evidence for this comes from visual search tasks in which a distractor similar to the memory interferes with the detection of a simultaneous target. Here we provide additional evidence using one of the most popular paradigms in the literature for demonstrating an active attentional set: The contingent spatial orienting paradigm of Folk and colleagues. This paradigm allows memory-based attentional biases to be more directly attributed to spatial orienting. Experiment 1 demonstrated a memory-contingent spatial attention effect for colour but not for shape contents of VWM. Experiment 2 tested the hypothesis that the placeholders used for spatial cueing interfered with the shape processing, and showed that memory-based attentional capture for shape returned when placeholders were removed. The results of the present study are consistent with earlier findings from distractor interference paradigms, and provide additional evidence that biases in spatial orienting contribute to memory-based influences on attention.     

Using the visual world paradigm to study language processing: a review and critical evaluation

We describe the key features of the visual world paradigm and review the main research areas where it has been used. In our discussion we highlight that the paradigm provides information about the way language users integrate linguistic information with information derived from the visual environment. Therefore the paradigm is well suited to study one of the key issues of current cognitive psychology, namely the interplay between linguistic and visual information processing. However, conclusions about linguistic processing (e.g., about activation, competition, and timing of access of linguistic representations) in the absence of relevant visual information must be drawn with caution.     

Mask similarity impacts short-term consolidation in visual working memory

Short-term consolidation is the process by which perceptual representations are stabilized into visual working memory (VWM) representations to prevent interference from subsequent visual input. The present article reports how short-term consolidation is affected by the similarity of the subsequent visual items (i.e., visual masks) by using a color change detection task. In the task, masks were either similar or dissimilar to the memory stimuli and were displayed at varying intervals following the memory array. The similar masks were made up of the same colored squares as the to-be-remembered stimuli, whereas the dissimilar masks were black-and-white grids. The results showed more interference from similar than from dissimilar masks: Similar masks required more time to consolidate and elicited lower overall performance than did dissimilar masks. These results suggest that a simple overwriting process cannot fully account for the impact of mask type on short-term consolidation performance and that other cognitive mechanisms are involved (e.g., controlled attention).     

The components of visual attention and the ubiquitous Simon effect

Spatial responding is influenced by the degree of correspondence between the stimulus–response (S–R) code activated by the target's task-irrelevant location and the S–R code activated by the target's non-spatial, task-relevant feature. A generally accepted explanation of this “Simon effect,” named after its discoverer, is that there is a natural tendency to respond towards the source of stimulation. First we will review the ubiquity of the Simon effect. Then we will review the literature, including our own studies when appropriate, that has explored the relationship between the Simon effect and the components of attention: alertness, orienting and executive control, with an emphasis on visual orienting. The Simon effect is reduced when participants are not alert and when executive control is effective in filtering out the irrelevant location information. When attention is oriented endogenously, or is captured exogenously by uninformative peripheral stimulation, the Simon effect is additive with attentional facilitation (i.e., the Simon effect is the same magnitude for targets presented at attended and unattended locations). Yet, some forms of orienting, such as orienting directed by gaze and biased by inhibition of return, modulate the Simon effect. We will explore the implications of these patterns of additivity and interaction for our understanding of both the Simon effect and spatial attention.     

Long-term visual associations affect attentional guidance

When observers perform a visual search task, they are assumed to adopt an attentional set for what they are looking for. The present experiment investigates the influence of long-term visual memory associations on this attentional set. On each trial, observers were asked to search a display for a grayscale version of a known traffic sign. On each trial, a distractor sign was drawn in full color. This color could either be related or unrelated to the target sign. Distractors interfered more with search when their color was related (e.g. red when the target was a stop sign), implying that long-term color associations resulted in inadvertent attentional guidance, even though color was irrelevant to the task. The results add to the growing body of evidence that long-term memory representations automatically affect attentional orienting.     

Distinguishing non-spatial from spatial biases in visual selection: neuropsychological evidence

Recently, it has been shown that patients with visual extinction can show enhanced awareness of contralesional stimuli that match the contents of working memory [Soto, D., & Humphreys, G.W. (2006). Proceedings of the National Academy of Sciences, USA, 103, 4789–4792]. Here we investigate whether these effects extend to cases where the working memory cue is verbal rather than visual (Study 1), and we show that non-spatial cues can overcome problems in spatial disengagement in patients, and this can affect the first eye movement made in search (Study 2). We discuss the implications of the data for understanding the relations between the cueing of attention from visual and conceptual information held in working memory and the cueing of spatial attention, and for understanding the relations between non-spatial and spatial biases in selection.     

Automatic and strategic effects in the guidance of attention by working memory representations

Theories of visual attention suggest that working memory representations automatically guide attention toward memory-matching objects. Some empirical tests of this prediction have produced results consistent with working memory automatically guiding attention. However, others have shown that individuals can strategically control whether working memory representations guide visual attention. Previous studies have not independently measured automatic and strategic contributions to the interactions between working memory and attention. In this study, we used a classic manipulation of the probability of valid, neutral, and invalid cues to tease apart the nature of such interactions. This framework utilizes measures of reaction time (RT) to quantify the costs and benefits of attending to memory-matching items and infer the relative magnitudes of automatic and strategic effects. We found both costs and benefits even when the memory-matching item was no more likely to be the target than other items, indicating an automatic component of attentional guidance. However, the costs and benefits essentially doubled as the probability of a trial with a valid cue increased from 20% to 80%, demonstrating a potent strategic effect. We also show that the instructions given to participants led to a significant change in guidance distinct from the actual probability of events during the experiment. Together, these findings demonstrate that the influence of working memory representations on attention is driven by both automatic and strategic interactions.     

Dynamic states in working memory modulate guidance of visual attention: Evidence from an n-back paradigm

Items held in working memory (WM) can automatically bias attention when they reappear in visual displays. Recent evidence, however, suggests that WM biases of attention may be reduced under certain conditions, for example with increasing memory load. We employed a dual task paradigm to investigate how WM biases are affected by dynamic updating of memory contents. 1-back and 2-back versions of a memory task with colour stimuli were interrupted at intervals by an unrelated visual search task. Reappearance in the search display of the item that was currently active in WM guided attention, while suppressed or inactive items did not. We conclude that the rapid updating of memory contents facilitates the shifting of memory representations into different activity states on a moment-to-moment basis. The finding is consistent with models that propose that only one item can be “active” in WM at any one time to guide attention.     

Attention on our mind: The role of spatial attention in visual working memory

The current study shows that spatial visual attention is used to retrieve information from visual working memory. Participants had to keep four colored circles in visual working memory. While keeping this information in memory we asked whether one of the colors was present in the array. While retrieving this information, on some trials a probe dot was presented. When this probe dot was presented at the location of the color that had to be retrieved, participants responded faster than when it was presented at another location. Our findings further elaborate the role of visual attention in working memory: not only is attention the mechanism by which information is stored into working memory, it is also the mechanism by which information is retrieved from visual working memory.     

Voluntazy and automatic attentional control of visual working memory

Previous studies of attention-directing cues have focused largely on the effects of cuing on perceptual processes, but cuing may also influence the transfer of perceptual representations into visual working memory. In the present study, we examined this potential role of cues, using both predictive and non-predictive cues in the context of a visual working memory task. Each trial began with a cue, followed by an array of six colored squares, a delay interval, and then a probe square presented at th e location of one of the squares in the previous array. The subjects were required to indicate whether the color of the probe square was the same as the color of the square that had previously been presented at the same location. Performance on this working memory task was more accurate when the cued location was probed than when an uncued location was probed, even when the cued location was no more likely to be probed than any of the uncued locations. An additional experiment using the abrupt-onset paradigm of Yantis and Jonides (1984) yielded similar results. Thus, visual transients may automatically influence the transfer of perceptual representations into visual working memory.     

Feature-based memory-driven attentional capture: visual working memory content affects visual attention

In 7 experiments, the authors explored whether visual attention (the ability to select relevant visual information) and visual working memory (the ability to retain relevant visual information) share the same content representations. The presence of singleton distractors interfered more strongly with a visual search task when it was accompanied by an additional memory task. Singleton distractors interfered even more when they were identical or related to the object held in memory, but only when it was difficult to verbalize the memory content. Furthermore, this content-specific interaction occurred for features that were relevant to the memory task but not for irrelevant features of the same object or for once-remembered objects that could be forgotten. Finally, memory-related distractors attracted more eye movements but did not result in longer fixations. The results demonstrate memory-driven attentional capture on the basis of content-specific representations.     

Influence of semantic consistency and perceptual features on visual attention during scene viewing in toddlers

Conceptual representations of everyday scenes are built in interaction with visual environment and these representations guide our visual attention. Perceptual features and object-scene semantic consistency have been found to attract our attention during scene exploration. The present study examined how visual attention in 24-month-old toddlers is attracted by semantic violations and how perceptual features (i. e. saliency, centre distance, clutter and object size) and linguistic properties (i. e. object label frequency and label length) affect gaze distribution. We compared eye movements of 24-month-old toddlers and adults while exploring everyday scenes which either contained an inconsistent (e.g., soap on a breakfast table) or consistent (e.g., soap in a bathroom) object. Perceptual features such as saliency, centre distance and clutter of the scene affected looking times in the toddler group during the whole viewing time whereas looking times in adults were affected only by centre distance during the early viewing time. Adults looked longer to inconsistent than consistent objects either if the objects had a high or a low saliency. In contrast, toddlers presented semantic consistency effect only when objects were highly salient. Additionally, toddlers with lower vocabulary skills looked longer to inconsistent objects while toddlers with higher vocabulary skills look equally long to both consistent and inconsistent objects. Our results indicate that 24-month-old children use scene context to guide visual attention when exploring the visual environment. However, perceptual features have a stronger influence in eye movement guidance in toddlers than in adults. Our results also indicate that language skills influence cognitive but not perceptual guidance of eye movements during scene perception in toddlers.     

Representations in mental imagery and working memory: Evidence from different types of visual masks

Although few studies have systematically investigated the relationship between visual mental imagery and visual working memory, work on the effects of passive visual interference has generally demonstrated a dissociation between the two functions. In four experiments, we investigated a possible commonality between the two functions: We asked whether both rely on depictive representations. Participants judged the visual properties of letters using visual mental images or pictures of unfamiliar letters stored in short-term memory. Participants performed both tasks with two different types of interference: sequences of unstructured visual masks (consisting of randomly changing white and black dots) or sequences of structured visual masks (consisting of fragments of letters). The structured visual noise contained elements of depictive representations (i.e., shape fragments arrayed in space), and hence should interfere with stored depictive representations; the unstructured visual noise did not contain such elements, and thus should not interfere as much with such stored representations. Participants did in fact make more errors in both tasks with sequences of structured visual masks. Various controls converged in demonstrating that in both tasks participants used representations that depicted the shapes of the letters. These findings not only constrain theories of visual mental imagery and visual working memory, but also have direct implications for why some studies have failed to find that dynamic visual noise interferes with visual working memory.     

Mental scanning of sonifications reveals flexible encoding of nonspeech sounds and a universal per-item scanning cost

A mental scanning paradigm was used to examine the representation of nonspeech sounds in working memory. Participants encoded sonifications – nonspeech auditory representations of quantitative data – as either verbal lists, visuospatial images, or auditory images. The number of tones and overall frequency changes in the sonifications were also manipulated to allow for different hypothesized patterns of reaction times across encoding strategies. Mental scanning times revealed different patterns of reaction times across encoding strategies, despite the fact that all internal representations were constructed from the same nonspeech sound stimuli. Scanning times for the verbal encoding strategy increased linearly as the number of items in the verbal representation increased. Scanning times for the visuospatial encoding strategy were generally slower and increased as the metric distance (derived metaphorically from frequency change) in the mental image increased. Scanning times for the auditory imagery strategy were faster and closest to the veridical durations of the original stimuli. Interestingly, the number of items traversed in scanning a representation significantly affected scanning times across all encoding strategies. Results suggested that nonspeech sounds can be flexibly represented, and that a universal per-item scanning cost persisted across encoding strategies. Implications for cognitive theory, the mental scanning paradigm, and practical applications are discussed.     

Voluntary and automatic attentional control of visual working memory

Previous studies of attention-directing cues have focused largely on the effects of cuing on perceptual processes, but cuing may also influence the transfer of perceptual representations into visual working memory. In the present study, we examined this potential role of cues, using both predictive and nonpredictive cues in the context of a visual working memory task. Each trial began with a cue, followed by an array of six colored squares, a delay interval, and then a probe square presented at the location of one of the squares in the previous array. The subjects were required to indicate whether the color of the probe square was the same as the color of the square that had previously been presented at the same location. Performance on this working memory task was more accurate when the cued location was probed than when an uncued location was probed, even when the cued location was no more likely to be probed than any of the uncued locations. An additional experiment using the abrupt-onset paradigm of Yantis and Jonides (1984) yielded similar results. Thus, visual transients may automatically influence the transfer of perceptual representations into visual working memory.     

Symbolic control of visual attention: The role of working memory and attentional control settings

This study examined how 1 symbol is selected to control the allocation of attention when several symbols appear in the visual field. In Experiments 1-3, the critical target feature was color, and it was found that uninformative central arrows that matched the color of the target were selected and produced unintentional shifts of attention (i.e., involuntary, initiated slowly, producing long-lasting facilitatory effects). Experiment 4 tested whether such selection is the result of an attentional filter or of a competition bias due to a match of incoming information against integrated object representations stored in working memory. Here, the critical feature was shape and color was irrelevant, but matching color arrows were still selected. Thus, features of objects in working memory will bias the selection of symbols in the visual field, and such selected symbols are capable of producing unintentional shifts of attention. ((c) 2003 APA, all rights reserved)     

Attention is required for maintenance of feature binding in visual working memory

Working memory and attention are intimately connected. However, understanding the relationship between the two is challenging. Currently, there is an important controversy about whether objects in working memory are maintained automatically or require resources that are also deployed for visual or auditory attention. Here we investigated the effects of loading attention resources on precision of visual working memory, specifically on correct maintenance of feature-bound objects, using a dual-task paradigm. Participants were presented with a memory array and were asked to remember either direction of motion of random dot kinematograms of different colour, or orientation of coloured bars. During the maintenance period, they performed a secondary visual or auditory task, with varying levels of load. Following a retention period, they adjusted a coloured probe to match either the motion direction or orientation of stimuli with the same colour in the memory array. This allowed us to examine the effects of an attention-demanding task performed during maintenance on precision of recall on the concurrent working memory task. Systematic increase in attention load during maintenance resulted in a significant decrease in overall working memory performance. Changes in overall performance were specifically accompanied by an increase in feature misbinding errors: erroneous reporting of nontarget motion or orientation. Thus in trials where attention resources were taxed, participants were more likely to respond with nontarget values rather than simply making random responses. Our findings suggest that resources used during attention-demanding visual or auditory tasks also contribute to maintaining feature-bound representations in visual working memory—but not necessarily other aspects of working memory.     

Content-context binding in verbal working memory updating: on-line and off-line effects

The role of content–context binding in working memory updating has received only marginal interest, despite its undoubted relevance for the updating process. In the classical updating paradigm, the main focus has been on the process of discarding information when new information is presented. Its efficacy has been determined by measurement of the accuracy of recalled, updated information. In the current study we measured the working memory updating process directly, employing a dynamic memory task composed of alternating sequences of learning, active maintenance and updating phases. Binding was manipulated by changing the type of updating (total or partial) and the strength of the perceptual connection between items. The on-line updating process and the off-line efficacy were separately analysed. Both on-line and off-line measures indicated that partial updating conditions, where the operation of updating content–context binding was required, were more demanding than both total updating conditions and memory and maintenance phases. Our results suggest that working memory updating can be identified not only as a process of substitution of information, but also as inhibition of no longer relevant information and, above all, as a binding updating.     

Selective working memory disables inhibition of visual features

Recent research suggests that information held in working memory can facilitate subsequent attentional processing. Here, we explore the negative corollary of this conception: Under which circumstances does information in working memory disrupt subsequent processing? Seventy participants performed visual discriminations in a dual-task paradigm. They were asked to judge colors or shapes in an online attention task under three different working-memory conditions: Same, Switch, or Unknown. In the Same condition, participants selectively maintained one visual feature in working memory, from the same dimension as in the online attention task. In the Switch condition, participants selectively maintained one visual feature in working memory, but had to focus on another visual dimension in the online attention task. In the Unknown condition, participants could not predict which visual feature would be relevant for the working-memory task. We found that irrelevant features in the online attention task were particularly difficult to ignore in the Switch condition, that is, when the irrelevant features belong to a visual dimension that is simultaneously prioritized in selective working memory. The findings are consistent with accounts in terms of neural overlap between working-memory and attention circuits, and suggest that mechanisms of selection, rather than resource limitations, critically determine the extent of visual interference.