The case for sensorimotor coding in working memory

The highly influential Baddeley and Hitch model of working memory (Baddeley & Hitch, 1974; see also Baddeley, 1986) posited analogical forms of representation that can be broadly characterized as sensorimotor, both for verbal and for visuospatial material. However, difficulties with the model of verbal working memory in particular have led investigators to develop alternative models that avoid appealing either to sensory coding or to motoric coding, or to both. This paper examines the evidence for sensorimotor coding in working memory, including evidence from neuropsychology and from sign language research, as well as from standard working memory paradigms, and concludes that only a sensorimotor model can accommodate the broad range of effects that characterize verbal working memory. In addition, several findings that have been considered to speak against sensorimotor involvement are reexamined and are argued to be in fact compatible with sensorimotor coding. These conclusions have broad implications, in that they support the emerging theoretical viewpoint of embodied cognition.     

Sensorimotor simulation and emotion processing: Impairing facial action increases semantic retrieval demands

Sensorimotor models suggest that understanding the emotional content of a face recruits a simulation process in which a viewer partially reproduces the facial expression in their own sensorimotor system. An important prediction of these models is that disrupting simulation should make emotion recognition more difficult. Here we used electroencephalogram (EEG) and facial electromyogram (EMG) to investigate how interfering with sensorimotor signals from the face influences the real-time processing of emotional faces. EEG and EMG were recorded as healthy adults viewed emotional faces and rated their valence. During control blocks, participants held a conjoined pair of chopsticks loosely between their lips. During interference blocks, participants held the chopsticks horizontally between their teeth and lips to generate motor noise on the lower part of the face. This noise was confirmed by EMG at the zygomaticus. Analysis of EEG indicated that faces expressing happiness or disgust—lower face expressions—elicited larger amplitude N400 when they were presented during the interference than the control blocks, suggesting interference led to greater semantic retrieval demands. The selective impact of facial motor interference on the brain response to lower face expressions supports sensorimotor models of emotion understanding.     

Visuospatial working memory influences the interaction between space and time

How do representations of space inform our perception of time? In language, spatial vocabulary is frequently used to describe temporal concepts, and spatial information biases temporal perception even in non-verbal tasks. In contrast, temporal information typically exerts little, if any, influence on the perception of spatial extent. Here, we used spatial and temporal reproduction tasks, both with and without verbal and spatial dual tasks, to investigate the mechanism underlying the asymmetric relation between space and time. Specifically, we tested whether the asymmetric interference between spatial and temporal stimulus attributes arises from interference in verbal or visuospatial working memory. We found that loading visuospatial working memory, but not verbal working memory, eliminated the asymmetric pattern of interference. This suggests that the interference between spatial and temporal representations arises due to processing constraints in visuospatial working memory. These findings are discussed in terms of the load theory of attention and the relative automaticity with which spatial and temporal information is processed.     

Interference and the representation of order within associations

In the presence of interference, recall of pairs can critically depend on the diagnostic power of memory of the order of items within the pair. Models of pair memory make different assumptions about whether and how such order information is stored, from convolution-based models, which assume no explicit storage of order, to matrix models and several models that assume a pair is learned by concatenating the representations of the constituent items, which lead to perfect within-pair order memory (given retrieval of the pair). Here we investigate memory for associations and within-pair order by examining the relationship between forward and backward probes of pairs subject to order-dependent associative interference in a double-function list paradigm. Associative interference disrupted the high correlation between forward and backward recall accuracy that is typically observed in standard paired-associate learning, challenging matrix and concatenation-based models. However, participants could overcome some interference due to within-pair order ambiguity, challenging directionally ambiguous convolution-based models. Unexpectedly, the test–retest correlation was reduced for pairs under the influence of interference compared to control pairs. This finding is incompatible with all existing implementations of the model classes we consider. Any model must include an assumption that order encoding (but not retrieval) is unreliable, and the form of this additional mechanism may depend intimately on how a given model is designed. In sum, our findings suggest that within-pair order memory is neither poor nor perfect, pointing to a fallible mechanism for within-pair order learning in verbal association-memory tasks.     

Interference and the representation of order within associations

In the presence of interference, recall of pairs can critically depend on the diagnostic power of memory of the order of items within the pair. Models of pair memory make different assumptions about whether and how such order information is stored, from convolution-based models, which assume no explicit storage of order, to matrix models and several models that assume a pair is learned by concatenating the representations of the constituent items, which lead to perfect within-pair order memory (given retrieval of the pair). Here we investigate memory for associations and within-pair order by examining the relationship between forward and backward probes of pairs subject to order-dependent associative interference in a double-function list paradigm. Associative interference disrupted the high correlation between forward and backward recall accuracy that is typically observed in standard paired-associate learning, challenging matrix and concatenation-based models. However, participants could overcome some interference due to within-pair order ambiguity, challenging directionally ambiguous convolution-based models. Unexpectedly, the test-retest correlation was reduced for pairs under the influence of interference compared to control pairs. This finding is incompatible with all existing implementations of the model classes we consider. Any model must include an assumption that order encoding (but not retrieval) is unreliable, and the form of this additional mechanism may depend intimately on how a given model is designed. In sum, our findings suggest that within-pair order memory is neither poor nor perfect, pointing to a fallible mechanism for within-pair order learning in verbal association-memory tasks.     

时空干扰效应：基于贝叶斯模型的解释

时空干扰效应是指时间知觉受空间信息干扰或空间知觉受时间信息干扰而出现错觉的现象。部分研究认为时空干扰是不对称的,空间对时间的干扰总是更大;还有研究认为时间和空间相互干扰的强度受实验因素影响,一般来说,空间对时间的干扰更大,但时间也能对空间产生同等程度甚至更大的干扰。在回顾隐喻理论和量值理论的主要观点之后,重点分析贝叶斯模型对时空干扰效应的解释,最后提出未来研究应关注的三个问题,即拓展贝叶斯模型对时空干扰效应的解释范围,探明基于贝叶斯推断的时空干扰神经机制和探索时空干扰的调控方法。     

Situation models and retrieval interference: pictures and words

Previous studies have found that interference in long-term memory retrieval occurs when information cannot be integrated into a single situation model, but this interference is greatly reduced or absent when the information can be so integrated. The current study looked at the influence of presentation format-sentences or pictures-on this observed pattern. When sentences were used at memorisation and recognition, a spatial organisation was observed. In contrast, when pictures were used, a different pattern of results was observed. Specifically, there was an overall speed-up in response times, and consistent evidence of interference. Possible explanations for this difference were examined in a third experiment using pictures during learning, but sentences during recognition. The results from Experiment 3 were consistent with the organisation of information into situation models in long-term memory, even from pictures. This suggests that people do create situation models when learning pictures, but their recognition memory may be oriented around more "verbatim", surface-form memories of the pictures.     

Can infant self-locomotion and spatial exploration predict spatial memory at school age?

According to an embodied view of development sensorimotor activity plays a central role in cognitive development. Following this idea, we studied whether the age of achieving self-locomotion milestones and spatial exploration during the first years of life predict spatial memory at (pre)school age. Spatial memory was assessed in 51 children at ages four and six years. Parents reported retrospectively about ages of attainment of self-locomotion milestones and about the children's spatial exploration behaviour during infancy and early toddlerhood. Results show that spatial exploration positively predicts spatial memory at both ages. The age of attainment of self- locomotion does not predict spatial memory at ages four and six. These findings extend previous work that showed a relation between exploration and spatial cognition over a short period of time. Results provide preliminary support to the hypothesis, suggesting that spatial exploration predicts spatial memory also over longer periods of time.     

Sensorimotor alignment effects in the learning environment and in novel environments

Four experiments investigated the conditions contributing to sensorimotor alignment effects (i.e., the advantage for spatial judgments from imagined perspectives aligned with the body). Through virtual reality technology, participants learned object locations around a room (learning room) and made spatial judgments from imagined perspectives aligned or misaligned with their actual facing direction. Sensorimotor alignment effects were found when testing occurred in the learning room but not after walking 3 m into a neighboring (novel) room. Sensorimotor alignment effects returned after returning to the learning room or after providing participants with egocentric imagery instructions in the novel room. Additionally, visual and spatial similarities between the test and learning environments were independently sufficient to cause sensorimotor alignment effects. Memory alignment effects, independent from sensorimotor alignment effects, occurred in all testing conditions. Results are interpreted in the context of two-system spatial memory theories positing separate representations to account for sensorimotor and memory alignment effects.     

Neurocognitive Contributions to Motor Skill Learning: The Role of Working Memory

ABSTRACT

Researchers have begun to delineate the precise nature and neural correlates of the cognitive processes that contribute to motor skill learning. The authors review recent work from their laboratory designed to further understand the neurocognitive mechanisms of skill acquisition. The authors have demonstrated an important role for spatial working memory in 2 different types of motor skill learning, sensorimotor adaptation and motor sequence learning. They have shown that individual differences in spatial working memory capacity predict the rate of motor learning for sensorimotor adaptation and motor sequence learning, and have also reported neural overlap between a spatial working memory task and the early, but not late, stages of adaptation, particularly in the right dorsolateral prefrontal cortex and bilateral inferior parietal lobules. The authors propose that spatial working memory is relied on for processing motor error information to update motor control for subsequent actions. Further, they suggest that working memory is relied on during learning new action sequences for chunking individual action elements together.     

Individual and developmental differences in working memory across the life span

The effects of secondary tasks on verbal and spatial working memory were examined in multiple child, young adult, and older adult samples. Although memory span increased with age in the child samples and decreased with age in the adult samples, there was little evidence of systematic change in the magnitude of interference effects. Surprisingly, individuals who had larger memory spans when there was no secondary task showed greater interference effects than their age-mates. These findings are inconsistent with the hypothesis that age and individual differences in working memory are due to differences in the ability to inhibit irrelevant information, at least as this hypothesis is currently formulated. Moreover, our results suggest that different mechanisms underlie developmental and individual differences in susceptibility to interference across the life span. A model is proposed in which memory span and processing speed both increase with development but are relatively independent abilities within age groups.     

Egocentric metric representations in peripersonal space: A bridge between motor resources and spatial memory

Research on visuospatial memory has shown that egocentric (subject-to-object) and allocentric (object-to-object) reference frames are connected to categorical (non-metric) and coordinate (metric) spatial relations, and that motor resources are recruited especially when processing spatial information in peripersonal (within arm reaching) than extrapersonal (outside arm reaching) space. In order to perform our daily-life activities, these spatial components cooperate along a continuum from recognition-related (e.g., recognizing stimuli) to action-related (e.g., reaching stimuli) purposes. Therefore, it is possible that some types of spatial representations rely more on action/motor processes than others. Here, we explored the role of motor resources in the combinations of these visuospatial memory components. A motor interference paradigm was adopted in which participants had their arms bent behind their back or free during a spatial memory task. This task consisted in memorizing triads of objects and then verbally judging what was the object: (1) closest to/farthest from the participant (egocentric coordinate); (2) to the right/left of the participant (egocentric categorical); (3) closest to/farthest from a target object (allocentric coordinate); and (4) on the right/left of a target object (allocentric categorical). The triads appeared in participants' peripersonal (Experiment 1) or extrapersonal (Experiment 2) space. The results of Experiment 1 showed that motor interference selectively damaged egocentric-coordinate judgements but not the other spatial combinations. The results of Experiment 2 showed that the interference effect disappeared when the objects were in the extrapersonal space. A third follow-up study using a within-subject design confirmed the overall pattern of results. Our findings provide evidence that motor resources play an important role in the combination of coordinate spatial relations and egocentric representations in peripersonal space.     

A temporal ratio model of memory

A model of memory retrieval is described. The model embodies 4 main claims: (a) temporal memory--traces of items are represented in memory partly in terms of their temporal distance from the present; (b) scale-similarity--similar mechanisms govern retrieval from memory over many different timescales; (c) local distinctiveness--performance on a range of memory tasks is determined by interference from near psychological neighbors; and (d) interference-based forgetting--all memory loss is due to interference and not trace decay. The model is applied to data on free recall and serial recall. The account emphasizes qualitative similarity in the retrieval principles involved in memory performance at all timescales, contrary to models that emphasize distinctions between short-term and long-term memory.     

对想象环境的空间更新

学界传统上将空间更新定义为个体随着身体运动对其所处真实环境空间表征的自动更新过程, 然而近年来有研究发现个体还可以对想象环境进行空间更新, 但其机制尚不明确。本研究实验1被试站在场景内学习物体空间位置之后, 保持学习朝向直线走到测试位置。其中, 0度组保持学习朝向站立, 180度组原地转动180度, 面对学习朝向的相反方向站立。两组被试均想象自己仍然站在学习位置, 面对学习朝向。然后被试旋转90度, 从3个想象朝向(记忆一致朝向、感觉运动一致朝向和不一致朝向)完成空间判断任务。结果发现0度组记忆一致朝向和感觉运动一致朝向成绩均优于不一致朝向, 而180度组无此优势。实验2被试从学习空间移动到测试空间过程中被迷向, 其他条件与实验1的180度组相同。但是, 实验2被试在记忆一致朝向和感觉运动一致朝向的成绩均优于不一致朝向。结果证明人们可以通过对在线空间表征进行想象平移或将离线记忆与空间更新系统相联结两种方式, 对想象环境进行空间更新。     

对听感觉运动门控自上而下调节的动物模型和神经机制

对惊反射的前脉冲抑制是减少干扰影响、保护脑内信息加工的重要机制。它既是研究感觉运动门控的模型, 也是探究精神分裂症机制的模型。前脉冲抑制可被选择性注意和情绪等高级认知活动所调节。本论文工作围绕着恐惧条件化和知觉空间分离去掩蔽(空间选择性注意)对听觉前脉冲抑制多层次化的自上而下调节, 在大鼠行为模型、神经通路、神经电生理机制等几个层次上开展了系统性研究, 并引入了精神分裂症的神经发育模型, 证实早期社会隔离饲养对前脉冲抑制注意调节的破坏影响。本论文研究成果不仅对认识正常情况下脑在复杂刺激场景中的信息加工机制有重要意义, 以感觉运动门控认知调节功能缺失为基础的动物模型也将推动精神分裂症心理学和神经生物学机制的研究。     

Exploiting redundancy for flexible behavior: unsupervised learning in a modular sensorimotor control architecture

Autonomously developing organisms face several challenges when learning reaching movements. First, motor control is learned unsupervised or self-supervised. Second, knowledge of sensorimotor contingencies is acquired in contexts in which action consequences unfold in time. Third, motor redundancies must be resolved. To solve all 3 of these problems, the authors propose a sensorimotor, unsupervised, redundancy-resolving control architecture (SURE_REACH), based on the ideomotor principle. Given a 3-degrees-of-freedom arm in a 2-dimensional environment, SURE_REACH encodes 2 spatial arm representations with neural population codes: a hand end-point coordinate space and an angular arm posture space. A posture memory solves the inverse kinematics problem by associating hand end-point neurons with neurons in posture space. An inverse sensorimotor model associates posture neurons with each other action-dependently. Together, population encoding, redundant posture memory, and the inverse sensorimotor model enable SURE_REACH to learn and represent sensorimotor grounded distance measures and to use dynamic programming to reach goals efficiently. The architecture not only solves the redundancy problem but also increases goal reaching flexibility, accounting for additional task constraints or realizing obstacle avoidance. While the spatial population codes resemble neurophysiological structures, the simulations confirm the flexibility and plausibility of the model by mimicking previously published data in arm-reaching tasks.     

躯体运动促进空间更新的环境依赖效应

Kelly, Avraamides和Loomis (2007)发现被试在新环境中空间更新失败, 而肖承丽和刘传军(2014)发现被试在新环境中可使用想象平移等策略来实现空间更新。为了探索该两项研究的异同, 本研究采用与Kelly等相同的实验范式进行研究。实验1被试在原学习环境中完成实验任务。实验2被试记忆完物体空间位置后, 转移至新环境, 在只依靠离线表征、离线表征与在线表征相协调和离线表征与在线表征相矛盾三种条件下完成相同的实验任务。结果发现, 被试在原环境中通过躯体运动和记忆两种方式促进空间更新, 具有同等有效性, 而在新环境中躯体运动显著差于记忆对空间更新的促进作用; 躯体运动和记忆对空间更新的促进作用在两种环境中均高度相关。研究表明, 躯体运动促进空间更新具有环境依赖效应, 与记忆对空间更新的促进作用相比, 躯体运动对空间更新的促进作用会随着环境的改变而降低。     

Visuospatial working memory and mental representation of spatial descriptions

The purpose of the present research is to investigate whether different components of working memory (WM) are involved in processing spatial and nonspatial texts. The interference effects of two concurrent tasks on comprehension and recall of two kinds of text were investigated in two experiments. Each participant listened to a spatial and a nonspatial text, with one of two concurrent tasks: articulatory suppression or spatial tapping. The dependent variables in Experiment 1 were accuracy of recall and verification of information inferred from the texts. In Experiment 2 response times in the verification task were also considered. Results support the hypothesis that verbal and spatial components of working memory are differentially involved in the comprehension and memory of spatial and nonspatial texts, with a selective interference effect of the spatial concurrent task on the spatial text and an interference effect of the verbal concurrent task on both the spatial and nonspatial texts. These effects emerged for recall, sentence verification, and response times. Our findings confirm previous results showing that the verbal component of working memory is involved in the process of text comprehension and memory. In addition, they show that visuospatial working memory is involved, in so far as the text conveys visuospatial information.     

Interference between object-based attention and object-based memory

Research has shown that there are at least two kinds of visual selective attention: location based and object based. In the present study, we sought to determine the locus of spatially invariant object-based selection using a dual-task paradigm. In four experiments, observers performed an attention task (object feature report or visual search) with a concurrent memory task (object memory or spatial memory). Object memory was interfered with more by a concurrent object-based attention task than by a concurrent location-based attention task. However, this interference pattern was reversed for spatial memory, with greater interference by a location-based attention task than by an object-based attention task. These findings suggest that object-based attention and locationbased attention are functionally dissociable and that some forms of object-based selection operate within visual short-term memory.