Computational model of enactive visuospatial mental imagery using saccadic perceptual actions

From the onset of cognitive revolution, the concept of mental imagery has been given different, many times opposing, theoretical accounts. Mental imagery appears to be a ubiquitous, yet wholly individual, easy to explain experience on the one hand, being hard to deal with scientifically on the other hand. The focus of this research is on an enactive approach to visuospatial mental imagery, inspired by Sima’s perceptual instantiation theory. We designed a hybrid computational model, composed of a forward model, an inverse model, both implemented as neural networks, and a memory/controller module, that grounds simple mental concepts, such as a triangle and a square, in perceptual actions, and is able to reimagine these objects by performing the necessary perceptual actions in a simulated humanoid robot. We tested the model on three tasks – salience-based object recognition, imagination-based object recognition and object imagination – and achieved very good results showing, as a proof of concept, that perceptual actions are a viable candidate for grounding the visuospatial mental concepts as well as the credible substrate of visuospatial mental imagery.     

A quantitative evaluation of the AVITEWRITE model of handwriting learning

Much sensory-motor behavior develops through imitation, as during the learning of handwriting by children. Such complex sequential acts are broken down into distinct motor control synergies, or muscle groups, whose activities overlap in time to generate continuous, curved movements that obey an inverse relation between curvature and speed. The adaptive vector integration to endpoint handwriting (AVITEWRITE) model of Grossberg and Paine (2000) [A neural model of corticocerebellar interactions during attentive imitation and predictive learning of sequential handwriting movements. Neural Networks, 13, 999-1046] addressed how such complex movements may be learned through attentive imitation. The model suggested how parietal and motor cortical mechanisms, such as difference vector encoding, interact with adaptively-timed, predictive cerebellar learning during movement imitation and predictive performance. Key psychophysical and neural data about learning to make curved movements were simulated, including a decrease in writing time as learning progresses; generation of unimodal, bell-shaped velocity profiles for each movement synergy; size scaling with isochrony, and speed scaling with preservation of the letter shape and the shapes of the velocity profiles; an inverse relation between curvature and tangential velocity; and a two-thirds power law relation between angular velocity and curvature. However, the model learned from letter trajectories of only one subject, and only qualitative kinematic comparisons were made with previously published human data. The present work describes a quantitative test of AVITEWRITE through direct comparison of a corpus of human handwriting data with the model's performance when it learns by tracing the human trajectories. The results show that model performance was variable across the subjects, with an average correlation between the model and human data of 0.89+/-0.10. The present data from simulations using the AVITEWRITE model highlight some of its strengths while focusing attention on areas, such as novel shape learning in children, where all models of handwriting and the learning of other complex sensory-motor skills would benefit from further research.     

A computer simulation model of rats’ place navigation in the Morris water maze

Rats rapidly learn to swim from a variety of starting locations, including novel ones, to a small invisible platform submerged in a pool of cool opaque liquid. A computer simulation based on a simple perceptual memory-matching model successfully mimics this ability. The model assumes that the rat, when it successfully reaches the platform, notes the distance to prominent extramaze landmarks and stores this perceptual information in memory. When placed in the pool on subsequent trials, the simulated rat attempts to match perceived distance between itself and the landmarks to the remembered distances from the platform to the landmark. The model accounts for many of the known facts about rats’ behavior in the swimming pool and makes some interesting predictions that could be easily tested experimentally. The model has the advantage, relative to other cognitive map models, of specifying how spatial information is represented in memory and how this information guides behavior.     

A parallel distributed processing model of stimulus-stimulus and stimulus-response compatibility.

A parallel distributed processing (PDP) model is proposed to account for choice reaction time (RT) performance in diverse cognitive and perceptual tasks such as the Stroop task, the Simon task, the Eriksen flanker task, and the stimulus-response compatibility task that are interrelated in terms of stimulus-stimulus and stimulus-response overlap (Kornblum, 1992). In multilayered (input-intermediate-output) networks, neuron-like nodes that represent stimulus and response features are grouped into mutually inhibitory modules that represent stimulus and response dimensions. The stimulus-stimulus overlap is implemented by a convergence of two input modules onto a common intermediate module, and the stimulus-response overlap by direct pathways representing automatic priming of outputs. Mean RTs are simulated in various simple tasks and, furthermore, predictions are generated for complex tasks based on performance in simpler tasks. The match between simulated and experimental results lends strong support for our PDP model of compatibility.     

Spatial But Not Oculomotor Information Biases Perceptual Memory: Evidence From Face Perception and Cognitive Modeling

Recent research put forward the hypothesis that eye movements are integrated in memory representations and are reactivated when later recalled. However, “looking back to nothing” during recall might be a consequence of spatial memory retrieval. Here, we aimed at distinguishing between the effect of spatial and oculomotor information on perceptual memory. Participants’ task was to judge whether a morph looked rather like the first or second previously presented face. Crucially, faces and morphs were presented in a way that the morph reactivated oculomotor and/or spatial information associated with one of the previously encoded faces. Perceptual face memory was largely influenced by these manipulations. We considered a simple computational model with an excellent match (4.3% error) that expresses these biases as a linear combination of recency, saccade, and location. Surprisingly, saccades did not play a role. The results suggest that spatial and temporal rather than oculomotor information biases perceptual face memory.     

Independence of long-term contextual memory and short-term perceptual hypotheses: Evidence from contextual cueing of interrupted search

Observers are able to resume an interrupted search trial faster relative to responding to a new, unseen display. This finding of rapid resumption is attributed to short-term perceptual hypotheses generated on the current look and confirmed upon subsequent looks at the same display. It has been suggested that the contents of perceptual hypotheses are similar to those of other forms of memory acquired long-term through repeated exposure to the same search displays over the course of several trials, that is, the memory supporting “contextual cueing.” In three experiments, we investigated the relationship between short-term perceptual hypotheses and long-term contextual memory. The results indicated that long-term, contextual memory of repeated displays neither affected the generation nor the confirmation of short-term perceptual hypotheses for these displays. Furthermore, the analysis of eye movements suggests that long-term memory provides an initial benefit in guiding attention to the target, whereas in subsequent looks guidance is entirely based on short-term perceptual hypotheses. Overall, the results reveal a picture of both long- and short-term memory contributing to reliable performance gains in interrupted search, while exerting their effects in an independent manner.     

The perception of space and form recognition in a simulated environment: the case of minimalist sensory-substitution devices

Whenever we explore a simulated environment, the sensorimotor interactions that underlie our perception of space may be modified. We investigated the conditions under which it is possible to acquire the mastery of new sensorimotor laws and thereby to infer new perceptual spaces. A computer interface, based on the principles of minimalist sensory-substitution devices, was designed to enable different possible links between a user's actions (manipulation of a mouse and/or keys of a keyboard) and the resulting pattern of sensory stimulation (visual or auditory) to be established. The interface generated an all-or-none stimulus whose activation varied as a function of the participant's exploration of a hidden form. In this study we addressed the following questions: What are the conditions necessary for participants to understand their actions as constituting a displacement in a simulated space? What are the conditions required for participants to conceive of sensations as originating from the encounter with an object situated in this space? Finally, what are the conditions required for participants to recognise forms within this space? The results of the two experiments reported here show that, under certain conditions, participants can interpret the new sensorimotor laws as movements in a new perceptual space and can recognise simple geometric forms, and that this occurs no matter whether the sensory stimulation is presented in the visual or auditory modality.     

The end-state comfort effect facilitates joint action

Motor experts can accurately predict the future actions of others by observing their movements. This report describes three experiments that investigate such predictions in everyday object manipulations and test whether these predictions facilitate responses to the actions of others. Observing video excerpts showing an actor reaching for a vertically mounted dial, participants in Experiment 1 needed to predict how the actor would rotate it. Their predictions were specific to the direction and extent of the dial rotation and improved proportionate to the length of the video clip shown. Testing whether such predictions facilitate responses, in the subsequent experiments responders had to undo an actor's actions, back-rotating a dial (Exp 2) and a bar (Exp 3). The responders' actions were initiated faster when the actors' movements obeyed the so-called end-state comfort principle than when they did not. Our experiments show that humans exploit the end-state comfort effect to tweak their predictions of the future actions of others. The results moreover suggest that the precision of these predictions is mediated by perceptual learning rather than by motor simulation.     

Early evidence affects later decisions: Why evidence accumulation is required to explain response time data

Models of decision making differ in how they treat early evidence as it recedes in time. Standard models, such as the drift diffusion model, assume that evidence is gradually accumulated until it reaches a boundary and a decision is initiated. One recent model, the urgency gating model, has proposed that decision making does not require the accumulation of evidence at all. Instead, accumulation could be replaced by a simple urgency factor that scales with time. To distinguish between these fundamentally different accounts of decision making, we performed an experiment in which we manipulated the presence, duration, and valence of early evidence. We simulated the associated response time and error rate predictions from the drift diffusion model and the urgency gating model, fitting the models to the empirical data. The drift diffusion model predicted that variations in the evidence presented early in the trial would affect decisions later in that same trial. The urgency gating model predicted that none of these variations would have any effect. The behavioral data showed clear effects of early evidence on the subsequent decisions, in a manner consistent with the drift diffusion model. Our results cannot be explained by the urgency gating model, and they provide support for an evidence accumulation account of perceptual decision making.     

Three ideal observer models for rule learning in simple languages

Children learning the inflections of their native language show the ability to generalize beyond the perceptual particulars of the examples they are exposed to. The phenomenon of “rule learning”—quick learning of abstract regularities from exposure to a limited set of stimuli—has become an important model system for understanding generalization in infancy. Experiments with adults and children have revealed differences in performance across domains and types of rules. To understand the representational and inferential assumptions necessary to capture this broad set of results, we introduce three ideal observer models for rule learning. Each model builds on the next, allowing us to test the consequences of individual assumptions. Model 1 learns a single rule, Model 2 learns a single rule from noisy input, and Model 3 learns multiple rules from noisy input. These models capture a wide range of experimental results—including several that have been used to argue for domain-specificity or limits on the kinds of generalizations learners can make—suggesting that these ideal observers may be a useful baseline for future work on rule learning.     

Influence of swing leg movement on running stability

The aim of this study was to investigate the role of the swing leg movement on running stability. A simple model was used describing a forward hopping motion. The model consisted of two sub-models, namely a spring-mass system for the stance phase and a functional control model for the swing phase (represented by a passive or actively driven pendulum). To verify the main simulation results, an experimental study on treadmill running was performed. The results of the model indicated that for certain running speeds and pendulum lengths, the behavior of the mechanical system was stable. The following characteristic dependencies between the model parameters were observed. (1) Pendulum length and hip muscle activity determined running height and therefore swing duration. (2) Horizontal velocity was inversely related to leg angle of attack. Increased speed corresponded to flatter leg angles at touch-down, which is in agreement with experimental studies and previous predictions of spring-mass running. It was shown that a biologically motivated control approach with oscillating leg movements is well capable of generating stable hopping movements. Due to its simplicity, however, the monopedal model failed to explain more detailed mechanisms like the swing-leg to stance-leg interaction or the functional role of the leg segmentation. This simple model is therefore considered as a functional mechanical template for legged locomotion, which could help to build more elaborate models in the future.     

Internal models in the cerebellum

This review will focus on the possibility that the cerebellum contains an internal model or models of the motor apparatus. Inverse internal models can provide the neural command necessary to achieve some desired trajectory. First, we review the necessity of such a model and the evidence, based on the ocular following response, that inverse models are found within the cerebellar circuitry. Forward internal models predict the consequences of actions and can be used to overcome time delays associated with feedback control. Secondly, we review the evidence that the cerebellum generates predictions using such a forward model. Finally, we review a computational model that includes multiple paired forward and inverse models and show how such an arrangement can be advantageous for motor learning and control.     

Perceptual organization of two-dimensional patterns

The perceptual organization of image patterns is considered from 2 standpoints. First, a theoretical framework is presented from which computational models of perceptual organization can be constructed and tested. Second, a specific computational model for perceptual organization of line images is described. In this model, input images are first processed by a dense array of neurons that have properties consistent with recent analyses of single-neuron responses in primary visual cortex. Then, complex image structure is discovered by interleaved pattern-matching and grouping processes constrained by a generalized uniqueness principle. A series of 3-pattern grouping experiments was performed to test a restricted version of the model and to estimate critical parameters. Using the estimated parameters, an extended version of the model was tested by generating predictions for a series of "textbook" perceptual organization demonstrations.     

Comparison of decision learning models using the generalization criterion method

It is a hallmark of a good model to make accurate a priori predictions to new conditions ( Busemeyer & Wang, 2000 ). This study compared 8 decision learning models with respect to their generalizability. Participants performed 2 tasks (the Iowa Gambling Task and the Soochow Gambling Task), and each model made a priori predictions by estimating the parameters for each participant from 1 task and using those same parameters to predict on the other task. Three methods were used to evaluate the models at the individual level of analysis. The first method used a post hoc fit criterion, the second method used a generalization criterion for short-term predictions, and the third method again used a generalization criterion for long-term predictions. The results suggest that the models with the prospect utility function can make generalizable predictions to new conditions, and different learning models are needed for making short-versus long-term predictions on simple gambling tasks.     

The man who executed "imagined" movements: evidence for dissociable components of the body schema

We examined the nature of representations underlying motor imagery and execution in a patient (CW) with bilateral parietal lesions. When imagining hand movements, CW executed the imagined motor act but was unaware of the movements. These movements were significantly more accurate than volitional movements for the left but not right hand. CW also exhibited preserved motor imagery for the left but not right hand. Consistent with previous accounts, these findings suggest that motor imagery may normally involve the inhibition of movements. CW's unawareness of movements during motor imagery may reflect inattention or misattribution of the unexpected sensory feedback. Furthermore, in line with current models of motor control, motor imagery may depend on the integrity of a "forward model" derived from motor outflow information to generate a prediction of the consequences of a motor command. Such predictions appear to be preserved for imagery of left but not right hand movements in CW. Action may additionally depend on precise updating of effector position derived from the comparison of predicted and actual sensory information. We propose that CW's impaired volitional movements may be attributable to the degradation of such an updating mechanism.     

Extending Dynamical Systems Theory to Model Embodied Cognition

We define a mathematical formalism based on the concept of an ‘‘open dynamical system” and show how it can be used to model embodied cognition. This formalism extends classical dynamical systems theory by distinguishing a ‘‘total system’’ (which models an agent in an environment) and an ‘‘agent system’’ (which models an agent by itself), and it includes tools for analyzing the collections of overlapping paths that occur in an embedded agent's state space. To illustrate the way this formalism can be applied, several neural network models are embedded in a simple model environment. Such phenomena as masking, perceptual ambiguity, and priming are then observed. We also use this formalism to reinterpret examples from the embodiment literature, arguing that it provides for a more thorough analysis of the relevant phenomena.     

A neural network model for development of reaching and pointing based on the interaction of forward and inverse transformations

Pointing is one of the communicative actions that infants acquire during their first year of life. Based on a hypothesis that early pointing is triggered by emergent reaching behavior toward objects placed at out‐of‐reach distances, we proposed a neural network model that acquires reaching without explicit representation of ‘targets’. The proposed model controls a two‐joint arm in a horizontal plane, and it learns a loop of internal forward and inverse transformations; the former predicts the visual feedback of hand position and the latter generates motor commands from the visual input through random generation of the motor commands. In the proposed model, the motor output and visual input were represented by broadly tuned neural units. Even though explicit ‘targets’ were not presented during learning, the simulation successfully generated reaching toward visually presented objects at within‐reach and out‐of‐reach distances.     

Selective interference with image retention and generation: Evidence for the workspace model

We address three types of model of the relationship between working memory (WM) and long-term memory (LTM): (a) the gateway model, in which WM acts as a gateway between perceptual input and LTM; (b) the unitary model, in which WM is seen as the currently activated areas of LTM; and (c) the workspace model, in which perceptual input activates LTM, and WM acts as a separate workspace for processing and temporary retention of these activated traces. Predictions of these models were tested, focusing on visuospatial working memory and using dual-task methodology to combine two main tasks (visual short-term retention and image generation) with two interference tasks (irrelevant pictures and spatial tapping). The pictures selectively disrupted performance on the generation task, whereas the tapping selectively interfered with the retention task. Results are consistent with the predictions of the workspace model.     

Using reinforcement learning to understand the emergence of "intelligent" eye-movement behavior during reading

The eye movements of skilled readers are typically very regular (K. Rayner, 1998). This regularity may arise as a result of the perceptual, cognitive, and motor limitations of the reader (e.g., limited visual acuity) and the inherent constraints of the task (e.g., identifying the words in their correct order). To examine this hypothesis, reinforcement learning was used to allow an artificial "agent" to learn to move its eyes to read as efficiently as possible. The resulting patterns of simulated eye movements resembled those of skilled readers and suggest that important aspects of eye-movement behavior might emerge as a consequence of satisfying the constraints that are imposed on readers. These results also suggest novel interpretations of some contentious empirical results, such as the fixation duration costs associated with word skipping (R. Kliegl & R. Engbert, 2005), and theoretical assumptions, for example the familiarity check in the E-Z Reader model of eye-movement control (E. D. Reichle, A. Pollatsek, D. L. Fisher, & K. Rayner, 1998).     

Training of efficient oculomotor strategies enhances skill acquisition

The present study investigated the hypothesis that efficient oculomotor behaviours can be acquired through practice on a series of simple tasks and can be transferred subsequently to a complex visuomotor task, such as a video game. Each of two groups of subjects were exposed to a different set of simple tasks, or drills. One group, the efficient eye movement experimental group, received training designed to minimize eye movements and optimize scan path behaviours, whereas a second group of subjects, the inefficient eye movement experimental group, received training designed to increase the frequency of eye movements. Oculomotor training was interspersed with practice on the video game. Performance of these two experimental groups in the video game was compared to a control group playing the video game but receiving no specific training and matched for total time in the experiment. The group receiving efficient oculomotor training exhibited significantly superior performance in the video game and fewer foveations than either the inefficient or control groups, which did not differ from each other. Overall there was a significant inverse correlation between the number of foveations in the game and game score. The results of this study are discussed in terms of their implications for the importance of oculomotor training in the acquisition of any complex perceptual motor task.