Exploiting redundancy for flexible behavior: unsupervised learning in a modular sensorimotor control architecture

Autonomously developing organisms face several challenges when learning reaching movements. First, motor control is learned unsupervised or self-supervised. Second, knowledge of sensorimotor contingencies is acquired in contexts in which action consequences unfold in time. Third, motor redundancies must be resolved. To solve all 3 of these problems, the authors propose a sensorimotor, unsupervised, redundancy-resolving control architecture (SURE_REACH), based on the ideomotor principle. Given a 3-degrees-of-freedom arm in a 2-dimensional environment, SURE_REACH encodes 2 spatial arm representations with neural population codes: a hand end-point coordinate space and an angular arm posture space. A posture memory solves the inverse kinematics problem by associating hand end-point neurons with neurons in posture space. An inverse sensorimotor model associates posture neurons with each other action-dependently. Together, population encoding, redundant posture memory, and the inverse sensorimotor model enable SURE_REACH to learn and represent sensorimotor grounded distance measures and to use dynamic programming to reach goals efficiently. The architecture not only solves the redundancy problem but also increases goal reaching flexibility, accounting for additional task constraints or realizing obstacle avoidance. While the spatial population codes resemble neurophysiological structures, the simulations confirm the flexibility and plausibility of the model by mimicking previously published data in arm-reaching tasks.     

Control of multimovement coordination: sensorimotor mechanisms in speech motor programming

The present paper provides some hypotheses concerning the role of sensorimotor mechanisms in the coordination and programming of multimovement behaviors. The primary database is from experiments on the control of speech, a motor behavior that inherently requires multimovement coordination. From these data, it appears that coordination may be implemented by calibrated, sensorimotor actions which couple multiple movements for the accomplishment of common functional goals. The data from speech and select observations in other motor systems also reveal that these sensorimotor linkages are task-dependent and may underlie the intermovement motor equivalence that characterizes many natural motor behaviors. In this context, it is hypothesized also that motor learning may involve the calibration of these intermovement sensorimotor actions. These observations in turn provide some alternative perspectives on the concept of a motor program, primarily suggesting that individual movements and muscle contractions are not wholly prespecified, but shaped by sensorimotor adjustments.     

Control of Multimovement Coordination

The present paper provides some hypotheses concerning the role of sensorimotor mechanisms in the coordination and programming of multimovement behaviors. The primary database is from experiments on the control of speech, a motor behavior that inherently requires multimovement coordination. From these data, it appears that coordination may be implemented by calibrated, sensorimotor actions which couple multiple movements for the accomplishment of common functional goals. The data from speech and select observations in other motor systems also reveal that these sensorimotor linkages are task-dependent and may underlie the intermovement motor equivalence that characterizes many natural motor behaviors. In this context, it is hypothesized also that motor learning may involve the calibration of these intermovement sensorimotor actions. These observations in turn provide some alternative perspectives on the concept of a motor program, primarily suggesting that individual movements and muscle contractions are not wholly prespecified, but shaped by sensorimotor adjustments.     

Multiple systems of spatial memory and action

Recent findings from spatial cognition and cognitive neuroscience suggest that different types of mental representations could mediate the off-line retrieval of spatial relations from memory and the on-line guidance of motor actions in space. As a result, a number of models proposing multiple systems of spatial memory have been recently formulated. In the present article we review these models and we evaluate their postulates based on available experimental evidence. Furthermore, we discuss how a multiple-system model can apply to situations in which people reason about their immediate surroundings or non-immediate environments by incorporating a model of sensorimotor facilitation/interference. This model draws heavily on previous accounts of sensorimotor interference and takes into account findings from the stimulus-response compatibility literature.     

Isometric force complexity may not fully originate from the nervous system

In humans and animals, spatial and temporal information from the nervous system are translated into muscle force enabling movements of body segments. To gain deeper understanding of this translation of information into movements, we investigated the motor control dynamics of isometric contractions in children, adolescents, young adults and older adults. Twelve children, thirteen adolescents, fourteen young adults, and fifteen older adults completed two minutes of submaximal isometric plantar- and dorsiflexion. Simultaneously, sensorimotor cortex EEG, tibialis anterior and soleus EMG and plantar- and dorsiflexion force was recorded. Surrogate analysis suggested that all signals were from a deterministic origin. Multiscale entropy analysis revealed an inverted U-shape relationship between age and complexity for the force but not for the EEG and EMG signals. This suggests that temporal information in from the nervous system is modulated by the musculoskeletal system during the transmission into force. The entropic half-life analyses indicated that this modulation increases the time scale of the temporal dependency in the force signal compared to the neural signals. Together this indicates that the information embedded in produced force does not exclusively reflect the information embedded in the underlying neural signal.     

Space coding for sensorimotor transformations can emerge through unsupervised learning

The posterior parietal cortex (PPC) is fundamental for sensorimotor transformations because it combines multiple sensory inputs and posture signals into different spatial reference frames that drive motor programming. Here, we present a computational model mimicking the sensorimotor transformations occurring in the PPC. A recurrent neural network with one layer of hidden neurons (restricted Boltzmann machine) learned a stochastic generative model of the sensory data without supervision. After the unsupervised learning phase, the activity of the hidden neurons was used to compute a motor program (a population code on a bidimensional map) through a simple linear projection and delta rule learning. The average motor error, calculated as the difference between the expected and the computed output, was less than 3°. Importantly, analyses of the hidden neurons revealed gain-modulated visual receptive fields, thereby showing that space coding for sensorimotor transformations similar to that observed in the PPC can emerge through unsupervised learning. These results suggest that gain modulation is an efficient coding strategy to integrate visual and postural information toward the generation of motor commands.     

Know thy sound: Perceiving self and others in musical contexts

This review article provides a summary of the findings from empirical studies that investigated recognition of an action's agent by using music and/or other auditory information. Embodied cognition accounts ground higher cognitive functions in lower level sensorimotor functioning. Action simulation, the recruitment of an observer's motor system and its neural substrates when observing actions, has been proposed to be particularly potent for actions that are self-produced. This review examines evidence for such claims from the music domain. It covers studies in which trained or untrained individuals generated and/or perceived (musical) sounds, and were subsequently asked to identify who was the author of the sounds (e.g., the self or another individual) in immediate (online) or delayed (offline) research designs. The review is structured according to the complexity of auditory–motor information available and includes sections on: 1) simple auditory information (e.g., clapping, piano, drum sounds), 2) complex instrumental sound sequences (e.g., piano/organ performances), and 3) musical information embedded within audiovisual performance contexts, when action sequences are both viewed as movements and/or listened to in synchrony with sounds (e.g., conductors' gestures, dance). This work has proven to be informative in unraveling the links between perceptual–motor processes, supporting embodied accounts of human cognition that address action observation. The reported findings are examined in relation to cues that contribute to agency judgments, and their implications for research concerning action understanding and applied musical practice.     

Deactivation in the sensorimotor area during observation of a human agent performing robotic actions

It is well established that several motor areas, called the mirror-neuron system (MNS), are activated when an individual observes other’s actions. However, whether the MNS responds similarly to robotic actions compared with human actions is still controversial. The present study investigated whether and how the motor area activity is influenced by appearance (human vs. robot) and/or kinematics (human vs. robot) of the observed action using near-infrared spectroscopy. The results showed that there was a strong interaction between these factors, revealing strong deactivations in sensorimotor areas when the subject saw a human agent performing robotic actions, which was significantly different from responses when observing the human agent acting in a human way and the robot agent performing robotic actions. These results indicate that MNS activity is sensitive to congruency between the appearance and kinematics of the agent, especially when the agent has a human appearance. We discuss the experience-dependent characteristics of MNS sensitivity to observed actions.     

Space coding for sensorimotor transformations can emerge through unsupervised learning

The posterior parietal cortex (PPC) is fundamental for sensorimotor transformations because it combines multiple sensory inputs and posture signals into different spatial reference frames that drive motor programming. Here, we present a computational model mimicking the sensorimotor transformations occurring in the PPC. A recurrent neural network with one layer of hidden neurons (restricted Boltzmann machine) learned a stochastic generative model of the sensory data without supervision. After the unsupervised learning phase, the activity of the hidden neurons was used to compute a motor program (a population code on a bidimensional map) through a simple linear projection and delta rule learning. The average motor error, calculated as the difference between the expected and the computed output, was less than 3°. Importantly, analyses of the hidden neurons revealed gain-modulated visual receptive fields, thereby showing that space coding for sensorimotor transformations similar to that observed in the PPC can emerge through unsupervised learning. These results suggest that gain modulation is an efficient coding strategy to integrate visual and postural information toward the generation of motor commands.     

Sensorimotor intentionality: The origins of intentionality in prospective agent action

Efficient prospective motor control, evident in human activity from birth, reveals an adaptive intentionality of a primary, pre-reflective, and pre-conceptual nature that we identify here as sensorimotor intentionality. We identify a structural continuity between the emergence of this earliest form of prospective movement and the structure of mental states as intentional or content-directed in more advanced forms. We base our proposal on motor control studies, from foetal observations through infancy. These studies reveal movements are guided by anticipations of future effects, even from before birth. This implies that these movements, even if they are simple and discrete, are the actions of an intentional agent. We develop this notion to present a theory of the developing organisation of a core feature of cognition as embodied agent action, from early single actions with proximal prospectivity to the complex serial ordering of actions into projects to reach distal goals. We claim the prospective structural continuity from early and simple actions to later complex projects of serially-ordered actions confirms the existence of an ontogenetically primary form of content–directedness that is a driver for learning and development. Its implications for understanding autism are discussed.     

Baby steps: investigating the development of perceptual–motor couplings in infancy

There are cells in our motor cortex that fire both when we perform and when we observe similar actions. It has been suggested that these perceptual‐motor couplings in the brain develop through associative learning during correlated sensorimotor experience. Although studies with adult participants have provided support for this hypothesis, there is no direct evidence that associative learning also underlies the initial formation of perceptual–motor couplings in the developing brain. With the present study we addressed this question by manipulating infants’ opportunities to associate the visual and motor representation of a novel action, and by investigating how this influenced their sensorimotor cortex activation when they observed this action performed by others. Pre‐walking 7–9‐month‐old infants performed stepping movements on an infant treadmill while they either observed their own real‐time leg movements (Contingent group) or the previously recorded leg movements of another infant (Non‐contingent control group). Infants in a second control group did not perform any steps and only received visual experience with the stepping actions. Before and after the training period we measured infants’ sensorimotor alpha suppression, as an index of sensorimotor cortex activation, while they watched videos of other infants’ stepping actions. While we did not find greater sensorimotor alpha suppression following training in the Contingent group as a whole, we nevertheless found that the strength of the visuomotor contingency experienced during training predicted the amount of sensorimotor alpha suppression at post‐test in this group. We did not find any effects of motor experience alone. These results suggest that the development of perceptual–motor couplings in the infant brain is likely to be supported by associative learning during correlated visuomotor experience.     

Neurocognitive Contributions to Motor Skill Learning: The Role of Working Memory

ABSTRACT

Researchers have begun to delineate the precise nature and neural correlates of the cognitive processes that contribute to motor skill learning. The authors review recent work from their laboratory designed to further understand the neurocognitive mechanisms of skill acquisition. The authors have demonstrated an important role for spatial working memory in 2 different types of motor skill learning, sensorimotor adaptation and motor sequence learning. They have shown that individual differences in spatial working memory capacity predict the rate of motor learning for sensorimotor adaptation and motor sequence learning, and have also reported neural overlap between a spatial working memory task and the early, but not late, stages of adaptation, particularly in the right dorsolateral prefrontal cortex and bilateral inferior parietal lobules. The authors propose that spatial working memory is relied on for processing motor error information to update motor control for subsequent actions. Further, they suggest that working memory is relied on during learning new action sequences for chunking individual action elements together.     

A hybrid POMDP-BDI agent architecture with online stochastic planning and plan caching

This article presents an agent architecture for controlling an autonomous agent in stochastic, noisy environments. The architecture combines the partially observable Markov decision process (POMDP) model with the belief-desire-intention (BDI) framework. The Hybrid POMDP-BDI agent architecture takes the best features from the two approaches, that is, the online generation of reward-maximizing courses of action from POMDP theory, and sophisticated multiple goal management from BDI theory. We introduce the advances made since the introduction of the basic architecture, including (i) the ability to pursue and manage multiple goals simultaneously and (ii) a plan library for storing pre-written plans and for storing recently generated plans for future reuse. A version of the architecture is implemented and is evaluated in a simulated environment. The results of the experiments show that the improved hybrid architecture outperforms the standard POMDP architecture and the previous basic hybrid architecture for both processing speed and effectiveness of the agent in reaching its goals.     

Frames of Reference for Human Perceptual-Motor Coordination: Space-Based Versus Joint-Based Adaptation

In this study, the authors addressed the issue of whether space-based motor planning occurs at a higher, equal, or lower level of central nervous system control than joint-based motor planning by using a computerized adaptation paradigm. Visual displays of participants' (N = 32) reaching movements to spatial targets were distorted either with respect to spatial hand displacements (space-based distortion) or with respect to joint angle displacements (joint-based distortion). Participants adapted more easily to space-based distortion than to joint-based distortion. The results suggest that when the participants were confronted with new visuomotor mappings, they aimed for virtual spatial targets whose positions were adjusted to compensate for the distortions associated with the new mappings. That strategy was preferred over a joint- or posture-based strategy, in which a posture is selected for the displayed spatial target and is then modified so that the new mapping between adopted and seen positions can be accommodated. The results support the widely held view that space-based planning occurs at a higher level than joint-based planning.     

Frames of reference for human perceptual-motor coordination: space-based versus joint-based adaptation

In this study, the authors addressed the issue of whether space-based motor planning occurs at a higher, equal, or lower level of central nervous system control than joint-based motor planning by using a computerized adaptation paradigm. Visual displays of participants' (N = 32) reaching movements to spatial targets were distorted either with respect to spatial hand displacements (space-based distortion) or with respect to joint angle displacements (joint-based distortion). Participants adapted more easily to space-based distortion than to joint-based distortion. The results suggest that when the participants were confronted with new visuomotor mappings, they aimed for virtual spatial targets whose positions were adjusted to compensate for the distortions associated with the new mappings. That strategy was preferred over a joint- or posture-based strategy, in which a posture is selected for the displayed spatial target and is then modified so that the new mapping between adopted and seen positions can be accommodated. The results support the widely held view that space-based planning occurs at a higher level than joint-based planning.     

Motor expertise modulates movement processing in working memory

A substantial amount of literature has demonstrated individuals' tendency to code verbally a series of movements for subsequent recall. However, the mechanisms underlying movement encoding remain unclear. In this paper, I argue that sensorimotor expertise influences the involvement of motor processes to store movements in working memory. Experts in motor activities and individuals with limited motor expertise were compared in three experimental conditions assessing movement recall: (a) without suppression task, (b) with verbal suppression, and (c) with motor suppression. Athletes outperformed controls in movement recall, but the suppression tasks affected the two groups differently. Verbal suppression affected controls more than athletes, whereas the effect was reversed with motor suppression. Together, these findings suggest that controls and athletes favor different mechanisms to encode movements, either based on verbal or on motor processes, providing further evidence for a tight relationship between sensorimotor and cognitive processes.     

Comparison of grasping movements made by healthy subjects in a 3-dimensional immersive virtual versus physical environment

Virtual reality (VR) technology is being used with increasing frequency as a training medium for motor rehabilitation. However, before addressing training effectiveness in virtual environments (VEs), it is necessary to identify if movements made in such environments are kinematically similar to those made in physical environments (PEs) and the effect of provision of haptic feedback on these movement patterns. These questions are important since reach-to-grasp movements may be inaccurate when visual or haptic feedback is altered or absent. Our goal was to compare kinematics of reaching and grasping movements to three objects performed in an immersive three-dimensional (3D) VE with haptic feedback (cyberglove/grasp system) viewed through a head-mounted display to those made in an equivalent physical environment (PE). We also compared movements in PE made with and without wearing the cyberglove/grasp haptic feedback system. Ten healthy subjects (8 women, 62.1 ± 8.8 years) reached and grasped objects requiring 3 different grasp types (can, diameter 65.6 mm, cylindrical grasp; screwdriver, diameter 31.6 mm, power grasp; pen, diameter 7.5 mm, precision grasp) in PE and visually similar virtual objects in VE. Temporal and spatial arm and trunk kinematics were analyzed. Movements were slower and grip apertures were wider when wearing the glove in both the PE and the VE compared to movements made in the PE without the glove. When wearing the glove, subjects used similar reaching trajectories in both environments, preserved the coordination between reaching and grasping and scaled grip aperture to object size for the larger object (cylindrical grasp). However, in VE compared to PE, movements were slower and had longer deceleration times, elbow extension was greater when reaching to the smallest object and apertures were wider for the power and precision grip tasks. Overall, the differences in spatial and temporal kinematics of movements between environments were greater than those due only to wearing the cyberglove/grasp system. Differences in movement kinematics due to the viewing environment were likely due to a lack of prior experience with the virtual environment, an uncertainty of object location and the restricted field-of-view when wearing the head-mounted display. The results can be used to inform the design and disposition of objects within 3D VEs for the study of the control of prehension and for upper limb rehabilitation.     

A bio-inspired model of behavior considering decision-making and planning,spatial attention and basic motor commands processes

Cognitive architectures (CA) are an IA approach to implement computer systems with human-like behavior. Fundamental exhibited human capabilities include planning and decision-making. In that regard, numerous AI systems successfully exhibit human-like behavior but are limited to either achieving specific objectives or are restrained to too heavily constrained environments, which makes them unsuitable in the presence of unforeseen situations where autonomy is required. To try to alleviate the problem, we present a bio-inspired computational model to solve the autonomous navigation problem of a computational entity in a controlled context. This proposal is the result of the interaction between planning and decision-making, spatial attention and the motor cognitive functions. The proposed model is based on neuroscientific evidence concerning the involved cognitive functions and is part of a more general cognitive architecture. In the case study developed to validate our idea, we can see that the processes previously identified play an important role to accomplish spatial navigation. In the case study presented, an agent achieves the navigation over an unexplored maze from an initial to a final position successfully. The reunited results motivate us to continue improving our model considering attentional information to influence the agent’s motor behavior.     

Probabilistic models in human sensorimotor control

Sensory and motor uncertainty form a fundamental constraint on human sensorimotor control. Bayesian decision theory (BDT) has emerged as a unifying framework to understand how the central nervous system performs optimal estimation and control in the face of such uncertainty. BDT has two components: Bayesian statistics and decision theory. Here we review Bayesian statistics and show how it applies to estimating the state of the world and our own body. Recent results suggest that when learning novel tasks we are able to learn the statistical properties of both the world and our own sensory apparatus so as to perform estimation using Bayesian statistics. We review studies which suggest that humans can combine multiple sources of information to form maximum likelihood estimates, can incorporate prior beliefs about possible states of the world so as to generate maximum a posteriori estimates and can use Kalman filter-based processes to estimate time-varying states. Finally, we review Bayesian decision theory in motor control and how the central nervous system processes errors to determine loss functions and select optimal actions. We review results that suggest we plan movements based on statistics of our actions that result from signal-dependent noise on our motor outputs. Taken together these studies provide a statistical framework for how the motor system performs in the presence of uncertainty.     

Motor and visual codes interact to facilitate visuospatial memory performance

The spatial working memory system constantly updates spatial representations and many studies have focused on the underlying principles of the encoding and maintenance of visual information. Here we investigated the question of how the production of actions influences spatial working memory. Participants were given a task that required concurrent maintenance of two spatial arrays, one encoded by visual observation accompanied with pointing movements, the other by only visual observation. Across two experiments, movement during encoding was found to facilitate recognition of spatial arrays in a load-dependent manner. The results suggest an action-based encoding principle within the working memory system, and possible underlying action-related mechanisms are discussed.