Tie my hands, tie my eyes

Previous studies have demonstrated that motor abilities allow us not only to execute our own actions and to predict their consequences, but also to predict others' actions and their consequences. But just how deeply are motor abilities implicated in action observation? If an observer is prevented from acting while witnessing others' actions, will this impact on their making sense of others' behavior? We recorded proactive eye movements while participants observed an actor grasping objects. The participants' hands were either freely resting on the table or tied behind their back. Proactivity of gaze behavior was dramatically impaired when participants observed others' actions with their hands tied. Since we don't literally perceive actions with our hands, the effect may be explained by the hypothesis that effective observation of action depends not only on motor abilities but on being in a position to exercise them. This suggests, for the first time, that actions are observed best when we are actually in the position to perform them.     

The handedness of imagined bodies in action and the role of perspective taking

Past research at the nexus of motor control and perception investigated the role of perspective taking in many behavioral and neuroimaging studies. Some investigators addressed the issue of one's own vs. others' action imagination, but the possible effects of a front or a back view in imagining others' actions have so far been neglected. We report two 'single trial' experiments in which a total of 640 participants were asked to imagine a person performing a manual action - either in a front or in a back view - and then to indicate the hand used by the imagined person during movement execution. In such a task, we assume the existence of two distinct biases: a perceptual-mnemonic bias due to subjects' visual experience of others' actions, encouraging them to imagine right-handed movements, and a motor bias due to subjects' experience of self-made actions, encouraging them to imagine movements performed with the same hand as their dominant hand. We hypothesized that a greater involvement of motor representations in the back view compared to the front view could result in an increased correspondence between one's own manual preference and the hand used by the imagined agent in the former condition. The results of both experiments were consistent with this hypothesis, suggesting that while imagining others' actions we employ motor simulations in different degrees according to the perspective adopted.     

Resolving ambiguous behavioral intentions by means of involuntary prioritization of gaze processing

Anticipation of others' actions is of paramount importance in social interactions. Cues such as gaze direction and facial expressions can be informative, but can also produce ambiguity with respect to others' intentions. We investigated the combined effect of an actor's gaze and expression on judgments made by observers about the end-point of the actor's head rotation toward the observer. Expressions of approach gave rise to an unambiguous intention to move toward the observer, while expressions of avoidance gave rise to an ambiguous behavioral intention (as the expression and motion cues were in conflict). In the ambiguous condition, observers overestimated how far the actor's head had rotated when the actor's gaze was directed ahead of head rotation (compared to congruent or lagging behind). In the unambiguous condition the estimations were not influenced by the gaze manipulation. These results show that social cue integration does not follow simple additive rules, and suggests that the involuntary allocation of attention to another's gaze depends on the perceived ambiguity of the agent's behavioral intentions.     

The predictive mirror: interactions of mirror and affordance processes during action observation

An important question for the study of social interactions is how the motor actions of others are represented. Research has demonstrated that simply watching someone perform an action activates a similar motor representation in oneself. Key issues include (1) the automaticity of such processes, and (2) the role object affordances play in establishing motor representations of others' actions. Participants were asked to move a lever to the left or right to respond to the grip width of a hand moving across a workspace. Stimulus-response compatibility effects were modulated by two task-irrelevant aspects of the visual stimulus: the observed reach direction and the match between hand-grasp and the affordance evoked by an incidentally presented visual object. These findings demonstrate that the observation of another person's actions automatically evokes sophisticated motor representations that reflect the relationship between actions and objects even when an action is not directed towards an object.     

Intact and impaired mechanisms of action understanding in autism

Typically developing children understand and predict others' behavior by extracting and processing relevant information such as the logic of their actions within the situational constraints and the intentions conveyed by their gaze direction and emotional expressions. Children with autism have difficulties understanding and predicting others' actions. With the use of eye tracking and behavioral measures, we investigated action understanding mechanisms used by 18 children with autism and a well-matched group of 18 typically developing children. Results showed that children with autism (a) consider situational constraints in order to understand the logic of an agent's action and (b) show typical usage of the agent's emotional expressions to infer his or her intentions. We found (c) subtle atypicalities in the way children with autism respond to an agent's direct gaze and (d) marked impairments in their ability to attend to and interpret referential cues such as a head turn for understanding an agent's intentions.     

Long- and short-term plastic modeling of action prediction abilities in volleyball

Athletes show superior abilities not only in executing complex actions, but also in anticipating others' moves. Here, we explored how visual and motor experiences contribute to forge elite action prediction abilities in volleyball players. Both adult athletes and supporters were more accurate than novices in predicting the fate of volleyball floating services by viewing the initial ball trajectory, while only athletes could base their predictions on body kinematics. Importantly, adolescents assigned to physical practice training improved their ability to predict the fate of the actions by reading body kinematics, while those assigned to the observational practice training improved only in understanding the ball trajectory. The results suggest that physical and observational practice might provide complementary and mutually reinforcing contributions to the superior perceptual abilities of elite athletes. Moreover, direct motor experience is required to establish novel perceptuo-motor representations that are used to predict others' actions ahead of realization.     

Out of your hand's reach, out of my eyes' reach

When witnessing another's action, people recruit the same motor resources that enable them to efficiently perform that action, thus gazing at its target well before the agent's hand. But just to what extent does this recruitment help people in grabbing another's action target? If the latter seems to be out of the agent's reach, will this impact on people's gaze behaviour? We recorded proactive eye movements while participants witnessed someone else trying to reach for and grasp objects located either within or outside his reach. Proactivity of gaze was impaired when the targets were just out of the agent's reach. This effect is likely to be due to an interpersonal bodily space representation that allows one to map another's reaching space, thus prompting proactive eye movements towards the target just in case the agent is in the position to act upon it.     

A role for eyebrows in regulating the visibility of eye gaze direction

The human eye is unique amongst those of primates in having white sclera against which the dark iris is clearly visible. This high-contrast structure makes the gaze direction of a human potentially easily perceptible to others. For a social creature such as a human, the ability to perceive the direction of another's gaze may be very useful, since gaze usually signals attention. We report data showing that the accuracy of gaze deviation detection is independent of viewing distance up to a certain critical distance, beyond which it collapses. This is, of itself, surprising since most visual tasks are performed better at closer viewing distances. Our data also show that the critical distance, but not accuracy, is affected by the position of the eyebrows so that lowering the eyebrows reduces the critical distance. These findings show that mechanisms exist by which humans could expand or restrict the availability of their gaze direction to others. A way to regulate the availability of the gaze direction signal could be an advantage. We show that an interpretation of eyebrow function in these terms provides a novel explanation for several well-known eyebrow actions, including the eyebrow flash.     

Self-recognition of highly skilled actions: A study of orchestral conductors

The influence of movement skill on action representations and identification of agency was investigated. Point-light displays were created of highly skilled gestures of thirteen orchestral conductors in visual, auditory, and audiovisual versions and compared to two control conditions (static images and gait cycles of the same participants). In subsequent experimental sessions, participants indicated whether displays presented them or other conductors, whether the soundtrack contained their or others' musical interpretations, and rated the quality and emotional content of the gestures. Self-recognition was more accurate in conditions presenting highly skilled conducting movements as compared to other displays. Participants judged the quality of their own movements to be better than those of others, independently of whether or not they recognized movement agency. Emotional content was perceived accurately across conditions both for own and others' actions. These results point to the influence of dynamical characteristics of motor skill, rather than merely type of movement or emotional content, on action representations and self-other identification.     

Did you see that? Dissociating advanced visual information and ball flight constrains perception and action processes during one-handed catching

The integration of separate, yet complimentary, cortical pathways appears to play a role in visual perception and action when intercepting objects. The ventral system is responsible for object recognition and identification, while the dorsal system facilitates continuous regulation of action. This dual-system model implies that empirically manipulating different visual information sources during performance of an interceptive action might lead to the emergence of distinct gaze and movement pattern profiles. To test this idea, we recorded hand kinematics and eye movements of participants as they attempted to catch balls projected from a novel apparatus that synchronised or de-synchronised accompanying video images of a throwing action and ball trajectory. Results revealed that ball catching performance was less successful when patterns of hand movements and gaze behaviours were constrained by the absence of advanced perceptual information from the thrower's actions. Under these task constraints, participants began tracking the ball later, followed less of its trajectory, and adapted their actions by initiating movements later and moving the hand faster. There were no performance differences when the throwing action image and ball speed were synchronised or de-synchronised since hand movements were closely linked to information from ball trajectory. Results are interpreted relative to the two-visual system hypothesis, demonstrating that accurate interception requires integration of advanced visual information from kinematics of the throwing action and from ball flight trajectory.     

Does how I look at what you're doing depend on what I'm doing?

Previous studies showed that people proactively gaze at the target of another's action by taking advantage of their own motor representation of that action. But just how selectively is one's own motor representation implicated in another's action processing? If people observe another's action while performing a compatible or an incompatible action themselves, will this impact on their gaze behaviour? We recorded proactive eye movements while participants observed an actor grasping small or large objects. The participants' right hand either freely rested on the table or held with a suitable grip a large or a small object, respectively. Proactivity of gaze behaviour significantly decreased when participants observed the actor reaching her target with a grip that was incompatible with respect to that used by them to hold the object in their own hand. This indicates that effective observation of action may depend on what one is actually doing, being actions observed best when the suitable motor representations may be readily recruited.     

Predictive tracking over occlusions by 4-month-old infants

Two experiments investigated how 16-20-week-old infants visually tracked an object that oscillated on a horizontal trajectory with a centrally placed occluder. To determine the principles underlying infants' tendency to shift gaze to the exiting side before the object arrives, occluder width, oscillation frequency, and motion amplitude were manipulated resulting in occlusion durations between 0.20 and 1.66 s. Through these manipulations, we were able to distinguish between several possible modes of behavior underlying 'predictive' actions at occluders. Four such modes were tested. First, if passage-of-time determines when saccades are made, the tendency to shift gaze over the occluder is expected to be a function of time since disappearance. Second, if visual salience of the exiting occluder edge determines when saccades are made, occluder width would determine the pre-reappearance gaze shifts but not oscillation frequency, amplitude, or velocity. Third, if memory of the duration of the previous occlusion determines when the subjects shift gaze over the occluder, it is expected that the gaze will shift after the same latency at the next occlusion irrespective of whether occlusion duration is changed or not. Finally, if infants base their pre-reappearance gaze shifts on their ability to represent object motion (cognitive mode), it is expected that the latency of the gaze shifts over the occluder is scaled to occlusion duration. Eye and head movements as well as object motion were measured at 240 Hz. In 49% of the passages, the infants shifted gaze to the opposite side of the occluder before the object arrived there. The tendency to make such gaze shifts could not be explained by the passage of time since disappearance. Neither could it be fully explained in terms of visual information present during occlusion, i.e. occluder width. On the contrary, it was found that the latency of the pre-reappearance gaze shifts was determined by the time of object reappearance and that it was a function of all three factors manipulated. The results suggest that object velocity is represented during occlusion and that infants track the object behind the occluder in their 'mind's eye'.     

Bridging the gap between the other and me: the functional role of motor resonance and action effects in infants' imitation

This paper investigates a two-stage model of infants' imitative learning from observed actions and their effects. According to this model, the observation of another person's action activates the corresponding motor code in the infants' motor repertoire (i.e. leads to motor resonance). The second process guiding imitative behavior results from the observed action effects. If the modeled action is followed by a salient action effect, the representation of this effect (i.e. perceptual code) will be associated with the activated motor code. If the infant later aims to obtain the same effect, the corresponding motor program will be activated and the model's action will therefore be imitated. Accordingly, the model assumes that for the imitation of novel actions the modeled action needs to elicit sufficient motor resonance and must be followed by a salient action effect. Using the head touch imitation paradigm, we tested these two assumptions derived from the model. To this end, we manipulated whether the actions demonstrated to the infants were or were not in the motor repertoire, i.e. elicited stronger or less strong motor resonance, and whether they were followed by salient action effects or not. The results were in line with the proposed two-stage model of infants' imitative learning and suggest that motor resonance is necessary, but not sufficient for infants' imitative learning from others' actions and their effects.     

Action experience in infancy predicts visual-motor functional connectivity during action anticipation

Despite substantial evidence indicating a close link between action production and perception in early child development, less is known about how action experience shapes the processes of perceiving and anticipating others’ actions. Here, we developed a novel approach to capture functional connectivity specific to certain brain areas to investigate how action experience changes the networks involved in action perception and anticipation. Nine- and-12-month-old infants observed familiar (grasping) and novel (tool-use) actions while their brain activity was measured using EEG. Infants’ motor competence of both actions was assessed. A link between action experience and connectivity patterns was found, particularly during the anticipation period. During action anticipation, greater motor competence in grasping predicted greater functional connectivity between visual (occipital alpha) and motor (central alpha) regions relative to global levels of whole-brain EEG connectivity. Furthermore, visual and motor regions tended to be more coordinated in response to familiar versus novel actions and for older than younger participants. Critically, these effects were not found in the control networks (frontal-central; frontal-occipital; parietal-central; parietal-occipital), suggesting a unique role of visual-motor networks on the link between motor skills and action encoding.

Highlights

• Infants’ motor development predicted functional connectivity patterns during action anticipation.
• Faster graspers, and older infants, showed a stronger ratio of visual-motor neural coherence.
• Overall whole-brain connectivity was modulated by age and familiarity with the actions.
• Measuring inter-site relative to whole-brain connectivity can capture specific brain-behavior links.
• Measures of phase-based connectivity over time are sensitive to anticipatory action.

     

Temporal coordination between performing musicians

Many common behaviours require people to coordinate the timing of their actions with the timing of others' actions. We examined whether representations of musicians' actions are activated in coperformers with whom they must coordinate their actions in time and whether coperformers simulate each other's actions using their own motor systems during temporal coordination. Pianists performed right-hand melodies along with simple or complex left-hand accompaniments produced by themselves or by another pianist. Individual performers' preferred performance rates were measured in solo performance of the right-hand melody. The complexity of the left-hand accompaniment influenced the temporal grouping structure of the right-hand melody in the same way when it was performed by the self or by the duet partner, providing some support for the action corepresentation hypothesis. In contrast, accompaniment complexity had little influence on temporal coordination measures (asynchronies and cross-correlations between parts). Temporal coordination measures were influenced by a priori similarities between partners' preferred rates; partners who had similar preferred rates in solo performance were better synchronized and showed mutual adaptation to each other's timing during duet performances. These findings extend previous findings of action corepresentation and action simulation to a task that requires precise temporal coordination of independent yet simultaneous actions.     

Differential reliance of chimpanzees and humans on automatic and deliberate control of motor actions

Humans are often unaware of how they control their limb motor movements. People pay attention to their own motor movements only when their usual motor routines encounter errors. Yet little is known about the extent to which voluntary actions rely on automatic control and when automatic control shifts to deliberate control in nonhuman primates. In this study, we demonstrate that chimpanzees and humans showed similar limb motor adjustment in response to feedback error during reaching actions, whereas attentional allocation inferred from gaze behavior differed. We found that humans shifted attention to their own motor kinematics as errors were induced in motor trajectory feedback regardless of whether the errors actually disrupted their reaching their action goals. In contrast, chimpanzees shifted attention to motor execution only when errors actually interfered with their achieving a planned action goal. These results indicate that the species differed in their criteria for shifting from automatic to deliberate control of motor actions. It is widely accepted that sophisticated motor repertoires have evolved in humans. Our results suggest that the deliberate monitoring of one’s own motor kinematics may have evolved in the human lineage.     

Self produced and observed actions influence emotion: the roles of action fluency and eye gaze

Affective responses to objects can be influenced by cognitive processes such as perceptual fluency. Here we investigated whether the quality of motor interaction with an object influences affective response to the object. Participants grasped and moved objects using either a fluent action or a non-fluent action (avoiding an obstacle). Liking ratings were higher for objects in the fluent condition. Two further studies investigated whether the fluency of another person’s actions influences affective response. Observers watched movie clips of the motor actions described above, in conditions where the observed actor could be seen to be looking towards the grasped object, or where the actor’s head and gaze were not visible. Two results were observed: First, when the actor’s gaze cannot be seen, liking ratings of the objects are reduced. Second, action fluency of observed actions does influence liking ratings, but only when the actor’s gaze towards the object is visible. These findings provide supporting evidence for the important role of observed eye gaze in action simulation, and demonstrate that non-emotive actions can evoke empathic states in observers. This research was supported by Economic and Social Research Council grant RES-000-23-0429 awarded to S. P. Tipper and A. E. Hayes.     

Visual attention and action: How cueing,direct mapping,and social interactions drive orienting

Despite considerable interest in both action perception and social attention over the last 2 decades, there has been surprisingly little investigation concerning how the manual actions of other humans orient visual attention. The present review draws together studies that have measured the orienting of attention, following observation of another’s goal-directed action. Our review proposes that, in line with the literature on eye gaze, action is a particularly strong orienting cue for the visual system. However, we additionally suggest that action may orient visual attention using mechanisms, which gaze direction does not (i.e., neural direct mapping and corepresentation). Finally, we review the implications of these gaze-independent mechanisms for the study of attention to action. We suggest that our understanding of attention to action may benefit from being studied in the context of joint action paradigms, where the role of higher level action goals and social factors can be investigated.     

How infants relate looker and object: evidence for a perceptual learning account of gaze following in infancy

In two experiments, it was investigated how preverbal infants perceive the relationship between a person and an object she is looking at. More specifically, it was examined whether infants interpret an adult's object-directed gaze as a marker of an intention to act or whether they relate the person and the object via a mechanism of associative learning. Fourteen-month-old infants observed an adult gazing repeatedly at one of two objects. When the adult reached out to grasp this object in the test trials, infants showed no systematic visual anticipations to it (i.e. first visual anticipatory gaze shifts) but only displayed longer looking times for this object than for another before her hand reached the object. However, they showed visual anticipatory gaze shifts to the correct action target when only the grasping action was presented. The second experiment shows that infants also look longer at the object a person has been gazing at when the person is still present, but is not performing any action during the test trials. Looking preferences for the objects were reversed, however, when the person was absent during the test trials. This study provides evidence for the claim that infants around 1 year of age do not employ other people's object-directed gaze to anticipate future actions, but to establish person-object associations. The implications of this finding for theoretical conceptions of infants' social-cognitive development are discussed.     

Do the eyes have it? Cues to the direction of social attention

The face communicates an impressive amount of visual information. We use it to identify its owner, how they are feeling and to help us understand what they are saying. Models of face processing have considered how we extract such meaning from the face but have ignored another important signal - eye gaze. In this article we begin by reviewing evidence from recent neurophysiological studies that suggests that the eyes constitute a special stimulus in at least two senses. First, the structure of the eyes is such that it provides us with a particularly powerful signal to the direction of another person's gaze, and second, we may have evolved neural mechanisms devoted to gaze processing. As a result, gaze direction is analysed rapidly and automatically, and is able to trigger reflexive shifts of an observer's visual attention. However, understanding where another individual is directing their attention involves more than simply analysing their gaze direction. We go on to describe research with adult participants, children and non-human primates that suggests that other cues such as head orientation and pointing gestures make significant contributions to the computation of another's direction of attention.