Event files: feature binding in and across perception and action

The primate brain codes perceived events in a distributed fashion, which raises the question of how the codes referring to the same event are related to each other. Recent findings suggest that they are integrated into 'object files', episodic bindings of object-related information. However, the problem of integrating distributed codes is not restricted to perception but applies to action planning and sensorimotor processing as well. Here I argue that the brain addresses these problems by creating multi-layered networks of bindings - 'event files' - that temporarily link codes of perceptual events, the current task context, and the actions performed therein. These bindings produce systematic but often surprising and counter-intuitive interactions between, and impairments in, perception and action planning.     

Action induction through action observation

Ideomotor movements may arise in individuals while they watch goal-directed actions or events. In a previous study we developed a paradigm for investigating ideomotor movements induced through watching the outcome of one's own action. In the present study we extended the paradigm to investigate both movements induced through watching the outcome of one's own as well as somebody else's action (player mode and observer mode respectively). We report three experiments, each with differing conditions for the player mode, but identical conditions for the observer mode. Results indicate that in both modes ideomotor movements are governed by two basic principles: Perceptual induction and intentional induction. In the player mode we replicated and extended previous findings, indicating dissociation between hand and head movements. In the observer mode no such dissociation was obtained. Our findings suggest that people perform, in their own actions, what they see being performed in other people's actions. Induction of action through observation can pertain to both the action's physical surface and underlying intentions. Furthermore, our results suggest that perceptual induction is ubiquitous but may be locally suspended for intentional action control. We discuss our results in the framework of theories invoking a strong overlap between representational structures for action perception and action planning.     

The time of perception and the other way around

The world we perceive is delayed in relation to its flowing content, as well as the outcome of our actions on the world in relation to the moment we decide to act. This mosaic of different latencies permeating both perception and action has to be taken into account critically in order for us to cope with the temporal challenges constantly imposed by the environment. Fundamental notions, such as the sense of agency and causality, depend on the temporal relationship of events occurring in well-defined windows of time. Here, we offer a broad, yet abridged, historical view of some thought-provoking issues concerning the time of perception and action. From the pioneering work of Wundt, Titchener, and Libet to recent findings and ideas related to the employment of visual illusions as psychophysical probes (such as the flash-lag effect), we have tried to expose some problems inherent to the act of measuring the time of both perception and action, and devise possible solutions as well.     

A neuroanatomical model of passivity phenomena

Any attempt to elucidate the nature and mechanism of passivity phenomena, i.e., experiences that one's conscious actions or thoughts have not been 'willed' by oneself, requires an integrative philosophical-neurobiological approach. The model proposed here adopts some fundamental positions that have long been advocated by philosophers and theoretical psychologists and have now found support from functional neuroanatomy. First, we experience our actions not from the standpoint of the executive but through the perception of its effects. Second, the 'self' is not an agent of behaviour. Third, behaviour is energized and integrated by basic drives (instincts). Fourth, the view that the experience of an acting self is related to drive reduction associated with voluntary actions is perhaps less well developed. The model thus proposes that passivity phenomena are actions that are induced by the perception of salient events but that are not integrated with or conducive to the overall motivational state of the organism. It has been suggested that, following the perception of salient events, competition arises between automatic response tendencies seeking expression. The prefrontal cortex appears to play an important role not only in determining which events are to be perceived but also which of the corresponding response dispositions is to be selected and actualized in overt behaviour. Thus, action selection is the outcome of competition between response tendencies in the context of prefrontal biasing signals that represent drives and strivings for goals. Action selection may be uncoupled from drives and strivings as a result of a lowering of the threshold for action selection--as is suggested to be the case in schizophrenic passivity phenomena--or due to disconnection from prefrontal regions--as may be the case in the alien limb syndrome.     

The Theory of Event Coding (TEC): a framework for perception and action planning

Traditional approaches to human information processing tend to deal with perception and action planning in isolation, so that an adequate account of the perception-action interface is still missing. On the perceptual side, the dominant cognitive view largely underestimates, and thus fails to account for, the impact of action-related processes on both the processing of perceptual information and on perceptual learning. On the action side, most approaches conceive of action planning as a mere continuation of stimulus processing, thus failing to account for the goal-directedness of even the simplest reaction in an experimental task. We propose a new framework for a more adequate theoretical treatment of perception and action planning, in which perceptual contents and action plans are coded in a common representational medium by feature codes with distal reference. Perceived events (perceptions) and to-be-produced events (actions) are equally represented by integrated, task-tuned networks of feature codes--cognitive structures we call event codes. We give an overview of evidence from a wide variety of empirical domains, such as spatial stimulus-response compatibility, sensorimotor synchronization, and ideomotor action, showing that our main assumptions are well supported by the data.     

Infants’ developing expectations of possible and impossible tool-use events between ages 8 and 12 months

Infants’ developing causal expectations for the outcome of a simple tool-use event from ages 8 to 12 months were investigated. Causal expectations were studied by comparing infants’ developing tool-use actions (i.e. as tool-use agents) with their developing perceptual reactions (i.e. as tool-use observers) to possible and impossible tool-use events. In Experiment 1, tool-use actions were studied by presenting infants, ages 8 and 12 months, with tool-use object-retrieval problems. In Experiment 2, a second age-matched sample of infants watched a comparable series of possible and impossible tool-use events in which a tool was used to retrieve a goal-object. Two core related findings were made. First, infants’ causal action and causal perception develop in parallel. In both action and perception, supporting tool-use develops before surrounding tool-use. Second, infants’ tool-use action develops before their causal perception of comparable tool-use events. The findings support the constructivist hypothesis that infants’ causal actions may develop before and inform their causal perceptions.     

Transfer of calibration in length perception by dynamic touch

Earlier studies suggested that the calibration of actions is functionally, rather than anatomically, specific; thus, calibration of an action ought to transfer to actions that serve the same goal (Rieser, Pick, Ashmead, & Garing, 1995). In the present study, we investigated whether the calibration of perception also follows a functional organization: If one means of detecting an information variable is recalibrated, are other means of detection recalibrated as well? In two experiments, visual feedback was used to recalibrate perceived length of a rod wielded by the right hand; the recalibration was found to transfer to length perception with the left hand. This implies that calibration in perception is organized functionally rather than anatomically, and supports the general view that calibration applies to functional systems.     

Actions and Events: Some Semantical Considerations

Since the publication of Davidson's influential article 'The Logical Form of Action Sentences', semantical considerations are widely thought to support the doctrine that actions are events. I shall argue that the semantics of action sentences do not imply that actions are events. This will involve defending a negative claim and a positive claim, as well as a proposal for how to formalize action sentences. The negative claim is that the semantics of action sentences do not require that we think of actions as events, even if these sentences are best formalized in the manner that Davidson himself favours . The positive claim is that the simplest way of formalizing actions sentences which captures all and only licit inferences requires quantification only , over the results of actions. If this is right, then the argument from semantics evaporates, and the claim that actions are events needs to be freshly argued for – or against.     

Depth perception by the active observer

The connection between perception and action has classically been studied in one direction only: the effect of perception on subsequent action. Although our actions can modify our perceptions externally, by modifying the world or our view of it, it has recently become clear that even without this external feedback the preparation and execution of a variety of motor actions can have an effect on three-dimensional perceptual processes. Here, we review the ways in which an observer's motor actions--locomotion, head and eye movements, and object manipulation--affect his or her perception and representation of three-dimensional objects and space. Allowing observers to act can drastically change the way they perceive the third dimension, as well as how scientists view depth perception.     

Unconscious modulation of the conscious experience of voluntary control

How does the brain generate our experience of being in control over our actions and their effects? Here, we argue that the perception of events as self-caused emerges from a comparison between anticipated and actual action-effects: if the representation of an event that follows an action is activated before the action, the event is experienced as caused by one's own action, whereas in the case of a mismatch it will be attributed to an external cause rather than to the self. In a subliminal priming paradigm we show that participants overestimated how much control they had over objectively uncontrollable stimuli, which appeared after free- or forced-choice actions, when a masked prime activated a representation of the stimuli immediately before each action. This prime-induced control-illusion was independent from whether primes were consciously perceived. Results indicate that the conscious experience of control is modulated by unconscious anticipations of action-effects.     

Perceptual resonance: action-induced modulation of perception

A direct relationship between perception and action implies bi-directionality, and predicts not only effects of perception on action but also effects of action on perception. Modern theories of social cognition have intensively examined the relation from perception to action and propose that mirroring the observed actions of others underlies action understanding. Here, we suggest that this view is incomplete, as it neglects the perspective of the actor. We will review empirical evidence showing the effects of self-generated action on perceptual judgments. We propose that producing action might prime perception in a way that observers are selectively sensitive to related or similar actions of conspecifics. Therefore, perceptual resonance, not motor resonance, might be decisive for grounding sympathy and empathy and, thus, successful social interactions.     

Does own experience affect perception of others’ actions in capuchin monkeys (Cebus apella)?

Anticipating another’s actions is an important ability in social animals. Recent research suggests that in human adults and infants one’s own action experience facilitates understanding and anticipation of others’ actions. We investigated the link between first-person experience and perception of another’s action in adult tufted capuchin monkeys (Sapajus apella spp., formerly Cebus apella spp.). In Experiment 1, the monkeys observed a familiar human (actor) trying to open a container using either a familiar or an unfamiliar action. They looked for longer when the actor tried to open the container using a familiar action. In Experiment 2, the actor performed two novel actions on a new container. The monkeys looked equally at the two actions. In Experiment 3, the monkeys were trained to open the container using one of the novel actions in Experiment 2. After training, we repeated the same procedure as in Experiment 2. The monkeys looked for longer when the actor manipulated the container using the action they had practiced than when she used the unfamiliar action. These results show that knowledge derived from one’s own experience impacts perception of another’s action in these New World monkeys.     

Experience matters: the impact of doing versus watching on infants' subsequent perception of tool-use events

Prior work suggests that active experience affects infants' understanding of simple actions. The present studies compared the impact of active and observational experience on infants' ability to identify the goal of a novel tool-use event. Infants either received active training and practice in using a cane to retrieve an out-of-reach toy or had matched observational experience before taking part in a habituation paradigm that we used to assess infants' ability to identify the goal of another person's tool-use acts. Active training alone facilitated 10-month-old infants' ability to identify the goal of the tool-use event. Active experience using tools may enable infants to build motor representations of tool-use events that subsequently guide action perception and support action understanding.     

The role of cue-response mapping in motorvisual impairment and facilitation: evidence for different roles of action planning and action control in motorvisual dual-task priming

Previous research has shown that actions impair the visual perception of categorically action-consistent stimuli. On the other hand, actions can also facilitate the perception of spatially action-consistent stimuli. We suggest that motorvisual impairment is due to action planning processes, while motorvisual facilitation is due to action control mechanisms. This implies that because action planning is sensitive to modulations by cue-response mapping so should motorvisual impairment, while motorvisual facilitation should be insensitive to manipulations of cue-response mapping as is action control. We tested this prediction in three dual-task experiments. The impact of performing left and right key presses on the perception of unrelated, categorically or spatially consistent, stimuli was studied. As expected, we found motorvisual impairment for categorically consistent stimuli and motorvisual facilitation for spatially consistent stimuli. In all experiments, we compared congruent with incongruent cue-key mappings. Mapping manipulations affected motorvisual impairment, but not motorvisual facilitation. The results support our suggestion that motorvisual impairment is due to action planning, and motorvisual facilitation to action control.     

At one with our actions,but at two with our bodies

Jennifer Hornsby's account of human action frees us from the temptation to think of the person who acts as ‘doing’ the events that are her actions, and thereby removes much of the allure of ‘agent causation’. But her account is spoiled by the claim that physical actions are ‘tryings’ that cause bodily movements. It would be better to think of physical actions and bodily movements as identical; but Hornsby refuses to do this, seemingly because she thinks that to do so would be to endorse the so–called ‘standard causal story’. But Hornsby misses a possibility here, for we can insist on this identity claim without endorsing the standard story if we embrace an account which parallels the disjunctive account in the philosophy of perception. This will leave us with a picture of physical action that saves the insights of Hornsby's account without succumbing to its distortions.     

Infants' Perception of Causal Chains

Three experiments examined infants' and adults' perception of causal sequences of events. In a causal-chain sequence, the first action causes a second action that then causes a final outcome; in a temporal-chain sequence, the first two actions are independent and the second action causes a final outcome. Infants and adults were shown the same event sequences; infants were tested using a visual habituation paradigm, whereas adults were given a questionnaire. Experiment 1 indicated that 15-month-old infants perceive the primary cause of the final outcome to be the first action in a causal chain but the second action in a temporal chain. Experiment 2 showed that adults interpret the causal sequences in a manner similar to that of 15-month-olds. Finally, Experiment 3 showed that 10-month-old infants do not yet perceive causal sequences in the same manner as 15-month-olds and adults. These results are interpreted in terms of both infants' developing knowledge of causal events and adults' attributions of causality in complex events.     

The valence of action outcomes modulates the perception of one's actions

When interacting with the world, we need to distinguish whether sensory information results from external events or from our own actions. The nervous system most likely draws this distinction by comparing the actual sensory input with an internal prediction about the sensory consequences of one's actions. However, interacting with the world also requires an evaluation of the outcomes of self-action, e.g. in terms of their affective valence. Here we show that subjects' perceived pointing direction does not only depend on predictive and sensory signals related to the performed action itself, but also on the affective valence of the action outcome: subjects perceived their movements as directed towards positive and away from negative outcomes. Our findings suggest that the non-conceptual perception of the sensory consequences of self-action builds on both sensorimotor information related directly to self-action and a post hoc evaluation of the affective action outcome.     

Baby do–baby see!: How action production influences action perception in infants

With a series of four experiments we show that self-produced actions influence infants’ perception of actions performed by others. After having played with an object, 7–11-month-olds simultaneously watched two videos presenting adults who act on either the same object or a different one. The 9- and 11-month-old preferred to watch the same-object video indicating an influence of action production on action perception at this age. Follow-up studies showed that this influence was restricted to object-related actions. Agentive experience enhanced interest in actions with objects, but not in watching objects or persons per se. These findings indicate that infants are not only interested in acting on objects themselves, but that this experience increases their interest in the actions of other people with the same object.     

The mirror reflects both ways: Action influences perception of others

Substantial evidence links perception of others’ bodies and mental representation of the observer’s own body; however, the overwhelming majority of this evidence is unidirectional, showing influence from perception to action. It has been proposed that the influence also runs from action to perception, but to date the evidence is scant. Here we report that ordinary motor actions performed by the subject affect concurrent psychophysical judgments of human-body stimuli. Subjects remained unaware of the connection between the action and the main task. The results show that perception can change as a result of the observer’s ongoing actions.