Action matters: The role of action plans and object affordances in selection for action

In a series of three experiments requiring selection of real objects for action, we investigated whether characteristics of the planned action and/or the “affordances” of target and distractor objects affected interference caused by distractors. In all ofthe experiments, the target object was selectedon the basis of colour and was presented alone or with a distractor object. We examined the effect of type of response (button press, grasping, or pointing), object affordances (compatibility with the acting hand, affordances for grasping or pointing), and target/distractor positions (left or right) on distractor interference (reaction time differences between trials with and without distractors). Different patterns of distractor interference were associated with different motor responses. In the button-press conditions of each experiment, distractor interference was largely determined by perceptual salience (e.g., proximity to initial visual fixation). In contrast, in tasks requiring action upon the objects in the array, distractors with handles caused greater interference than those without handles, irrespective of whether the intended action was pointing or grasping. Additionally, handled distractors were relatively more salient when their affordances for grasping were strong (handle direction compatible with the acting hand) than when affordances were weak. These data suggest that attentional highlighting of specific target and distractor features is a function of intended actions.     

Action representations activated by task‐irrelevant information: Is it really irrelevant?

Accessing action knowledge is believed to rely on the activation of action representations through the retrieval of functional, manipulative, and spatial information associated with objects. However, it remains unclear whether action representations can be activated in this way when the object information is irrelevant to the current judgment. The present study investigated this question by independently manipulating the correctness of three types of action‐related information: the functional relation between the two objects, the grip applied to the objects, and the orientation of the objects. In each of three tasks in Experiment 1, participants evaluated the correctness of only one of the three information types (function, grip or orientation). Similar results were achieved with all three tasks: “correct” judgments were facilitated when the other dimensions were correct; however, “incorrect” judgments were facilitated when the other two dimensions were both correct and also when they were both incorrect. In Experiment 2, when participants attended to an action‐irrelevant feature (object color), there was no interaction between function, grip, and orientation. These results clearly indicate that action representations can be activated by retrieval of functional, manipulative, and spatial knowledge about objects, even though this is task‐irrelevant information.     

Concurrent Cortical Representations of Function- and Size-Related Object Affordances: An fMRI Study

Previous work has shown that the perception of a graspable object may automatically potentiate actions that are tailored to specific action-related features of the object (e.g., its size) and may be related to its immediate grasping as well as to its long-term, functional use. We investigated the neural correlates of function- and size-related object affordances that may be concurrently potentiated by a graspable object. Participants were lying in a MR scanner holding a large switch in one hand and a small switch in the other hand. They passively attended a large or a small object with clearly separated functional and graspable end that was displayed centrally at an average angle of 45 degrees. Participants responded to the direction of an arrow that was overlaid on the object after a mean period of 1,000 ms after object onset and was pointing to the left or to the right with equal probability. Response times were shorter when the arrow pointed to the functional end of the object and when the responses were made with the switch that was congruent to the size of the perceived object. A clear distinction was found in the representation of function- and size-related affordances; the former was represented in the posterior parietal cortex and the latter in prefrontal, premotor, and primary sensorimotor cortices. We conclude that different aspects of object-directed actions may be automatically potentiated by individual object features and are represented in distinct brain areas.     

The impact of object carriage on independent locomotion

The current study examined whether carrying objects in one's hands influenced different parameters associated with independent locomotion. Specifically, 14- and 24-month-olds walked in a straight path under four conditions of object carriage – no object (control), one object carried in one hand (one object-one hand), two objects carried in each of the hands (two objects-two hands), and one object carried in both hands simultaneously (one object-two hands). Although carrying objects failed to influence a variety of kinematic parameters of gait, it did affect children's arm postures, with children adopting less mature arm positions when carrying objects. Finally, arm position was related to walking skill, but only for older children when they were not carrying objects. These findings indicate that although a relation does exist between arm positions and gait parameters, this relation is easily disrupted by carrying loads, even small ones.     

Dissociating object-based and space-based affordances

In what we term the orientation effect, faster spatial responses are made to the corresponding task irrelevant orientation of an object. We ask how this effect relates to object affordances, how attention may be involved, and how the effect relates to the better understood Simon effect. Two separate stimulus-response compatibility effects (an orientation effect and a Simon effect) were observed when spatial responses were made to photographs of objects whose orientation and location had been simultaneously manipulated. When attentional demands were high these separate effects were found using hand responses and foot responses, suggesting an abstract rather than specific coding of object affordances. However, when attentional demands were low only the Simon effect was observed, suggesting that, in order to obtain the orientation effect, objects must be represented at the level of an object.     

Conflict in the kitchen: Contextual modulation of responsiveness to affordances

Recently, human behavior has been considered the product of continuous interactions between perception, cognition and action in which “affordances” (action possibilities the environment has to offer) play an important role. Converging evidence suggests that multiple action possibilities simultaneously compete for further processing, while external and internal factors (e.g., incoming sensory information, predictions) bias this competition. In the present study we used a stop-task to investigate whether context is able to modulate the strength of the responsiveness to affordances. We therefore placed participants in an actual kitchen and workshop during electroencephalographic recordings. A faster response to context congruent objects demonstrated that the direct surrounding is able to affect responsiveness to affordances. In addition, when responses needed to be withheld, context congruent objects evoked greater response conflict as indicated by an enhanced N2 Event Related Potential (ERP) component.     

The Sound of Grasp Affordances: Influence of Grasp‐Related Size of Categorized Objects on Vocalization

Previous research shows that simultaneously executed grasp and vocalization responses are faster when the precision grip is performed with the vowel [i] and the power grip is performed with the vowel [ɑ]. Research also shows that observing an object that is graspable with a precision or power grip can activate the grip congruent with the object. Given the connection between vowel articulation and grasping, this study explores whether grasp‐related size of observed objects can influence not only grasp responses but also vowel pronunciation. The participants had to categorize small and large objects into natural and manufactured categories by pronouncing the vowel [i] or [ɑ]. As predicted, [i] was produced faster when the object's grasp‐related size was congruent with the precision grip while [ɑ] was produced faster when the size was congruent with the power grip (Experiment 1). The effect was not, however, observed when the participants were presented with large objects that are not typically grasped by the power grip (Experiment 2). This study demonstrates that vowel production is systematically influenced by grasp‐related size of a viewed object, supporting the account that sensory‐motor processes related to grasp planning and representing grasp‐related properties of viewed objects interact with articulation processes. The paper discusses these findings in the context of size–sound symbolism, suggesting that mechanisms that transform size‐grasp affordances into corresponding grasp‐ and articulation‐related motor programs might provide a neural basis for size‐sound phenomena that links small objects with closed‐front vowels and large objects with open‐back vowels.     

Visually guided grasping of common objects: Effects of priming

Priming effects were tested on the planning of the grasping of common objects under full vision during action performance. Healthy participants took part in four experiments which manipulated the nature of the prime (objects, circular block, rectangular bar) and priming context (blocked vs. mixed). Each experiment relied on four priming conditions: (1) congruent orientation, (2) incongruent orientation, (3) neutral prime, and (4) no prime. Priming was observed to have a facilitating effect on visually guided grasping when the object to be grasped was primed by a congruently oriented identical object. This effect was rather independent of the priming context (experimental set-up). Our data suggest an object's functional identity may contribute to the priming effect, as well as its intrinsic (e.g., shape, size) and extrinsic (orientation) visual characteristics. We showed that the planning of visually guided grasping is influenced by prior visual experience, and thus that grasping is not based exclusively on real-time processing of visual information.     

Time course of action representations evoked during sentence comprehension

The nature of hand-action representations evoked during language comprehension was investigated using a variant of the visual–world paradigm in which eye fixations were monitored while subjects viewed a screen displaying four hand postures and listened to sentences describing an actor using or lifting a manipulable object. Displayed postures were related to either a functional (using) or volumetric (lifting) interaction with an object that matched or did not match the object mentioned in the sentence. Subjects were instructed to select the hand posture that matched the action described in the sentence. Even before the manipulable object was mentioned in the sentence, some sentence contexts allowed subjects to infer the object's identity and the type of action performed with it and eye fixations immediately favored the corresponding hand posture. This effect was assumed to be the result of ongoing motor or perceptual imagery in which the action described in the sentence was mentally simulated. In addition, the hand posture related to the manipulable object mentioned in a sentence, but not related to the described action (e.g., a writing posture in the context of a sentence that describes lifting, but not using, a pencil), was favored over other hand postures not related to the object. This effect was attributed to motor resonance arising from conceptual processing of the manipulable object, without regard to the remainder of the sentence context.     

Infant perception of a manual pick-up event

Manual pick-up of an object is a simple causal event frequently observed by infants. A habituation-recovery of looking technique with filmed stimuli is used in experiments which seek to investigate aspects of the perception and encoding of such events in infants. In the first study with 28-week-olds, it is found that ‘lateral mirror-image’ pick-ups are hardly discriminable, while a change in the contact relation of hand and object is readily discriminable. In the second, again with 28-week-olds, the discriminability of the contact relation appears to be specific to a dynamic context involving a hand (rather than another inanimate object). The results of a further experiment make it appear unlikely that the previous results were simply due to the partial occlusion of the picked-up object produced by the grasping. The implications of these results for infant perception of causality are briefly considered.     

On the relations between action planning, object identification, and motor representations of observed actions and objects

Recent evidence suggests that viewing a static prime object (a hand grasp), can activate action representations that affect the subsequent identification of graspable target objects. The present study explored whether stronger effects on target object identification would occur when the prime object (a hand grasp) was made more action-rich and dynamic. Of additional interest was whether this type of action prime would affect the generation of motor activity normally elicited by the target object. Three experiments demonstrated that grasp observation improved the identification of grasp-congruent target objects relative to grasp-incongruent target objects. We argue from this data that identifying a graspable object includes the processing of its action-related attributes. In addition, grasp observation was shown to influence the motor activity elicited by the target object, demonstrating interplay between action-based and object-based motor coding.     

Affordance Compatibility Effect for Word Learning in Virtual Reality

Rich sensorimotor interaction facilitates language learning and is presumed to ground conceptual representations. Yet empirical support for early stages of embodied word learning is currently lacking. Finding evidence that sensorimotor interaction shapes learned linguistic representations would provide crucial support for embodied language theories. We developed a gamified word learning experiment in virtual reality in which participants learned the names of six novel objects by grasping and manipulating objects with either their left or right hand. Participants then completed a word–color match task in which they were tested on the same six words and objects. Participants were faster to respond to stimuli in the match task when the response hand was compatible with the hand used to interact with the named object, an effect we refer to as affordance compatibility. In two follow up experiments, we found that merely observing virtual hands interact with the objects was sufficient to acquire a smaller affordance compatibility effect, and we found that the compatibility effect was driven primarily by responses with a compatible hand and not by responses in a compatible spatial location. Our results support theoretical views of language which ground word representations in sensorimotor experiences, and they suggest promising future routes to explore the sensorimotor foundations of higher cognition through immersive virtual experiments.     

Object representation guides infants' reaching in the dark.

Infants were presented with two sounding objects of different sizes in light and dark, in which sound cued the object's identity. Reaching behavior was assessed to determine if object size influenced preparation for grasping the object. In both light and dark, infants aligned their hands when contacting the large object compared with the small object, which resulted in a reach with both hands extended for the large object and reach with one hand more extended for the small object. Infants contacted the large object more frequently on the bottom and sides rather than the top, where the sound source was located. Reaching in the dark by 6 1/2-month-olds is not merely directed toward a sound source but rather shows preparation in relation to the object's size. These findings were interpreted as evidence that mental representation of previously seen objects can guide subsequent motor action by 6 1/2-month-old infants.     

Motor inhibition associated with the affordance of briefly displayed objects

Research has demonstrated that left- and right-hand responses are facilitated when they are performed with the hand compatible with the orientation of a viewed object. This suggests that graspable objects automatically activate the motor representations that correspond to their orientation. It has recently been proposed that similar positive stimulus–response compatibility effects (PCE) may turn into negative compatibility effects (NCE) when a prime object is displayed very briefly. These NCEs are suggested to reflect motor inhibition mechanisms—motor activation triggered by briefly viewed objects may be treated by the motor system as unwanted, and thus it is rapidly inhibited. We examined whether the motor activation triggered by the orientation of a task-irrelevant object is similarly inhibited when the object is displayed briefly. In Experiment 1, a NCE was observed between the orientation of an object and the responding hand when the object was displayed for 30 or 70 ms. The effect turned into a PCE when the object was displayed for 370 ms. Experiment 2 confirmed that this motor inhibition effect was produced by the handle affordance of the object rather than some abstract visual properties of the object.     

Motor inhibition associated with the affordance of briefly displayed objects

Research has demonstrated that left- and right-hand responses are facilitated when they are performed with the hand compatible with the orientation of a viewed object. This suggests that graspable objects automatically activate the motor representations that correspond to their orientation. It has recently been proposed that similar positive stimulus-response compatibility effects (PCE) may turn into negative compatibility effects (NCE) when a prime object is displayed very briefly. These NCEs are suggested to reflect motor inhibition mechanisms--motor activation triggered by briefly viewed objects may be treated by the motor system as unwanted, and thus it is rapidly inhibited. We examined whether the motor activation triggered by the orientation of a task-irrelevant object is similarly inhibited when the object is displayed briefly. In Experiment 1, a NCE was observed between the orientation of an object and the responding hand when the object was displayed for 30 or 70 ms. The effect turned into a PCE when the object was displayed for 370 ms. Experiment 2 confirmed that this motor inhibition effect was produced by the handle affordance of the object rather than some abstract visual properties of the object.     

Action comprehension: deriving spatial and functional relations

A perceived action can be understood only when information about the action carried out and the objects used are taken into account. It was investigated how spatial and functional information contributes to establishing these relations. Participants observed static frames showing a hand wielding an instrument and a potential target object of the action. The 2 elements could either match or mismatch, spatially or functionally. Participants were required to judge only 1 of the 2 relations while ignoring the other. Both irrelevant spatial and functional mismatches affected judgments of the relevant relation. Moreover, the functional relation provided a context for the judgment of the spatial relation but not vice versa. The results are discussed in respect to recent accounts of action understanding.     

Action-related objects influence the distribution of visuospatial attention

Previous studies have shown that attention is drawn to the location of manipulable objects and is distributed across pairs of objects that are positioned for action. Here, we investigate whether central, action-related objects can cue attention to peripheral targets. Experiment 1 compared the effect of uninformative arrow and object cues on a letter discrimination task. Arrow cues led to spatial-cueing benefits across a range of stimulus onset asynchronies (SOAs: 0 ms, 120 ms, 400 ms), but object-cueing benefits were slow to build and were only significant at the 400-ms SOA. Similar results were found in Experiment 2, in which the targets were objects that could be either congruent or incongruent with the cue (e.g., screwdriver and screw versus screwdriver and glass). Cueing benefits were not influenced by the congruence between the cue and target, suggesting that the cueing effects reflected the action implied by the central object, not the interaction between the objects. For Experiment 3 participants decided whether the cue and target objects were related. Here, the interaction between congruent (but not incongruent) targets led to significant cueing/positioning benefits at all three SOAs. Reduced cueing benefits were obtained in all three experiments when the object cue did not portray a legitimate action (e.g., a bottle pointing towards an upper location, since a bottle cannot pour upwards), suggesting that it is the perceived action that is critical, rather than the structural properties of individual objects. The data suggest that affordance for action modulates the allocation of visual attention.     

Perception and production of object-related grasping in 6-month-olds

In this study, 6-month-olds’ perception of an object-related human grasping action was compared with their level of grasping performance using a within-participants design. In the action perception task, infants were presented with the video of an actor’s grasping movement toward an occluded target object. Subsequently, an expected and an unexpected final state of this grasping movement were presented simultaneously, and infants’ looking times were measured. In the action production task, infants were presented with three graspable objects. Infants’ grasping behavior was coded to be either palmar or thumb-opposite grasping. Results indicate that infants who were already able to perform a thumb-opposite grasp differentiated between the two final states in the action perception task by looking longer toward the unexpected final state. In contrast, infants who showed only palmar grasps looked equally long toward both final states. This finding supports the assumption that action perception and action control are already closely related in infants as young as 6 months.     

Distinguishing the effects of action relations and scene context on object perception

Objects are rarely viewed in isolation, and so how they are perceived is influenced by the context in which they are viewed and their interaction with other objects (e.g., whether objects are colocated for action). We investigated the combined effects of action relations and scene context on an object decision task. Experiment 1 investigated whether the benefit for positioning objects so that they interact is enhanced when objects are viewed within contextually congruent scenes. The results indicated that scene context influenced perception of nonaction-related objects (e.g., monitor and keyboard), but had no effect on responses to action-related objects (e.g., bottle and glass) that were processed more rapidly. In Experiment 2, we reduced the saliency of the object stimuli and found that, under these circumstances, scene context influenced responses to action-related objects. We discuss the data in terms of relatively late effects of scene processing on object perception.     

Binocular vision enhances a rapidly evolving affordance priming effect: Behavioural and TMS evidence

Extensive research has suggested that simply viewing an object can automatically prime compatible actions for object manipulation, known as affordances. Here we explored the generation of covert motor plans afforded by real objects with precision (‘pinchable’) or whole-hand/power (‘graspable’) grip significance under different types of vision. In Experiment 1, participants viewed real object primes either monocularly or binocularly and responded to orthogonal auditory stimuli by making precision or power grips. Pinchable primes facilitated congruent precision grip responses relative to incongruent power grips, and vice versa for graspable primes, but only in the binocular vision condition. To examine the temporal evolution of the binocular affordance effect, participants in Experiment 2 always viewed the objects binocularly but made no responses, instead receiving a transcranial magnetic stimulation pulse over their primary motor cortex at three different times (150, 300, 450 ms) after prime onset. Motor evoked potentials (MEPs) recorded from a pinching muscle were selectively increased when subjects were primed with a pinchable object, whereas MEPs from a muscle associated with power grips were increased when viewing graspable stimuli. This interaction was obtained both 300 and 450 ms (but not 150 ms) after the visual onset of the prime, characterising for the first time the rapid development of binocular grip-specific affordances predicted by functional accounts of the affordance effect.