Feature integration across perception and action: event files affect response choice

Five experiments investigated the spontaneous integration of stimulus and response features. Participants performed simple, prepared responses (R1) to the mere presence of Go signals (S1) before carrying out another, freely chosen response (R2) to another stimulus (S2), the main question being whether the likelihood of repeating a response depends on whether or not the stimulus, or some of its features, are repeated. Indeed, participants were more likely to repeat the previous response if stimulus form or color was repeated than if it was alternated. The same was true for stimulus location, but only if location was made task-relevant, whether by defining the response set in terms of location, by requiring the report of S2 location, or by having S1 to be selected against a distractor. These findings suggest that task-relevant stimulus and response features are spontaneously integrated into independent, local event files, each linking one stimulus to one response feature. Upon reactivation of one member of the binary link activation is spread to the other, thereby increasing the likelihood to repeat a response if one or more stimulus features are repeated. These findings support the idea that both perceptual events and action plans are cognitively represented in terms of their features, and that feature-integration processes cross borders between perception and action.     

Perceptual and attentional factors in encoding irrelevant spatial information

Numerous studies found superior performance when the irrelevant location of a stimulus and response location were corresponding than when they were not corresponding (Simon effect), suggesting that stimulus location is processed in an obligatory manner. The present study compared Simon effects from the location of a relevant (i.e., to-be-attended) object to those from the location of an irrelevant (i.e., to-be-ignored) object. In four experiments, participants were presented with a rectangular frame and a square, with the relevant object in green or red color and the irrelevant object in gray or white color. Participants’ task was to respond with a lateral keypress to the color of the relevant object, and we varied spatial correspondence between the location of the relevant or the irrelevant object and the response, respectively. Results consistently showed larger Simon effects from the location of the relevant than from the irrelevant object, even when the irrelevant object was made very salient. These results suggest that location processing is largely confined to relevant (i.e., attended) objects, stressing the role of attention shifts for location encoding.     

S-R compatibility and response selection

In serial stage models, perception and action are usually thought to be linked to each other by an S-R translation mechanism. However, phenomena of S-R compatibility suggest a more direct relationship between perceptual and action domains. We discuss behavioral and psychophysiological evidence that irrelevant stimulus information automatically activates response codes, but then decays over time.

In a series of reaction time studies and electrophysiological experiments, we investigated both temporal and functional properties of the assumed automatic response activation process. We found that the amount of interference due to irrelevant spatial information depends upon how long its availability precedes that of the information relevant for response selection. This indicates that response activation decays rather quickly. If response-relevant and irrelevant spatial information are simultaneously available, electrophysiological measurements show that automatic activation of the spatially corresponding response rises soon after stimulus onset, but then dissipates and gets replaced by the activation of the response indicated by the relevant stimulus attribute.

We conclude that these findings do not support a pure translation account, but rather suggest the presence of two parallel and (at least partially) independent routes from perception to action: A direct route, allowing for automatic activation of response codes if stimulus and response features overlap, and an indirect route linking S and R codes in an arbitrary manner. Via the direct route responses may be primed independent of task-specific contingencies, while the correct response is selected via the indirect route. This use suggests that (a) the transmission of stimulus information to response stages does not (fully) depend on task relevance and that (b) different stimulus features can be transmitted asynchronously and independently from one another.     

The Simon task with multi-component responses: two loci of response–effect compatibility

The stimulus–response compatibility (SRC) effect refers to the phenomenon that responses are faster and more accurate when stimulus and response correspond than when they do not. The phenomenon is robust in that it is observed even when SRC is irrelevant to performing the task, a variant known as the Simon effect. Recent studies also demonstrated that responses are faster when they are spatially compatible with their effects in the environment (action effects) than when they are incompatible. This response–effect (R–E) compatibility effect is thought to stem from the fact that stimuli first activate anticipated effect codes, which then activate corresponding action codes. In the present study, the Simon task was used to examine influences of multiple response components on performance. Three response components were orthogonally manipulated. The results of three experiments indicated that there are two separate processes that are influenced by R–E compatibility; one that is responsible for the SRC effect (S–R translation) and the other that is independent of SRC (action programming). The influence of R–E compatibility on the former process depended on manipulations that varied attentional demands of the task.     

Intentional control of attention: action planning primes action-related stimulus dimensions

Neurophysiological observations suggest that attending to a particular perceptual dimension, such as location or shape, engages dimension-related action, such as reaching and prehension networks. Here we reversed the perspective and hypothesized that activating action systems may prime the processing of stimuli defined on perceptual dimensions related to these actions. Subjects prepared for a reaching or grasping action and, before carrying it out, were presented with location- or size-defined stimulus events. As predicted, performance on the stimulus event varied with action preparation: planning a reaching action facilitated detecting deviants in location sequences whereas planning a grasping action facilitated detecting deviants in size sequences. These findings support the theory of event coding, which claims that perceptual codes and action plans share a common representational medium, which presumably involves the human premotor cortex.     

Binding object features to locations: Does the “spatial congruency bias” update with object movement?

One of the fundamental challenges of visual cognition is how our visual systems combine information about an object’s features with its spatial location. A recent phenomenon related to object–location binding, the “spatial congruency bias,” revealed that two objects are more likely to be perceived as having the same identity or features if they appear in the same spatial location, versus if the second object appears in a different location. The spatial congruency bias suggests that irrelevant location information is automatically encoded with and bound to other object properties, biasing perceptual judgments. Here we further explored this new phenomenon and its role in object–location binding by asking what happens when an object moves to a new location: Is the spatial congruency bias sensitive to spatiotemporal contiguity cues, or does it remain linked to the original object location? Across four experiments, we found that the spatial congruency bias remained strongly linked to the original object location. However, under certain circumstances—for instance, when the first object paused and remained visible for a brief time after the movement—the congruency bias was found at both the original location and the updated location. These data suggest that the spatial congruency bias is based more on low-level visual information than on spatiotemporal contiguity cues, and reflects a type of object–location binding that is primarily tied to the original object location and that may only update to the object’s new location if there is time for the features to be re-encoded and rebound following the movement.     

How much attention does an event file need?

Attentional requirements for the spontaneous integration of stimulus and response features were analyzed. In line with previous findings, carrying out a prepared response to the onset of a stimulus created bindings between the response and the features of that stimulus, thereby impairing subsequent performance on mismatching stimulus-response combinations. Findings demonstrate that a stimulus gets bound to a response even if its presence is neither necessary nor useful for the task at hand, it follows rather than precedes the response in time, it competes with a task-relevant stimulus, and if the response is suppressed--but only if the stimulus appears close to the response's eventual execution or abandonment. A multiple-integration model is suggested that assumes that the integration of stimulus features in perception and of response features in action planning are local processes that are independent of stimulus-response integration, which presumably is triggered by the success of the perception-action episode.     

Sequential analysis of a Simon task--evidence for an attention-shift account

 We investigated the attention-shift hypothesis of the Simon effect by analysing the effect of repeating relevant colour or irrelevant location of the stimulus in four serial reaction time tasks. In Experiment 1 with short response-stimulus intervals (RSI), we assume that there is no time to engage attention at the fixation cross before the onset of a new stimulus. In agreement with the hypothesis, Experiment 1 reveals no Simon effect when the stimulus location is repeated. In Experiment 2 with long RSI, we observe a Simon effect for location repetitions and alternations. In Experiment 3 with long RSI, we hinder the disengagement of attention by displaying the stimulus after response execution. As expected, the Simon effect is reduced for location repetitions. In Experiment 4 with stimuli additionally presented at the fixation cross, responses are faster if the attention shift towards the centrally presented stimulus corresponds with the location of the required response. Additionally, we argue that binding of the stimulus features into an object or event file better explains the so-called blocking of the automatic response-priming route after a noncorresponding trial. Received: 2 February 2000 / Accepted: 10 November 2000     

Transcranial magnetic stimulation over MT/MST fails to impair judgments of implied motion

The medial temporal and medial superior temporal cortex (MT/MST) is involved in the processing of visual motion, and fMRI experiments indicate that there is greater activation when subjects view static images that imply motion than when they view images that do not imply motion at all. We applied transcranial magnetic stimulation (TMS) to MT/MST in order to assess the functional necessity of this region for the processing of implied motion represented in static images. Area MT/MST was localized by the use of a TMS-induced misperception of visual motion, and its location was verified through the monitored completion of a motion discrimination task. We controlled for possible impairments in general visual processing by having subjects perform an object categorization task with and without TMS. Although MT/MST stimulation impaired performance in a motion discrimination task (and vertex stimulation did not), there was no difference in performance between the two forms of stimulation in the implied motion discrimination task. MT/MST stimulation did, however, improve subjects’ performance in the object categorization task. These results indicate that, within 150 msec of stimulus presentation, MT/MST is not directly involved in the visual processing of static images in which motion is implied. The results do, however, confirm previous findings that disruption of MT/MST may improve efficiency in more ventral visual processing streams.     

An action sequence withheld in memory can delay execution of visually guided actions: the generalization of response compatibility interference

Withholding an action plan in memory for later execution can delay execution of another action, if the actions share a similar (compatible) action feature (i.e., response hand). This phenomenon, termed compatibility interference (CI), was found for identity-based actions that do not require visual guidance. The authors examined whether CI can generalize to both identity-based and location-based actions that require visual guidance. Participants withheld a planned action based on the identity of a stimulus and then immediately executed a visually guided action (touch response) to a 2nd stimulus based on its color identity (Experiment 1), its spatial location (Experiment 2), or an intrinsic spatial location within an object (Experiment 3). Results showed CI for both left- and right-hand responses in Experiment 1. However, CI occurred for left- but not right-hand responses in Experiment 2 and 3. This suggests that CI can generalize to visually guided actions under cognitive control but not to actions that invoke automatic visual-control mechanisms where the left hemisphere may play a special role (C. Gonzalez, T. Ganel, & M. Goodale, 2006). The code occupation account for CI (G. Stoet & B. Hommel, 2002) is also discussed.     

Priming and binding in and across perception and action: A correlational analysis of the internal structure of event files

Individual performance was compared across three different tasks that tap into the binding of stimulus features in perception, the binding of action features in action planning, and the emergence of stimulus–response bindings (“event files”). Within a task correlations between the size of binding effects were found within visual perception (e.g., the strength of shape–location binding correlated positively with the strength of shape–colour binding) but not between perception and action planning, suggesting different, domain-specific binding mechanisms. To some degree, binding strength was predicted by priming effects of the respective features, especially if these features varied on a dimension that matched the current attentional set.     

A Simon effect in memory retrieval: Evidence for the response-discrimination account

According to the traditional view, the effects of irrelevant stimulus location on the selection of a spatial response to a nonspatial stimulus feature (Simon effect) result from long-term associations between spatial stimulus codes and spatially corresponding response codes. According to an alternative view, the response-discrimination account, Simon effects arise from interactions between spatial stimulus codes and response labels in working memory (WM). The latter account predicts Simon effects when participants use spatial labels for response representation in WM, even when the actual responses have no spatial features (e.g., saying the word "plate"). The prediction was tested in an experiment, in which participants first encoded two words at different locations, and then responded to a stimulus by saying the word from the location indicated by stimulus color. The manipulation concerned the correspondence between irrelevant location of the colored stimulus and the retrieval cue for the vocal responses (i.e., word location in the encoding display). A Simon effect in memory retrieval was observed, supporting the response-discrimination account.     

Action history influences eye movements

Recent research has revealed that simple actions can have a profound effect on subsequent perception – people are faster to find a target that shares features with a previously acted on object even when those features are irrelevant to their task (the action effect). However, the majority of the evidence for this interaction between action and perception has come from manual response data. Therefore, it is unknown whether action affects early visual search processes, if it modulates post-attentional-selection processes, or both. To investigate this, we tracked participants’ spontaneous eye movements as they performed an action effect task. In two experiments we found that participants looked more quickly to the colour of an object they had previously acted on, compared to if they had viewed but not acted on the object, showing that action influenced early visual search processes. Additionally, there was evidence for post-selection effects as well. The results suggest that prior action affects both pre-selection and post-selection processes – spontaneously guiding attention to, and maintaining it on, objects that were previously important to the observer.     

The microgenesis of action-effect binding

Ideomotor theories of human action control assume that performing a movement leads to the automatic integration of the underlying motor pattern with codes of its perceptual consequences. We studied the microgenesis of action-effect integration by varying the mapping of action effects upon actions from trial to trial. Experiments 1 and 2 showed that perceiving a tone repetition systematically affects one’s tendency to carry out the response that produced that tone in the previous trial, suggesting that even the unintentional production of a stimulus creates a temporary binding of that stimulus with the action that brought it about. Experiments 3 and 4 extended this finding in suggesting that the integration and/or retrieval of action effects is modulated by attentional factors: Ongoing performance is more impacted by action effects if they are salient or match the current attentional set.

---

Attention,instruction, and response representation

A two-dimensional Simon-type task was devised to investigate the impact of task requirements and explicit instructions on spatial action coding. Subjects performed actions that were defined on two spatial dimensions: horizontal (left-right) or vertical (top-bottom). The relevant stimulus feature was nonspatial but the stimuli varied on the horizontal and the vertical dimension, so that horizontal and vertical S-R compatibility effects could be measured separately. Implicit task requirements were manipulated by having the subjects perform an unrelated task before the Simon task—a task in which only one of the two spatial dimensions was relevant. Instructions were varied by describing the responses in the unrelated priming task and/or in the Simon task in spatial terms or by referring to nonspatial features of the response keys. Priming a particular dimension increased the Simon effect on that dimension, whereas instructions had no differential effect. These findings suggest that, first, drawing attention to a particular dimension leads to a stronger contribution to event representation of those features defined on that dimension (intentional weighting) and, second, that instructions do not affect action coding if the manipulation does not change the task goal.     

Flexible working memory representation of the relationship between an object and its location as revealed by interactions with attention

Working memory (WM) selectively maintains a limited amount of currently relevant information in an active state to influence future perceptual processing, thought, and behavior. The representation of the information held in WM is unknown, particularly the degree of separation between the representation of an object’s identity and its location. The present experiments examined the flexibility of object and location WM representations by measuring reaction times on a visual discrimination task during the delay period of a WM recognition task for object identities, locations, or both. The results demonstrate that during WM delay periods, attention is biased toward information that matches the current contents of WM. Attention is not biased toward information that was present in the encoded memory sample but not relevant for the memory recognition test. This specificity of the interaction between WM and attention applies to both the identity and the location of the remembered stimulus and to the relationship between an object and its location. The results suggest that when this relationship is necessary for task performance, WM represents an object and its identity in an integrated manner. However, if this relationship is not task relevant, the object and location information are represented in WM separately.     

2D location biases depth-from-disparity judgments but not vice versa

Visual cognition in our 3D world requires understanding how we accurately localize objects in 2D and depth, and what influence both types of location information have on visual processing. Spatial location is known to play a special role in visual processing, but most of these findings have focused on the special role of 2D location. One such phenomena is the spatial congruency bias, where 2D location biases judgments of object features but features do not bias location judgments. This paradigm has recently been used to compare different types of location information in terms of how much they bias different types of features. Here we used this paradigm to ask a related question: whether 2D and depth-from-disparity location bias localization judgments for each other. We found that presenting two objects in the same 2D location biased position-in-depth judgments, but presenting two objects at the same depth (disparity) did not bias 2D location judgments. We conclude that an object’s 2D location may be automatically incorporated into perception of its depth location, but not vice versa, which is consistent with a fundamentally special role for 2D location in visual processing.     

S-R compatibility effects due to context-dependent spatial stimulus coding

Responses are faster with spatial S-R correspondence than with noncorrespondence (spatial compatibility effect), even if stimulus location is irrelevant (Simon effect). In two experiments, we sought to determine whether stimuli located above and below a fixation point are coded as left and right (and thus affect the selection of left and right responses) if the visual context suggests such a coding. So, stimuli appeared on the left or right eye of a face’s image that was tilted by 90° to one side or the other (Experiment 1) or varied between upright and 45° or 90° tilting (Experiment 2). Whether stimulus location was relevant (Experiment 1) or not (Experiment 2), responses were faster with correspondence of (face-based) stimulus location and (egocentrically defined) response location, even if stimulus and response locations varied on physically orthogonal dimensions. This suggests that object-based spatial stimulus codes are formed automatically and thus influence the speed of response selection.     

Vision-for-action: The effects of object property discrimination and action state on affordance compatibility effects

When a person views an object, the action the object evokes appears to be activated independently of the person’s intention to act. We demonstrate two further properties of this vision-to-action process. First, it is not completely automatic, but is determined by the stimulus properties of the object that are attended. Thus, when a person discriminates the shape of an object, action affordance effects are observed; but when a person discriminates an object’s color, no affordance effects are observed. The former, shape property is associated with action, such as how an object might be grasped; the latter, color property is irrelevant to action. Second, we also show that the action state of an object influences evoked action. Thus, active objects, with which current action is implied, produce larger affordance effects than passive objects, with which no action is implied. We suggest that the active object activates action simulation processes similar to those proposed in mirror systems.     

The privileged role of location in visual working memory

Reports have conflicted about the possible special role of location in visual working memory (WM). One important question is: Do we maintain the locations of objects in WM even when they are irrelevant to the task at hand? Here we used a continuous response scale to study the types of reporting errors that participants make when objects are presented at the same or at different locations in space. When several objects successively shared the same location, participants exhibited a higher tendency to report features of the wrong object in memory; that is, they responded with features that belonged to objects retained in memory but not probed at retrieval. On the other hand, a similar effect was not observed when objects shared a nonspatial feature, such as color. Furthermore, the effect of location on reporting errors was present even when its manipulation was orthogonal to the task at hand. These findings are consistent with the view that binding together different nonspatial features of an object in memory might be mediated through an object’s location. Hence, spatial location may have a privileged role in WM. The relevance of these findings to conceptual models, as well as to neural accounts of visual WM, is discussed.