Action–perception coupling in pianists: Learned mappings or spatial musical association of response codes (SMARC) effect?

The principle of common coding suggests that a joint representation is formed when actions are repeatedly paired with a specific perceptual event. Musicians are occupationally specialized with regard to the coupling between actions and their auditory effects. In the present study, we employed a novel paradigm to demonstrate automatic action–effect associations in pianists. Pianists and nonmusicians pressed keys according to aurally presented number sequences. Numbers were presented at pitches that were neutral, congruent, or incongruent with respect to pitches that would normally be produced by such actions. Response time differences were seen between congruent and incongruent sequences in pianists alone. A second experiment was conducted to determine whether these effects could be attributed to the existence of previously documented spatial/pitch compatibility effects. In a “stretched” version of the task, the pitch distance over which the numbers were presented was enlarged to a range that could not be produced by the hand span used in Experiment 1. The finding of a larger response time difference between congruent and incongruent trials in the original, standard, version compared with the stretched version, in pianists, but not in nonmusicians, indicates that the effects obtained are, at least partially, attributable to learned action effects.     

Influence of auditory and audiovisual stimuli on the right–left prevalence effect

When auditory stimuli are used in two-dimensional spatial compatibility tasks, where the stimulus and response configurations vary along the horizontal and vertical dimensions simultaneously, a right–left prevalence effect occurs in which horizontal compatibility dominates over vertical compatibility. The right–left prevalence effects obtained with auditory stimuli are typically larger than that obtained with visual stimuli even though less attention should be demanded from the horizontal dimension in auditory processing. In the present study, we examined whether auditory or visual dominance occurs when the two-dimensional stimuli are audiovisual, as well as whether there will be cross-modal facilitation of response selection for the horizontal and vertical dimensions. We also examined whether there is an additional benefit of adding a pitch dimension to the auditory stimulus to facilitate vertical coding through use of the spatial-musical association of response codes (SMARC) effect, where pitch is coded in terms of height in space. In Experiment 1, we found a larger right–left prevalence effect for unimodal auditory than visual stimuli. Neutral, non-pitch coded, audiovisual stimuli did not result in cross-modal facilitation, but did show evidence of visual dominance. The right–left prevalence effect was eliminated in the presence of SMARC audiovisual stimuli, but the effect influenced horizontal rather than vertical coding. Experiment 2 showed that the influence of the pitch dimension was not in terms of influencing response selection on a trial-to-trial basis, but in terms of altering the salience of the task environment. Taken together, these findings indicate that in the absence of salient vertical cues, auditory and audiovisual stimuli tend to be coded along the horizontal dimension and vision tends to dominate audition in this two-dimensional spatial stimulus–response task.     

空间-时间关联的中介共同表征结构:来自反转STEARC效应的证据

通过3个双任务实验(诱导任务和特征任务)探讨空间-时间联合编码(STEARC)效应的加工机制。实验1采用时间信息作为诱导任务材料,实验2采用空间信息作为诱导任务材料,在特征任务中都发现映射不一致组被试(看到过去/左侧刺激时按右键反应,看到未来/右侧刺激时按左键反应)出现反转STEARC效应,映射一致组被试表现出常规的STEARC效应,表明从时间信息加工到空间反应过程符合中介共同表征结构。实验3分离两种任务的反应方式(手动和眼动),发现不一致映射规则下,被试仍然表现出常规的STEARC效应,表明这种中介共同表征结构存在特定联结效应,即在不同反应器中出现时间和空间相互独立的表征结构。总体而言,研究支持空间-时间关联符合中介共同表征结构,并且这种关联中存在反应器特定联结效应。     

Referential coding contributes to the horizontal SMARC effect

The present study tested whether coding of tone pitch relative to a referent contributes to the correspondence effect between the pitch height of an auditory stimulus and the location of a lateralized response. When left-right responses are mapped to high or low pitch tones, performance is better with the high-right/low-left mapping than with the opposite mapping, a phenomenon called the horizontal SMARC effect. However, when pitch height is task irrelevant, the horizontal SMARC effect occurs only for musicians. In Experiment 1, nonmusicians performed a pitch discrimination task, and the SMARC effect was evident regardless of whether a referent tone was presented. However, in Experiment 2, for a timbre-judgment task, nonmusicians showed a SMARC effect only when a referent tone was presented, whereas musicians showed a SMARC effect that did not interact with presence/absence of the referent. Dependence of the SMARC effect for nonmusicians on a reference tone was replicated in Experiment 3, in which judgments of the color of a visual stimulus were made in the presence of a concurrent high- or low-pitched pure tone. These results suggest that referential coding of pitch height is a key determinant for the horizontal SMARC effect when pitch height is irrelevant to the task.     

Does crossmodal correspondence modulate the facilitatory effect of auditory cues on visual search?

The ??pip-and-pop effect?? refers to the facilitation of search for a visual target (a horizontal or vertical bar whose color changes frequently) among multiple visual distractors (tilted bars also changing color unpredictably) by the presentation of a spatially uninformative auditory cue synchronized with the color change of the visual target. In the present study, the visual stimuli in the search display changed brightness instead of color, and the crossmodal congruency between the pitch of the auditory cue and the brightness of the visual target was manipulated. When cue presence and cue congruency were randomly varied between trials (Experiment 1), both congruent cues (low-frequency tones synchronized with dark target states or high-frequency tones synchronized with bright target states) and incongruent cues (the reversed mapping) facilitated visual search performance equally, relative to a no-cue baseline condition. However, when cue congruency was blocked and the participants were informed about the pitch?Cbrightness mapping in the cue-present blocks (Experiment 2), performance was significantly enhanced when the cue and target were crossmodally congruent as compared to when they were incongruent. These results therefore suggest that the crossmodal congruency between auditory pitch and visual brightness can influence performance in the pip-and-pop task by means of top-down facilitation.     

Generalizing attentional control across dimensions and tasks: Evidence from transfer of proportion-congruent effects

Three experiments investigated transfer of list-wide proportion congruent (LWPC) effects from a set of congruent and incongruent items with different frequency (inducer task) to a set of congruent and incongruent items with equal frequency (diagnostic task). Experiments 1 and 2 mixed items from horizontal and vertical Simon tasks. Tasks always involved different stimuli that varied on the same dimension (colour) in Experiment 1 and on different dimensions (colour, shape) in Experiment 2. Experiment 3 mixed trials from a manual Simon task with trials from a vocal Stroop task, with colour being the relevant stimulus in both tasks. There were two major results. First, we observed transfer of LWPC effects in Experiments 1 and 3, when tasks shared the relevant dimension, but not in Experiment 2. Second, sequential modulations of congruency effects transferred in Experiment 1 only. Hence, the different transfer patterns suggest that LWPC effects and sequential modulations arise from different mechanisms. Moreover, the observation of transfer supports an account of LWPC effects in terms of list-wide cognitive control, while being at odds with accounts in terms of stimulus–response (contingency) learning and item-specific control.     

Spatial stimulus-response compatibility and negative priming

According to Kornblum's (1992) dimensional overlap model, when an incongruent response to a stimulus is required, automatic activation of the congruent response must first be inhibited. Shiu and Kornblum (1996a) provided evidence for such inhibition in an incongruent symbolic negative priming task. Reaction time was longer when a trial's correct response was the name of the stimulus from the previous trial than when it was not. We report three experiments that test this inhibition hypothesis for spatial stimuli and responses. In Experiment 1, which used a spatial mapping analogous to the symbolic mapping used by Shiu and Kornblum (1996a), a similar negative priming effect was found. However, in Experiments 2 and 3, which used mappings that were conducive to simple transformational rules, a positive priming effect was obtained. The results suggest that inhibition in response selection may depend on the complexity of the relations between the stimuli and responses.     

视听刺激突显性对视觉主导效应的影响

本研究使用空间任务-转换范式,控制视、听刺激的突显性,探讨自下而上注意对视觉主导效应的影响。结果表明视、听刺激突显性显著地影响视觉主导效应,实验一中当听觉刺激为高突显性时,视觉主导效应显著减弱。实验二中当听觉刺激为高突显性并且视觉刺激为低突显性时,视觉主导效应进一步减弱但依然存在。结果支持偏向竞争理论,在跨通道视听交互过程中视觉刺激更具突显性,在多感觉整合过程中更具加工优势。     

Visual and auditory accessory stimulus offset and the Simon effect

We investigated the effect on the right and left responses of the disappearance of a task-irrelevant stimulus located on the right or left side. Participants pressed a right or left response key on the basis of the color of a centrally located visual target. Visual (Experiment 1) or auditory (Experiment 2) task-irrelevant accessory stimuli appeared or disappeared at locations to the right or left of the central target. In Experiment 1, responses were faster when onset or offset of the visual accessory stimulus was spatially congruent with the response. In Experiment 2, responses were again faster when onset of the auditory accessory stimulus and the response were on the same side. However, responses were slightly slower when offset of the auditory accessory stimulus and the response were on the same side than when they were on opposite sides. These findings indicate that transient change information is crucial for a visual Simon effect, whereas sustained stimulation from an ongoing stimulus also contributes to an auditory Simon effect.     

Spatial representation of pitch height: the SMARC effect

Through the preferential pairing of response positions to pitch, here we show that the internal representation of pitch height is spatial in nature and affects performance, especially in musically trained participants, when response alternatives are either vertically or horizontally aligned. The finding that our cognitive system maps pitch height onto an internal representation of space, which in turn affects motor performance even when this perceptual attribute is irrelevant to the task, extends previous studies on auditory perception and suggests an interesting analogy between music perception and mathematical cognition. Both the basic elements of mathematical cognition (i.e. numbers) and the basic elements of musical cognition (i.e. pitches), appear to be mapped onto a mental spatial representation in a way that affects motor performance.     

Item-specific congruency effects in nonverbal auditory Stroop

In the current study, participants judged as 'low' or 'high' either the location or the frequency of a single tone presented in one of two locations at one of two frequencies. The classification associated with the irrelevant feature could be either congruent or incongruent with the required response. The results of Experiment 1 revealed that responses were made more slowly on incongruent than on congruent trials, regardless of whether participants judged sounds according to their location or their pitch. In Experiment 2, we manipulated the probability that the classification associated with the irrelevant acoustic feature was consistent with the classification associated with the task-relevant dimension. In this experiment responses were made more quickly on congruent trials when the response associated with the irrelevant feature was likely to be consistent with the required response, and on incongruent trials when the response associated with the irrelevant feature was likely to be inconsistent with the required response.     

面孔表情和声音情绪信息整合对返回抑制的影响

返回抑制(inhibition of return, IOR)与情绪刺激都具有引导注意偏向、提高搜索效率的特点, 但二者间是否存在一定的交互作用迄今为止尚不明确。研究采用“线索-目标”范式并在视听双通道呈现情绪刺激来考察情绪刺激的加工与IOR的交互作用。实验1中情绪刺激以单通道视觉面孔或一致的视听双通道呈现, 实验2通过在视听通道呈现不一致的情绪刺激进一步考察视听双通道情绪一致刺激对IOR的影响是否是由听觉通道一致的情绪刺激导致的, 即是否对听觉通道的情绪刺激进行了加工。结果发现, 视听双通道情绪一致刺激能够削弱IOR, 但情绪不一致刺激与IOR之间不存在交互作用, 并且单双通道的IOR不存在显著差异。研究结果表明仅在视听双通道呈现情绪一致刺激时, 才会影响同一阶段的IOR, 这进一步支持了IOR的知觉抑制理论。     

Response inhibition under task switching: its strength depends on the amount of task-irrelevant response activation

Under task switch conditions, response repetitions usually produce benefits if the task also repeats, but costs if the task switches. So far, it is largely undecided how to account for these effects. In the present study, we provide additional evidence in favor of the account that each response is inhibited in order to prevent its accidental re-execution. To test this hypothesis, the risk of an accidental re-execution of a given response was manipulated by modulating the activation of the response in the previous task. In Experiment 1, this was done by means of congruent and incongruent stimuli. As expected, on task switch trials, the repetition costs were larger if a congruent rather than an incongruent stimulus occurred in the previous task. In Experiment 2, the same effect occurred for stimulus-response compatible versus incompatible stimuli in the previous task. In Experiment 3, both manipulations were applied together, which produced almost additive effects. Altogether, the results support the inhibition account for the response repetition effects under task switch conditions.     

The Spatial Musical Association of Response Codes does not depend on a normal visual experience: A study with early blind individuals

Converging evidence suggests that the perception of auditory pitch exhibits a characteristic spatial organization. This pitch–space association can be demonstrated experimentally by the Spatial Musical Association of Response Codes (SMARC) effect. This is characterized by faster response times when a low-positioned key is pressed in response to a low-pitched tone, and a high-positioned key is pressed in response to a high-pitched tone. To investigate whether the development of this pitch–space association is mediated by normal visual experience, we tested a group of early blind individuals on a task that required them to discriminate the timbre of different instrument sounds with varying pitch. Results revealed a comparable pattern in the SMARC effect in both blind participants and sighted controls, suggesting that the lack of prior visual experience does not prevent the development of an association between pitch height and vertical space.     

On the interplay of visuospatial and audiotemporal dominance: Evidence from a multimodal kappa effect

When participants judge multimodal audiovisual stimuli, the auditory information strongly dominates temporal judgments, whereas the visual information dominates spatial judgments. However, temporal judgments are not independent of spatial features. For example, in the kappa effect, the time interval between two marker stimuli appears longer when they originate from spatially distant sources rather than from the same source. We investigated the kappa effect for auditory markers presented with accompanying irrelevant visual stimuli. The spatial sources of the markers were varied such that they were either congruent or incongruent across modalities. In two experiments, we demonstrated that the spatial layout of the visual stimuli affected perceived auditory interval duration. This effect occurred although the visual stimuli were designated to be task-irrelevant for the duration reproduction task in Experiment 1, and even when the visual stimuli did not contain sufficient temporal information to perform a two-interval comparison task in Experiment 2. We conclude that the visual and auditory marker stimuli were integrated into a combined multisensory percept containing temporal as well as task-irrelevant spatial aspects of the stimulation. Through this multisensory integration process, visuospatial information affected even temporal judgments, which are typically dominated by the auditory modality.     

What happens to an individual visual working memory representation when it is interrupted?

This study tested the hypothesis that even the simplest cognitive tasks require the storage of information in working memory (WM), distorting any information that was previously stored in WM. Experiment 1 tested this hypothesis by requiring observers to perform a simple letter discrimination task while they were holding a single orientation in WM. We predicted that performing the task on the interposed letter stimulus would cause the orientation memory to become less precise and more categorical compared to when the letter was absent or when it was present but could be ignored. This prediction was confirmed. Experiment 2 tested the modality specificity of this effect by replacing the visual letter discrimination task with an auditory pitch discrimination task. Unlike the interposed visual stimulus, the interposed auditory stimulus produced little or no disruption of WM, consistent with the use of modality‐specific representations. Thus, performing a simple visual discrimination task, but not a simple auditory discrimination task, distorts information about a single feature being maintained in visual WM. We suggest that the interposed task eliminates information stored within the focus of attention, leaving behind a WM representation outside the focus of attention that is relatively imprecise and categorical.     

Neural correlates of partial transmission of sensorimotor information in the cerebral cortex

Using single neuron recordings in monkey primary motor (MI) cortex, two series of experiments were conducted in order to know whether response preparation can begin before perceptual processing finishes, thus providing evidence for a temporal overlap of perceptual and motor processes.

In Experiment 1, a “left/right, Go/No-Go” reaction time (RT) task was used. One monkey was trained to perform wrist flexion/extension movements to align a pointer with visual targets. The visual display was organized to provide a two-dimensional stimulus: side (an easy discrimination between left and right targets) which determined movement direction, and distance (a difficult discrimination between distal and proximal targets) which determined whether or not the movement was to be made. Changes in neuronal activity, when they were time-locked to the stimulus, were almost similar in the Go and No-Go trials, and when they were time-locked to movement onset, were markedly reduced in No-Go as compared to Go trials.

In Experiment 2, a stimulus-response compatibility (SRC) task was used. Two monkeys were trained to align a pointer with visual targets, on either left or right. In the spatially “compatible” trials, they had to point at the stimulus position, whereas in the “incompatible” trials, they had to point at the target located in the opposite side. For 12.5% of neurons, changes in activity associated with incompatible trials looked like changes in activity associated with movements performed in the opposite direction during compatible trials, thus suggesting the hypothesis of an automatic activation of the congruent, but incorrect response.

Results of both experiments provide evidence for a partial transmission of information from visual to motor cortical areas: that is, in the No-Go trials of the first task, information about movement direction, before the decision to perform or not this movement was made, and, in the incompatible trials of the SRC task, information about the congruent, but incorrect response, before the incongruent, but correct response was programmed.     

Processing irrelevant location information: Practice and transfer effects in choice-reaction tasks

Left or right keypresses to a relevant stimulus dimension are faster when the stimulus location, although irrelevant, corresponds with that of the response than when it does not. This phenomenon, called the Simon effect, persisted across 1,800 trials of practice, although its magnitude was reduced. Practice with the relevant stimulus dimension presented at a centered location had little influence on the magnitude of the Simon effect when irrelevant location was varied subsequently, and practice with location irrelevant prior to performing with location relevant slowed responses. After practice responding to stimulus location with an incompatible spatial mapping, the Simon effect was reversed (i.e., responses were slower when stimulus location corresponded with response location) when location was made irrelevant. When the response keys were labeled according to the relevant stimulus dimension (the Hedge and Marsh [1975] task variation), this reversal from practice with a spatially incompatible mapping was found for both the congruent and the incongruent relevant stimulus-response mappings. Thus, task-defined associations between stimulus location and response location affect performance when location is changed from relevant to irrelevant, apparently through producing automatic activation of the previously associated response.     

Action effects in saccade control

According to the ideomotor principle, action preparation involves the activation of associations between actions and their effects. However, there is only sparse research on the role of action effects in saccade control. Here, participants responded to lateralized auditory stimuli with spatially compatible saccades toward peripheral targets (e.g., a rhombus in the left hemifield and a square in the right hemifield). Prior to the imperative auditory stimulus (e.g., a left tone), an irrelevant central visual stimulus was presented that was congruent (e.g., a rhombus), incongruent (e.g., a square), or unrelated (e.g., a circle) to the peripheral saccade target (i.e., the visual effect of the saccade). Saccade targets were present throughout a trial (Experiment 1) or appeared after saccade initiation (Experiment 2). Results showed shorter response times and fewer errors in congruent (vs. incongruent) conditions, suggesting that associations between oculomotor actions and their visual effects play an important role in saccade control.     

Flanker and Simon effects interact at the response selection stage

The present study aimed at investigating the processing stage underlying stimulus–stimulus (S–S) congruency effects by examining the relation of a particular type of congruency effect (i.e., the flanker effect) with a stimulus–response (S–R) spatial correspondence effect (i.e., the Simon effect). Experiment 1 used a unilateral flanker task in which the flanker also acted as a Simon-like accessory stimulus. Results showed a significant S–S Congruency × S–R Correspondence interaction: An advantage for flanker–response spatially corresponding trials was observed in target–flanker congruent conditions, whereas, in incongruent conditions, there was a noncorresponding trials' advantage. The analysis of the temporal trend of the correspondence effects ruled out a temporal-overlap account for the observed interaction. Moreover, results of Experiment 2, in which the flanker did not belong to the target set, demonstrated that this interaction cannot be attributed to perceptual grouping of the target–flanker pairs and referential coding of the target with respect to the flanker in the congruent and incongruent conditions, respectively. Taken together, these findings are consistent with a response selection account of congruency effects: Both the position and the task-related attribute of the flanker would activate the associated responses. In noncorresponding-congruent trials and corresponding-incongruent trials, this would cause a conflict at the response selection stage.