Is it for real? Evaluating authenticity of musical pitch-space synesthesia

In spatial-sequence synesthesia, ordinal sequences are visualized in explicit spatial locations. We examined a recently documented subtype in which musical notes are represented in spatial configurations, to verify consistency and automaticity of musical pitch-space (M-S) synesthesia. An M-S synesthete performed a mapping pre-task (Exp. 1) used to indicate the locations of 7 auditory or visually presented notes, in 2 sessions a month apart. Results revealed strong correlations between sessions, suggesting the synesthete's musical forms were consistent over time. Experiment 2 assessed automaticity of M-S synesthesia. The same synesthete and matched controls preformed a spatial Stroop-like task. Participants were presented with an auditory or visual musical note and then had to reach for an asterisk (target) with a mouse cursor. The target appeared in a compatible or incompatible location (according to the synesthete's spatial representation). A compatibility effect was found for the synesthete but not for controls. The synesthete was significantly faster when the target appeared in compatible locations than in incompatible ones. Our findings show that for synesthetes, auditory and visually presented notes automatically trigger attention to specific spatial locations according to their specific M-S associations.     

Non-musicians also have a piano in the head: evidence for spatial–musical associations from line bisection tracking

The spatial representation of ordinal sequences (numbers, time, tones) seems to be a fundamental cognitive property. While an automatic association between horizontal space and pitch height (left-low pitch, right-high pitch) is constantly reported in musicians, the evidence for such an association in non-musicians is mixed. In this study, 20 non-musicians performed a line bisection task while listening to irrelevant high- and low-pitched tones and white noise (control condition). While pitch height had no influence on the final bisection point, participants’ movement trajectories showed systematic biases: When approaching the line and touching the line for the first time (initial bisection point), the mouse cursor was directed more rightward for high-pitched tones compared to low-pitched tones and noise. These results show that non-musicians also have a subtle but nevertheless automatic association between pitch height and the horizontal space. This suggests that spatial–musical associations do not necessarily depend on constant sensorimotor experiences (as it is the case for musicians) but rather reflect the seemingly inescapable tendency to represent ordinal information on a horizontal line.     

Temporal sequences,synesthetic mappings,and cultural biases: The geography of time

Time–space synesthetes report that they experience the months of the year as having a spatial layout. In Study 1, we characterize the phenomenology of calendar sequences produced by synesthetes and non-synesthetes, and show a conservative estimate of time–space synesthesia at 2.2% of the population. We demonstrate that synesthetes most commonly experience the months in a circular path, while non-synesthetes default to linear rows or rectangles. Study 2 compared synesthetes’ and non-synesthetes’ ability to memorize a novel spatial calendar, and revealed better performance in synesthetes. The capacity to learn mappings between arbitrary spatial forms and temporal sequences is present in all individuals, and time–space synesthetes’ enhanced visuo-spatial memory abilities may underlie their creation of idiosyncratic spatial calendar forms.     

Superior encoding enhances recall in color-graphemic synesthesia

Synesthesia is a phenomenon in which particular stimuli, such as letters or sound, generate a secondary sensory experience in particular individuals. Reports of enhanced memory in synesthetes raise the question of its cognitive and neurological substrates. Enhanced memory in synesthetes could arise from the explicit or implicit use of a synesthetic cue to aid memory, from changes unique to the synesthete brain, or from both, depending on the task. To assess this question, we tested nine color-graphemic synesthetes using standardized neuropsychological measures that should not trigger color-graphemic synesthesia (visuo-spatial tests) and measures that should trigger color-graphemic synesthesia (verbal tasks). We found a synesthetic advantage on both types of tests, primarily in the initial encoding of information. The pattern of results adds to existing evidence of advantages in synesthetic memory, as well as provides novel evidence that synesthetes may have enhanced encoding rather than superior recall. Synesthetes learn more initially, rather than forgetting less over time.     

The representation of pitch in auditory imagery: Evidence from S-R compatibility and distance effects

In three experiments we explored the nature of representations constructed during the perception and imagination of pitch. We employed a same–different task to eliminate the influence of nonauditory information and to minimise use of cognitive strategies on auditory imagery. A reference tone of frequency 1000, 1500, or 2000 Hz, or an imagined tone of a pitch indicated by a visual cue, was followed by a comparison tone (1000, 1500, or 2000 Hz) to which either a speeded same or different response was required. In separate experiments, same–different judgements were mapped to vertically (Experiments 1 and 2) and horizontally arranged responses (Experiment 3). Judgements of tones closer in pitch yielded longer reaction times and higher error rates than more distant tones, indicating a pitch distance effect for perceptual and imagery tasks alike. In addition, in the imagery task, same–different responses were faster when low-pitched tones demanded a bottom or left key response and high-pitched tones a top or right response than vice versa, suggesting that pitch is coded spatially. Together, these behavioural effects support the assumption that both perceived and imagined pitch are translated into an analogical representation in the spatial domain.     

Spatial cueing in time–space synesthetes: An event-related brain potential study

Some people report that they consistently and involuntarily associate time events, such as months of the year, with specific spatial locations; a condition referred to as time–space synesthesia. The present study investigated the manner in which such synesthetic time–space associations affect visuo-spatial attention via an endogenous cuing paradigm. Reaction times and ERPs were recorded as 12 time–space synesthetes and 12 control participants did a peripheral target detection task, cued by three different types of centrally presented cues: arrows pointing left or right, direction words “left” or “right”, and month names associated with either the left or the right side of the synesthete’s mental calendar (e.g., “October” or “May”). Cues were followed by probes on the left or right side of the screen, and participants responded to the probes with button presses. Behavioral and ERP data suggested that for synesthetes, month words functioned more effectively as cues to direct attention in space. In synesthetes but not controls, a comparison of ERPs to probes cued by months revealed effects of cue validity on the P3b component peaking 370 ms post-onset and on the subsequent positive slow wave (pSW) observed 600–900 ms post-onset (both larger for invalid probes). No effects of cue validity were observed on early visual potentials (N1) for probes cued by months. The findings suggest that in these time–space synesthetes cue validity influenced post-perceptual processes, such as stimulus evaluation and categorization, with no evidence for enhanced visual processing.     

Do you see what I’m singing? Visuospatial movement biases pitch perception

The nature of the connection between musical and spatial processing is controversial. While pitch may be described in spatial terms such as “high” or “low”, it is unclear whether pitch and space are associated but separate dimensions or whether they share representational and processing resources. In the present study, we asked participants to judge whether a target vocal note was the same as (or different from) a preceding cue note. Importantly, target trials were presented as video clips where a singer sometimes gestured upward or downward while singing that target note, thus providing an alternative, concurrent source of spatial information. Our results show that pitch discrimination was significantly biased by the spatial movement in gesture, such that downward gestures made notes seem lower in pitch than they really were, and upward gestures made notes seem higher in pitch. These effects were eliminated by spatial memory load but preserved under verbal memory load conditions. Together, our findings suggest that pitch and space have a shared representation such that the mental representation of pitch is audiospatial in nature.     

Perceptual interactions between musical pitch and timbre.

These experiments examined perceptual interactions between musical pitch and timbre. Experiment 1, through the use of the Garner classification tasks, found that pitch and timbre of isolated tones interact. Classification times showed interference from uncorrelated variation in the irrelevant attribute and facilitation from correlated variation; the effects were symmetrical. Experiments 2 and 3 examined how musical pitch and timbre function in longer sequences. In recognition memory tasks, a target tone always appeared in a fixed position in the sequences, and listeners were instructed to attend to either its pitch or its timbre. For successive tones, no interactions between timbre and pitch were found. That is, changing the pitches of context tones did not affect timbre recognition, and vice versa. The tendency to perceive pitch in relation to other context pitches was strong and unaffected by whether timbre was constant or varying. In contrast, the relative perception of timbre was weak and was found only when pitch was constant. These results suggest that timbre is perceived more in absolute than in relative terms. Perceptual implications for creating patterns in music with timbre variations are discussed.     

The effect of exogenous spatial attention on auditory information processing

This study investigated the effect of exogenous spatial attention on auditory information processing. In Experiments 1, 2 and 3, temporal order judgment tasks were performed to examine the effect. In Experiment 1 and 2, a cue tone was presented to either the left or right ear, followed by sequential presentation of two target tones. The subjects judged the order of presentation of the target tones. The results showed that subjects heard both tones simultaneously when the target tone, which was presented on the same side as the cue tone, was presented after the target tone on the opposite side. This indicates that spatial exogenous attention was aroused by the cue tone, and facilitated subsequent auditory information processing. Experiment 3 examined whether both cue position and frequency influence the resulting information processing. The same effect of spatial attention was observed, but the effect of attention to a certain frequency was only partially observed. In Experiment 4, a tone fusion judgment task was performed to examine whether the effect of spatial attention occurred in the initial stages of hearing. The result suggests that the effect occurred in the later stages of hearing.     

Spatial representation of pitch height: the SMARC effect

Through the preferential pairing of response positions to pitch, here we show that the internal representation of pitch height is spatial in nature and affects performance, especially in musically trained participants, when response alternatives are either vertically or horizontally aligned. The finding that our cognitive system maps pitch height onto an internal representation of space, which in turn affects motor performance even when this perceptual attribute is irrelevant to the task, extends previous studies on auditory perception and suggests an interesting analogy between music perception and mathematical cognition. Both the basic elements of mathematical cognition (i.e. numbers) and the basic elements of musical cognition (i.e. pitches), appear to be mapped onto a mental spatial representation in a way that affects motor performance.     

An investigation of spatial representation of pitch in individuals with congenital amusia

Spatial representation of pitch plays a central role in auditory processing. However, it is unknown whether impaired auditory processing is associated with impaired pitch–space mapping. Experiment 1 examined spatial representation of pitch in individuals with congenital amusia using a stimulus–response compatibility (SRC) task. For amusic and non-amusic participants, pitch classification was faster and more accurate when correct responses involved a physical action that was spatially congruent with the pitch height of the stimulus than when it was incongruent. However, this spatial representation of pitch was not as stable in amusic individuals, revealed by slower response times when compared with control individuals. One explanation is that the SRC effect in amusics reflects a linguistic association, requiring additional time to link pitch height and spatial location. To test this possibility, Experiment 2 employed a colour-classification task. Participants judged colour while ignoring a concurrent pitch by pressing one of two response keys positioned vertically to be congruent or incongruent with the pitch. The association between pitch and space was found in both groups, with comparable response times in the two groups, suggesting that amusic individuals are only slower to respond to tasks involving explicit judgments of pitch.     

Does crossmodal correspondence modulate the facilitatory effect of auditory cues on visual search?

The ??pip-and-pop effect?? refers to the facilitation of search for a visual target (a horizontal or vertical bar whose color changes frequently) among multiple visual distractors (tilted bars also changing color unpredictably) by the presentation of a spatially uninformative auditory cue synchronized with the color change of the visual target. In the present study, the visual stimuli in the search display changed brightness instead of color, and the crossmodal congruency between the pitch of the auditory cue and the brightness of the visual target was manipulated. When cue presence and cue congruency were randomly varied between trials (Experiment 1), both congruent cues (low-frequency tones synchronized with dark target states or high-frequency tones synchronized with bright target states) and incongruent cues (the reversed mapping) facilitated visual search performance equally, relative to a no-cue baseline condition. However, when cue congruency was blocked and the participants were informed about the pitch?Cbrightness mapping in the cue-present blocks (Experiment 2), performance was significantly enhanced when the cue and target were crossmodally congruent as compared to when they were incongruent. These results therefore suggest that the crossmodal congruency between auditory pitch and visual brightness can influence performance in the pip-and-pop task by means of top-down facilitation.     

Functional heterogeneity of inferior frontal gyrus is shaped by linguistic experience

A crosslinguistic, positron emission tomography (PET) study was conducted to determine the influence of linguistic experience on the perception of segmental (consonants and vowels) and suprasegmental (tones) information. Chinese and English subjects (10 per group) were presented binaurally with lists consisting of five Chinese monosyllabic morphemes (speech) or low-pass-filtered versions of the same stimuli (nonspeech). The first and last items were targeted for comparison; the time interval between target tones was filled with irrelevant distractor tones. A speeded-response, selective attention paradigm required subjects to make discrimination judgments of the target items while ignoring intervening distractor tones. PET scans were acquired for five tasks presented twice: one passive listening to pitch (nonspeech) and four active (speech = consonant, vowel, and tone; nonspeech = pitch). Significant regional changes in blood flow were identified from comparisons of group-averaged images of active tasks relative to passive listening. Chinese subjects show increased activity in left premotor cortex, pars opercularis, and pars triangularis across the four tasks. English subjects, on the other hand, show increased activity in left inferior frontal gyrus regions only in the vowel task and in right inferior frontal gyrus regions in the pitch task. Findings suggest that functional circuits engaged in speech perception depend on linguistic experience. All linguistic information signaled by prosodic cues engages left-hemisphere mechanisms. Storage and executive processes of working memory that are implicated in phonological processing are mediated in discrete regions of the left frontal lobe.     

Visual orienting in response to attentional cues: Spatial correspondence is critical,conscious awareness is not

Three experiments examined visual orienting in response to spatial precues. In Experiments 1 and 2, attentional effects of central letters were stimulus driven: Orienting was dependent on the spatial layout of the cue display. When there were no correspondences between spatial features of the cue display and target location, attentional effects were absent, despite a conscious intention to orient in response to the symbolic information carried by the cue letters. In Experiment 3 clear orienting effects were observed when target location corresponded with spatial features of the cue display, but the magnitude of these effects was unaffected by whether participants were aware or unaware of the cue–target relationship. These findings are consistent with the view that (1) spatial correspondences between cues and targets are a critical factor driving visual orienting in cueing paradigms, and (2) attentional effects of spatial precues are largely independent of participants’ conscious awareness of the cue–target relationship.     

Metacontrast masking is processed before grapheme–color synesthesia

We investigated the physiological mechanism of grapheme–color synesthesia using metacontrast masking. A metacontrast target is rendered invisible by a mask that is delayed by about 60 ms; the target and mask do not overlap in space or time. Little masking occurs, however, if the target and mask are simultaneous. This effect must be cortical, because it can be obtained dichoptically. To compare the data for synesthetes and controls, we developed a metacontrast design in which nonsynesthete controls showed weaker dichromatic masking (i.e., the target and mask were in different colors) than monochromatic masking. We accomplished this with an equiluminant target, mask, and background for each observer. If synesthetic color affected metacontrast, synesthetes should show monochromatic masking more similar to the weak dichromatic masking among controls, because synesthetes could add their synesthetic color to the monochromatic condition. The target–mask pairs used for each synesthete were graphemes that elicited strong synesthetic colors. We found stronger monochromatic than dichromatic U-shaped metacontrast for both synesthetes and controls, with optimal masking at an asynchrony of 66 ms. The difference in performance between the monochromatic and dichromatic conditions in the synesthetes indicates that synesthesia occurs at a later processing stage than does metacontrast masking.     

Identification and location tasks rely on different mental processes: a diffusion model account of validity effects in spatial cueing paradigms with emotional stimuli

Spatial cueing paradigms are popular tools to assess human attention to emotional stimuli, but different variants of these paradigms differ in what participants’ primary task is. In one variant, participants indicate the location of the target (location task), whereas in the other they indicate the shape of the target (identification task). In the present paper we test the idea that although these two variants produce seemingly comparable cue validity effects on response times, they rest on different underlying processes. Across four studies (total N?=?397; two in the supplement) using both variants and manipulating the motivational relevance of cue content, diffusion model analyses revealed that cue validity effects in location tasks are primarily driven by response biases, whereas the same effect rests on delay due to attention to the cue in identification tasks. Based on this, we predict and empirically support that a symmetrical distribution of valid and invalid cues would reduce cue validity effects in location tasks to a greater extent than in identification tasks. Across all variants of the task, we fail to replicate the effect of greater cue validity effects for arousing (vs. neutral) stimuli. We discuss the implications of these findings for best practice in spatial cueing research.     

Grapheme–color synesthesia can enhance immediate memory without disrupting the encoding of relational cues

Previous evidence has suggested that grapheme–color synesthesia can enhance memory for words, but little is known about how these photisms cue retrieval. Often, the encoding of specific features of individual words can disrupt the encoding of ordered relations between words, resulting in an overall decrease in recall accuracy. Here we show that the photisms arising from grapheme–color synesthesia do not function like these item-specific cues. The influences of high and low word frequency on the encoding of ordered relations and the accuracy of immediate free recall were compared across a group of 10 synesthetes and 48 nonsynesthetes. The main findings of Experiment 1 showed that the experience of synesthesia had no adverse effect on the encoding of ordered relations (as measured by input–output correspondence); furthermore, it enhanced recall accuracy in a strictly additive fashion across the two word frequency conditions. Experiment 2 corroborated these findings by showing that the synesthetes only outperformed the nonsynesthetes when the materials involved words and letters, not when they involved digits and spatial locations. Altogether, the present findings suggest that synesthesia can boost immediate memory performance without disrupting the encoding of ordered relations.     

Enhanced mental rotation ability in time-space synesthesia

Time-space synesthesia is a variant of sequence–space synesthesia and involves the involuntary association of months of the year with 2D and 3D spatial forms, such as arcs, circles, and ellipses. Previous studies have revealed conflicting results regarding the association between time-space synesthesia and enhanced spatial processing ability. Here, we tested 15 time-space synesthetes, and 15 non-synesthetic controls matched for age, education, and gender on standard tests of mental rotation ability, spatial working memory, and verbal working memory. Synesthetes performed better than controls on our test of mental rotation, but similarly to controls on tests of spatial and verbal working memory. Results support a dissociation between visuo-spatial imagery and spatial working memory capacity, and suggest time-space synesthesia is associated only with enhanced visuo-spatial imagery. These data are consistent with the time-space connectivity thesis that time-space synesthesia results from enhanced connectivity in the parietal lobe between regions supporting the representation of temporal sequences and those underlying visuo-spatial imagery.     

Memory for surface features of unfamiliar melodies: independent effects of changes in pitch and tempo

A melody’s identity is determined by relations between consecutive tones in terms of pitch and duration, whereas surface features (i.e., pitch level or key, tempo, and timbre) are irrelevant. Although surface features of highly familiar recordings are encoded into memory, little is known about listeners’ mental representations of melodies heard once or twice. It is also unknown whether musical pitch is represented additively or interactively with temporal information. In two experiments, listeners heard unfamiliar melodies twice in an initial exposure phase. In a subsequent test phase, they heard the same (old) melodies interspersed with new melodies. Some of the old melodies were shifted in key, tempo, or key and tempo. Listeners’ task was to rate how well they recognized each melody from the exposure phase while ignoring changes in key and tempo. Recognition ratings were higher for old melodies that stayed the same compared to those that were shifted in key or tempo, and detrimental effects of key and tempo changes were additive in between-subjects (Experiment 1) and within-subjects (Experiment 2) designs. The results confirm that surface features are remembered for melodies heard only twice. They also imply that key and tempo are processed and stored independently.     

The effect of visual signals on spatial decision making

We examined the effect of an irrelevant visual transient on the decision where to look for a hidden object. Participants also performed a conventional ‘inhibition of return’ localization task. In Experiments 1 and 2 the two tasks were blocked and in Experiments 3 and 4 they were randomly interleaved. In every experiment there was a bias to select the cued location in the spatial decision task. This facilitory effect was greatest when the cue occurred at a strategically unfavored location and even occurred for participants who reported strategically selecting a non-cued location, indicating that the facilitory effect is automatic and independent of other strategic biases. Inhibition of return was observed only when the tasks were blocked and the localization task preceded the decision task. The findings suggests that spatial decisions engage different attentional control settings than those engaged when detecting visual transients; and that this attentional mode affects the processing of visual transients such that they do not inhibit the subsequent speeded detection of onset targets.