Can synaesthesia research inform cognitive science?

The renaissance of synaesthesia research has produced many insights regarding the aetiology and mechanisms that might underlie this intriguing phenomenon, which abnormally binds features between and within modalities. Synaesthesia is interesting in its own right, but whether it contributes to our knowledge of neurocognitive systems that underlie non-synaesthete experience is an open question. In this review, we show that results from the field of synaesthesia can constrain cognitive theories in numerical cognition, automaticity, crossmodal interaction and awareness. Therefore, research of synaesthesia provides a unique window into other domains of cognitive neuroscience. We conclude that the study of synaesthesia could advance our understanding of the normal and abnormal human brain and cognition.     

Synthetic synaesthesia and sensory substitution

Visual information can be provided to blind users through sensory substitution devices that convert images into sound. Through extensive use to develop expertise, some blind users have reported visual experiences when using such a device. These blind expert users have also reported visual phenomenology to other sounds even when not using the device. The blind users acquired synthetic synaesthesia, with visual experience evoked by sounds only after gaining such expertise. Sensorimotor learning may facilitate and perhaps even be required to develop expertise in the use of multimodal information. Furthermore, other areas where expertise is acquired in dividing attention amongst cross-modal information or integrating such information might also give rise to synthetic synaesthesia.     

Brain areas involved in synaesthesia: a review

Despite a recent upsurge of research, much remains unknown about the neurobiological mechanisms underlying synaesthesia. By integrating results obtained so far in Magnetic Resonance Imaging (MRI) studies, this contribution sheds light on the role of particular brain regions in synaesthetic experiences. First, in accordance with its sensory nature, it seems that the sensory brain areas corresponding to the type of synaesthetic experience are activated. Synaesthetic colour experiences can activate colour regions in occipito-temporal cortex, but this is not necessarily restricted to V4. Furthermore, sensory and motor brain regions have been obtained that extend beyond the particular type of synaesthesia studied. Second, differences in experimental setup, number and type of synaesthetes tested, and method to delineate regions of interest may help explain inconsistent results obtained in the BOLD-MRI (Blood Oxygen Level Dependent functional MRI) studies. Third, an overview of obtained results shows that a network of brain areas rather than a single brain region underlies synaesthesia. Six brain regions of overlapping results emerge, these regions are in sensory and motor regions as well as 'higher level' regions in parietal and frontal lobe. We propose that these regions are related to three different cognitive processes inherently part of synaesthesia; the sensory processes, the (attentional) 'binding' processes, and cognitive control processes. Finally, we discuss how these functional and structural brain properties might relate to the development of synaesthesia. In particular, we believe this relationship is better understood by separating the question what underlies the presence of synaesthesia ('trait') from what determines particular synaesthetic associations ('type').     

Synaesthesia: learned or lost?

The question why synaesthesia, an atypical binding within or between modalities, occurs is both enduring and important. Two explanations have been provided: (1) a congenital explanation: we are all born as synaesthetes but most of us subsequently lose the experience due to brain development; (2) a learning explanation: synaesthesia is related to some learning process during childhood. Three recent studies provide conflicting support for these explanations. Two studies supported the idea that synaesthesia is learned by showing that the frequency of everyday language implicitly modulates the synaesthetic experience. Another study argued that synaesthesia reflects basic, innate magnitude representations. In this paper we reassess these points of view, and show that it is possible for both to be valid. These findings are integrated into an interactive specialization account of development in order to explain the neuronal mechanism underlying synaesthesia.     

What is the relationship between Aphantasia,Synaesthesia and Autism?

For people with aphantasia, visual imagery is absent or markedly impaired. Here, we investigated the relationship between aphantasia and two other neurodevelopmental conditions also linked to imagery differences: synaesthesia, and autism. In Experiment 1a and 1b, we asked whether aphantasia and synaesthesia can co-occur, an important question given that synaesthesia is linked to strong imagery. Taking grapheme-colour synaesthesia as a test case, we found that synaesthesia can be objectively diagnosed in aphantasics, suggesting visual imagery is not necessary for synaesthesia to occur. However, aphantasia influenced the type of synaesthesia experienced (favouring ‘associator’ over ‘projector’ synaesthesia - a distinction tied to the phenomenology of the synaesthetic experience). In Experiment 2, we asked whether aphantasics have traits associated with autism, an important question given that autism – like aphantasia – is linked to weak imagery. We found that aphantasics reported more autistic traits than controls, with weaknesses in imagination and social skills.     

The neural basis of illusory gustatory sensations: two rare cases of lexical-gustatory synaesthesia

Lexical-gustatory synaesthesia is a rare phenomenon in which the individual experiences flavour sensations when they read, hear, or imagine words. In this study, we provide insight into the neural basis of this form of synaesthesia using functional neuroimaging. Words known to evoke pleasant, neutral, and unpleasant synaesthetic tastes and synaesthetically tasteless words were presented to two lexical-gustatory synaesthetes, during fMRI scanning. Ten non-synaesthetic participants were also scanned on the same list of words. The synaesthetic brain displayed a different pattern of activity to words when compared to the non-synaesthetes, with insula activation related to viewing words that elicited tastes that have an associated emotional valence (i.e., pleasant or unpleasant tastes). The subjective intensity of the synaesthesia was correlated with activity in the medial parietal lobes (precuneus/retrosplenial cortex), which are implicated in polymodal imagery and self-directed thought. This region has also previously been activated in studies of lexical-colour synaesthesia, suggesting its role may not be limited to the type of synaesthesia explored here.     

Is the sky 2? Contextual priming in grapheme-color synaesthesia

Grapheme-color synaesthesia is a neurological phenomenon in which particular graphemes, such as the numeral 9, automatically induce the simultaneous perception of a particular color, such as the color red. To test whether the concurrent color sensations in grapheme-color synaesthesia are treated as meaningful stimuli, we recorded event-related brain potentials as 8 synaesthetes and 8 matched control subjects read sentences such as "Looking very clear, the lake was the most beautiful hue of 7." In synaesthetes, but not control subjects, congruous graphemes, compared with incongruous graphemes, elicited a more negative N1 component, a less positive P2 component, and a less negative N400 component. Thus, contextual congruity of synaesthetically induced colors altered the brain response to achromatic graphemes beginning 100 ms postonset, affecting pattern-recognition, perceptual, and meaning-integration processes. The results suggest that grapheme-color synaesthesia is automatic and perceptual in nature and also suggest that the connections between colors and numbers are bidirectional.     

保留的跨通道启动与老化：对于近期观点的总结

通道内重复启动的研究提示老年人内隐记忆未受损, 这不只体现在视觉通道上还包括其他感觉通道(例如触觉、听觉和嗅觉)。然而很少有研究考察启动任务是否具有通道特异性。在以年轻人为被试的研究中发现跨通道迁移(视觉到触觉和触觉到视觉)和通道内迁移(视觉到视觉, 触觉到触觉)具有相似性。一项最近的研究进一步探索老年人在跨通道启动任务上是否受损。结果显示视觉和触觉间的跨通道启动在年轻被试和老年被试上都是保留的且具有对称性。并且, 对于自然声响、图片的通道内和跨通道启动任务随着年老化发展仍旧保留。这些行为结果和其它最近神经科学结果显示跨通道启动发生于枕叶后纹状皮层区, 而这一区域在老年人中未损坏。这一领域未来的研究方向包括利用不同知觉通道间、利用熟悉的和新异的刺激并结合行为的和脑成像的方法, 通过设计完善的跨通道启动来研究正常老人与阿尔兹海默病人, 还包括将设计得完善的启动任务包括在用于改善老年人记忆功能的项目中。     

The aesthetic appeal of auditory-visual synaesthetic perceptions in people without synaesthesia

The term 'visual music' refers to works of art in which both hearing and vision are directly or indirectly stimulated. Our ability to create, perceive, and appreciate visual music is hypothesised to rely on the same multisensory processes that support auditory visual (AV) integration in other contexts. Whilst these mechanisms have been extensively studied, there has been little research on how these processes affect aesthetic judgments (of liking or preference). Studies of synaesthesia in which sound evokes vision and studies of cross-modal biases in non-synaesthetes have revealed non-arbitrary mappings between visual and auditory properties (eg high-pitch sounds being smaller and brighter). In three experiments, we presented members of the general population with animated AV clips derived from synaesthetic experiences and contrasted them with a number of control conditions. The control conditions consisted of the same clips rotated or with the colour changed, random AV pairings, or animated clips generated by non-synaesthetes. Synaesthetic AV animations were generally preferred over the control conditions. The results suggest that non-arbitrary AV mappings, present in the experiences of synaesthetes, can be readily appreciated by others and may underpin our tendency to engage with certain forms of art.     

Is synaesthesia one condition or many? A large-scale analysis reveals subgroups

Synaesthesia is a broadly defined neural phenomenon in which stimulation of a sense or concept triggers a second perception not normally associated with the stimulus. For example, letters or numbers may trigger a colour experience, sounds may trigger a taste sensation, or tastes may trigger a feeling of touch. Dozens of forms of synaesthesia have been reported, but the relationship between the different forms has not been studied: is someone with a particular form of synaesthesia likely to possess other types? If so, which ones? As an inroad to illuminating underlying mechanisms, we here examine which different synaesthesia types tend to co-occur. We analyzed reports of the forms of synaesthesia experienced by 19,133 participants who completed the Synaesthesia Battery (Eagleman, Kagan, Nelson, Sagaram, & Sarma, 2007), using correlation analysis, exploratory factor analysis (EFA), confirmatory factor analysis (CFA), and multidimensional scaling (MDS). Our analyses converged on the finding of five distinct groupings of synaesthesia forms. We label these coloured sequence synaesthesias (CSSs), coloured music synaesthesias, non-visual sequela synaesthesias, spatial sequence synaesthesia (SSS), and coloured sensation synaesthesias. Collectively, our findings reveal that synaesthesia is an umbrella term that encompasses several distinct groups with independent probabilities of expression, and this may in turn suggest distinct underlying mechanisms and the possibility of different genetic bases.     

The absence of an auditory-visual attentional blink is not due to echoic memory

The second of two targets is often missed when presented shortly after the first target--a phenomenon referred to as the attentional blink (AB). Whereas the AB is a robust phenomenon within sensory modalities, the evidence for cross-modal ABs is rather mixed. Here, we test the possibility that the absence of an auditory-visual AB for visual letter recognition when streams of tones are used is due to the efficient use of echoic memory, allowing for the postponement of auditory processing. However, forcing participants to immediately process the auditory target, either by presenting interfering sounds during retrieval or by making the first target directly relevant for a speeded response to the second target, did not result in a return of a cross-modal AB. Thefindings argue against echoic memory as an explanation for efficient cross-modal processing. Instead, we hypothesized that a cross-modal AB may be observed when the different modalities use common representations, such as semantic representations. In support of this, a deficit for visual letter recognition returned when the auditory task required a distinction between spoken digits and letters.     

Vision merges with touch in a purely tactile discrimination

To construct a coherent percept of the world, the brain continuously combines information across multiple sensory modalities. Simple stimuli from different modalities are usually assumed to be processed in distinct brain areas. However, there is growing evidence that simultaneous stimulation of multiple modalities can influence the activity in unimodal sensory areas and improve or impair performance in unimodal tasks. Do these effects reflect a genuine cross-modal integration of sensory signals, or are they due to changes in the perceiver's ability to locate the stimulus in time and space? We used a behavioral measure to differentiate between these explanations. Our results demonstrate that, under certain circumstances, a noninformative flash of light can have facilitative or detrimental effects on a simple tactile discrimination. The effect of the visual flash mimics that produced by a constant tactile pedestal stimulus. These findings reveal that sensory signals from different modalities can be integrated, even for perceptual judgments within a single modality.     

Hearing words and seeing colours: an experimental investigation of a case of synaesthesia

A case of 'chromatic-lexical' (colour-word) synaesthesia is described, and its genuineness confirmed using the criterion of stable cross-modality imagery across time. The synaesthesia could not be accounted for by a memory hypothesis, nor was it associated with any psychiatric condition. Further analysis did not identify any semantic relationship between real words and colours, but the colours of nonwords were determined by the colours of the individual letters. Numbers also had their own stable colours. The experience of synaesthesia was triggered by other auditory stimuli, but most strongly by words. Cortical electrophysiological recording failed to reveal any abnormalities. An unusual organisation of modalities in the brain is postulated to account for the phenomenon.     

Spatial grouping resolves ambiguity to drive temporal recalibration

Cross-modal temporal recalibration describes a shift in the point of subjective simultaneity (PSS) between 2 events following repeated exposure to asynchronous cross-modal inputs--the adaptors. Previous research suggested that audiovisual recalibration is insensitive to the spatial relationship between the adaptors. Here we show that audiovisual recalibration can be driven by cross-modal spatial grouping. Twelve participants adapted to alternating trains of lights and tones. Spatial position was manipulated, with alternating sequences of a light then a tone, or a tone then a light, presented on either side of fixation (e.g., left tone--left light--right tone--right light, etc.). As the events were evenly spaced in time, in the absence of spatial-based grouping it would be unclear if tones were leading or lagging lights. However, any grouping of spatially colocalized cross-modal events would result in an unambiguous sense of temporal order. We found that adapting to these stimuli caused the PSS between subsequent lights and tones to shift toward the temporal relationship implied by spatial-based grouping. These data therefore show that temporal recalibration is facilitated by spatial grouping.     

Synaesthesia is associated with enhanced, self-rated visual imagery

Although the condition known as synaesthesia is currently undergoing a scientific resurgence, to date the literature has largely focused on the heterogeneous nature of synaesthesia across individuals. In order to provide a better understanding of synaesthesia, however, general characteristics need to be investigated. Synaesthetic experiences are often described as occurring 'internally' or in the 'mind's eye', which is remarkably similar to how we would describe our experience of visual mental imagery. We assessed the role of visual imagery in synaesthesia by sampling a large group of synaesthetes and found that they report experiencing more vivid mental images than controls. These findings have important implications for our general understanding of synaesthesia and, in particular, emphasize the need to control for visual imagery in behavioural and neuroimaging paradigms.     

Colored-Speech Synaesthesia Is Triggered by Multisensory, Not Unisensory, Perception

ABSTRACT— Although it is estimated that as many as 4% of people experience some form of enhanced cross talk between (or within) the senses, known as synaesthesia, very little is understood about the level of information processing required to induce a synaesthetic experience. In work presented here, we used a well-known multisensory illusion called the McGurk effect to show that synaesthesia is driven by late, perceptual processing, rather than early, unisensory processing. Specifically, we tested 9 linguistic-color synaesthetes and found that the colors induced by spoken words are related to what is perceived (i.e., the illusory combination of audio and visual inputs) and not to the auditory component alone. Our findings indicate that color-speech synaesthesia is triggered only when a significant amount of information processing has occurred and that early sensory activation is not directly linked to the synaesthetic experience.     

Cross-modality correspondence between pitch and spatial location modulates attentional orienting

The brain constantly integrates incoming signals across the senses to form a cohesive view of the world. Most studies on multisensory integration concern the roles of spatial and temporal parameters. However, recent findings suggest cross-modal correspondences (eg high-pitched sounds associated with bright, small objects located high up) also affect multisensory integration. Here, we focus on the association between auditory pitch and spatial location. Surprisingly little is known about the cognitive and perceptual roots of this phenomenon, despite its long use in ergonomic design. In a series of experiments, we explore how this cross-modal mapping affects the allocation of attention with an attentional cuing paradigm. Our results demonstrate that high and low tones induce attention shifts to upper or lower locations, depending on pitch height. Furthermore, this pitch-induced cuing effect is susceptible to contextual manipulations and volitional control. These findings suggest the cross-modal interaction between pitch and location originates from an attentional level rather than from response mapping alone. The flexible contextual mapping between pitch and location, as well as its susceptibility to top-down control, suggests the pitch-induced cuing effect is primarily mediated by cognitive processes after initial sensory encoding and occurs at a relatively late stage of voluntary attention orienting.     

Training synaesthesia

Synaesthetic inducers such as graphemes are typically cultural artifacts. Thus, a learning component seems evident in synaesthesia (Simner et al, 2009 Brain 132 57-64). Normally, synaesthetes report to have their experiences since they can remember. Nevertheless, a recent training study suggests that synaesthesia can be mimicked in non-synaesthetes. To date, the role of learning during the development of synaesthesia is still debated. It is not clear whether synaesthesia can be learned or trained at all. To address this question, we compared a non-adaptive and an adaptive training for their effectiveness. We assessed their impact on two types of priming tasks, before and after the training. We found stronger priming in the adaptive training group suggesting that adaptive training is more efficient to mimic synaesthesia.     

Time–space synaesthesia – A cognitive advantage?

Is synaesthesia cognitively useful? Individuals with time–space synaesthesia experience time units (such as months of the year) as idiosyncratic spatial forms, and report that these forms aid them in mentally organising their time. In the present study, we hypothesised that time–space synaesthesia would facilitate performance on a time-related cognitive task. Synaesthetes were not specifically recruited for participation; instead, likelihood of time–space synaesthesia was assessed on a continuous scale based on participants’ responses during a semi-structured interview. Participants performed a month-manipulation task, which involved naming every second month or every third month in reverse-chronological order, beginning and ending with a target month. Using hierarchical multiple regression, we found that time–space synaesthesia corresponded with faster performance on both versions of the task. We propose that time–space synaesthesia may expedite the cognitive manipulation of time-based information. Our results also indicate that synaesthesia is far less unusual than widely believed.