Body-scaled affordances in sensory substitution

The research field on sensory substitution devices has strong implications for theoretical work on perceptual consciousness. One of these implications concerns the extent to which the devices allow distal attribution. The present study applies a classic empirical approach on the perception of affordances to the field of sensory substitution. The reported experiment considers the perception of the stair-climbing affordance. Participants judged the climbability of steps apprehended through a vibrotactile sensory substitution device. If measured with standard metric units, climbability judgments of tall and short participants differed, but if measured in units of leg length, judgments did not differ. These results are similar to paradigmatic results in regular visual perception. We conclude that our sensory substitution device allows the perception of affordances. More generally, we argue that the theory of affordances may enrich theoretical debates concerning sensory substitution to a larger extent than has hitherto been the case.     

Training synaesthesia

Synaesthetic inducers such as graphemes are typically cultural artifacts. Thus, a learning component seems evident in synaesthesia (Simner et al, 2009 Brain 132 57-64). Normally, synaesthetes report to have their experiences since they can remember. Nevertheless, a recent training study suggests that synaesthesia can be mimicked in non-synaesthetes. To date, the role of learning during the development of synaesthesia is still debated. It is not clear whether synaesthesia can be learned or trained at all. To address this question, we compared a non-adaptive and an adaptive training for their effectiveness. We assessed their impact on two types of priming tasks, before and after the training. We found stronger priming in the adaptive training group suggesting that adaptive training is more efficient to mimic synaesthesia.     

Human spatial navigation via a visuo-tactile sensory substitution system

Spatial navigation within a real 3-D maze was investigated to study space perception on the sole basis of tactile information transmitted by means of a 'tactile vision substitution system' (TVSS) allowing the conversion of optical images-collected by a micro camera-into 'tactile images' via a matrix in contact with the skin. The development of such a device is based on concepts of cerebral and functional plasticity, enabling subjective reproduction of visual images from tactile data processing. Blindfolded sighted subjects had to remotely control the movements of a robot on which the TVSS camera was mounted. Once familiarised with the cues in the maze, the subjects were given two exploration sessions. Performance was analysed according to an objective point of view (exploration time, discrimination capacity), as well as a subjective one (speech). The task was successfully carried out from the very first session. As the subjects took a different path during each navigation, a gradual improvement in performance (discrimination and exploration time) was noted, generating a phenomenon of learning. Moreover, subjective analysis revealed an evolution of the spatialisation process towards distal attribution. Finally, some emotional expressions seemed to reflect the genesis of 'qualia' (emotional qualities of stimulation).     

Handedness and calendar orientations in time-space synaesthesia

In one common variant of time-space synaesthesia, individuals report the consistent experience of months bound to a spatial arrangement, commonly described as a circle extending outside of the body. Whereas the layout of these calendars has previously been thought to be relatively random and to differ greatly between synaesthetes, Study 1 provides the first evidence suggesting one critical aspect of these calendars is mediated by handedness: clockwise versus counter-clockwise orientation. A study of 34 time-space synaesthetes revealed a strong association between handedness and the orientation of circular calendars. That is, left-handed time-space synaesthetes tended to report counter-clockwise arrangements and right-handed synaesthetes clockwise. Study 2 tested whether a similar bias was present in non-synaesthetes whose task was to memorize and recall the spatial configuration of a clockwise and counter-clockwise calendar. Non-synaesthetes' relative performance on these two sorts of calendars was significantly correlated with their handedness scores in a pattern similar to synaesthetes. Specifically, left-handed controls performed better on counter-clockwise calendars compared to clockwise, and right-handed controls on clockwise over counter-clockwise. We suggest that the implicit biases seen in controls are mediated by similar mechanisms as in synaesthesia, highlighting the graded nature of synaesthetic associations.     

Lexical-gustatory synaesthesia: linguistic and conceptual factors

This study documents an unusual case of developmental synaesthesia, in which speech sounds induce an involuntary sensation of taste that is subjectively located in the mouth. JIW shows a highly structured, non-random relationship between particular combinations of phonemes (rather than graphemes) and the resultant taste, and this is influenced by a number of fine-grained phonemic properties (e.g. allophony, phoneme ordering). The synaesthesia is not found for environmental sounds. The synaesthesia, in its current form, is likely to have originated during vocabulary acquisition, since it is guided by learned linguistic and conceptual knowledge. The phonemes that trigger a given taste tend to also appear in the name of the corresponding foodstuff (e.g. /I/, /n/ and /s/ can trigger a taste of mince /mIns/) and there is often a semantic association between the triggering word and taste (e.g. the word blue tastes "inky"). The results suggest that synaesthesia does not simply reflect innate connections from one perceptual system to another, but that it can be mediated and/or influenced by a symbolic/conceptual level of representation.     

Frequency correlates in grapheme-color synaesthesia

Individuals with synaesthesia experience certain stimuli in more than one sensory modality. Most common is the linkage of letters and digits (graphemes) to colors. Whereas synaesthesia might be partly genetically determined, the linkages to specific colors are assumed to be learned. We present a systematic statistical analysis of synaesthetic color perception based on subjects' reproduction of individual colors for each grapheme, instead of simple verbal categorizations. The statistical analysis revealed that the color perceptions, measured with the HSL (hue, saturation, and luminance) scale, varied systematically among the different digits and letters. The frequencies of the digits and letters (in the German language) partly explained these systematic variations. However, digit frequency was more strongly related to color perception in the synaesthetes than was letter frequency. The results for digit and letter frequency indicate that experience with graphemes may shape synaesthetic color perception.     

Grapheme-color synaesthesia benefits rule-based Category learning

Researchers have long suspected that grapheme-color synaesthesia is useful, but research on its utility has so far focused primarily on episodic memory and perceptual discrimination. Here we ask whether it can be harnessed during rule-based Category learning. Participants learned through trial and error to classify grapheme pairs that were organized into categories on the basis of their associated synaesthetic colors. The performance of synaesthetes was similar to non-synaesthetes viewing graphemes that were physically colored in the same way. Specifically, synaesthetes learned to categorize stimuli effectively, they were able to transfer this learning to novel stimuli, and they falsely recognized grapheme-pair foils, all like non-synaesthetes viewing colored graphemes. These findings demonstrate that synaesthesia can be exploited when learning the kind of material taught in many classroom settings.     

Assessment of sensory substitution prosthesis potentialities in minimalist conditions of learning

Pattern recognition with a prosthesis substituting vision by audition was investigated. During 15 1‐hour sessions, nine blindfolded sighted subjects were trained to recognise 2D patterns by trial and error. In addition to a global assessment, recognition of pattern element nature (vertical bars, horizontal bars…), element size and element spatial arrangement were independently assessed for each pattern. Influence of experimental parameters (complexity level of patterns, exploration number of a pattern) on recognition was studied. Performances improved over sessions. As a rule, patterns element nature was less well recognised than element size and spatial arrangement. Experimental parameters influenced pattern recognition performance. Results are discussed in relation with auditory and visual perception as well as in the perspective to implement a learning protocol for future users of prosthesis. Copyright © 2006 John Wiley & Sons, Ltd.     

Perception with compensatory devices: from sensory substitution to sensorimotor extension

Sensory substitution devices provide through an unusual sensory modality (the substituting modality, e.g., audition) access to features of the world that are normally accessed through another sensory modality (the substituted modality, e.g., vision). In this article, we address the question of which sensory modality the acquired perception belongs to. We have recourse to the four traditional criteria that have been used to define sensory modalities: sensory organ, stimuli, properties, and qualitative experience ( Grice, 1962 ), to which we have added the criteria of behavioral equivalence ( Morgan, 1977 ), dedication ( Keeley, 2002 ), and sensorimotor equivalence ( O'Regan & Noë, 2001 ). We discuss which of them are fulfilled by perception through sensory substitution devices and whether this favors the view that perception belongs to the substituting or to the substituted modality. Though the application of a number of criteria might be taken to point to the conclusion that perception with a sensory substitution device belongs to the substituted modality, we argue that the evidence leads to an alternative view on sensory substitution. According to this view, the experience after sensory substitution is a transformation, extension, or augmentation of our perceptual capacities, rather than being something equivalent or reducible to an already existing sensory modality. We develop this view by comparing sensory substitution devices to other "mind-enhancing tools" such as pen and paper, sketchpads, or calculators. An analysis of sensory substitution in terms of mind-enhancing tools unveils it as a thoroughly transforming perceptual experience and as giving rise to a novel form of perceptual interaction with the environment.     

Visual experiences in the blind induced by an auditory sensory substitution device

In this report, the phenomenology of two blind users of a sensory substitution device – “The vOICe” – that converts visual images to auditory signals is described. The users both report detailed visual phenomenology that developed within months of immersive use and has continued to evolve over a period of years. This visual phenomenology, although triggered through use of The vOICe, is likely to depend not only on online visualization of the auditory signal but also on the users’ previous (albeit distant) experience of veridical vision (e.g. knowledge of shapes and visual perspective). Once established, the sensory substitution mapping between the auditory and visual domains is not confined to when the device is worn and, thus, may constitute an example of acquired synaesthesia.     

A self-organizing learning account of number-form synaesthesia

Some people automatically and involuntarily “see” mental images of numbers in spatial arrays when they think of numbers. This phenomenon, called number forms, shares three key characteristics with the other types of synaesthesia, within-individual consistency, between-individual variety, and mixture of regularity and randomness. A theoretical framework called SOLA (self-organizing learning account of number forms) is proposed, which explains the generation process of number forms and the origin of those three characteristics. The simulations replicated the qualitative properties of the shapes of number forms, the property that numbers are aligned in order of size, that discontinuity usually occurs at the point of carry, and that continuous lines tend to have many bends.     

Brain areas involved in synaesthesia: a review

Despite a recent upsurge of research, much remains unknown about the neurobiological mechanisms underlying synaesthesia. By integrating results obtained so far in Magnetic Resonance Imaging (MRI) studies, this contribution sheds light on the role of particular brain regions in synaesthetic experiences. First, in accordance with its sensory nature, it seems that the sensory brain areas corresponding to the type of synaesthetic experience are activated. Synaesthetic colour experiences can activate colour regions in occipito-temporal cortex, but this is not necessarily restricted to V4. Furthermore, sensory and motor brain regions have been obtained that extend beyond the particular type of synaesthesia studied. Second, differences in experimental setup, number and type of synaesthetes tested, and method to delineate regions of interest may help explain inconsistent results obtained in the BOLD-MRI (Blood Oxygen Level Dependent functional MRI) studies. Third, an overview of obtained results shows that a network of brain areas rather than a single brain region underlies synaesthesia. Six brain regions of overlapping results emerge, these regions are in sensory and motor regions as well as 'higher level' regions in parietal and frontal lobe. We propose that these regions are related to three different cognitive processes inherently part of synaesthesia; the sensory processes, the (attentional) 'binding' processes, and cognitive control processes. Finally, we discuss how these functional and structural brain properties might relate to the development of synaesthesia. In particular, we believe this relationship is better understood by separating the question what underlies the presence of synaesthesia ('trait') from what determines particular synaesthetic associations ('type').     

Ovals of time: time-space associations in synaesthesia

We examine a condition in which units of time, such as months of the year, are associated with specific locations in space. For individuals with this time-space synaesthesia, contiguous time units such as months are spatially linked forming idiosyncratically shaped patterns such as ovals, oblongs or circles. For some individuals, each time unit appears in a highly specific colour. For instance, one of the synaesthetes we studied experienced December as a red area located at arms length to the left of their body. For the same individual May was a blue area located roughly at arms length to the right of their body. We studied four synaesthetes who report spatial associations for the months of the year. We found that the time-space associations experienced by these individuals were consistent across test-retest. In addition, month names directed visual attention to particular locations in space. For some synaesthetes, this directing of spatial attention was quite rapid-in accord with their reports that month names involuntarily bring to mind spatial locations.     

Colours sometimes count: awareness and bidirectionality in grapheme-colour synaesthesia

Three experiments were conducted with 10 grapheme-colour synaesthetes and 10 matched controls to investigate (a) whether awareness of the inducer grapheme is necessary for synaesthetic colour induction and (b) whether grapheme-colour synaesthesia may be bidirectional in the sense that not only do graphemes induce colours, but that colours influence the processing of graphemes. Using attentional blink and Stroop paradigms with digit targets, we found that some synaesthetes did report “seeing” synaesthetic colours even when they were not able to report the inducing digit. Moreover, congruency effects (effects of matching the colour of digit presentation with the synaesthetic colour associated with that digit) suggested that grapheme-colour synaesthesia can be bidirectional, at least for some synaesthetes.     

Creativity and involvement in art in different types of synaesthesia

The aim of the present study was to test the relationship between different types of synaesthesia and their involvement in art, creative, and visual abilities. We tested 20 grapheme‐colour, 18 sound‐colour, 19 grapheme‐colour‐and‐sound‐colour, 20 sequence‐space synaesthetes, and the same number of controls matched by age, gender, and education. We assessed the number of artistic professions, involvement in art, and the performance in psychometric tests of divergent and convergent creativity, as well as visual and visuo‐spatial abilities. Results show a higher prevalence of artists among synaesthetes, especially sound‐colour synaesthetes. Sound‐colour synaesthetes also showed a higher involvement in artistic activities overall while sequence‐space synaesthetes showed higher involvement in visual art. Only grapheme‐colour‐and‐sound‐colour synaesthetes showed significantly higher divergent creativity compared to matched controls. Additionally, overall, synaesthetes scored higher in visuo‐spatial abilities (i.e., mental rotation). For synaesthetes and controls, visuo‐spatial abilities correlated with divergent creativity. We discuss that synaesthetes’ higher involvement in art is not necessarily reflected in their basic creative abilities.