Integrating face and voice in person perception

Integration of information from face and voice plays a central role in our social interactions. It has been mostly studied in the context of audiovisual speech perception: integration of affective or identity information has received comparatively little scientific attention. Here, we review behavioural and neuroimaging studies of face-voice integration in the context of person perception. Clear evidence for interference between facial and vocal information has been observed during affect recognition or identity processing. Integration effects on cerebral activity are apparent both at the level of heteromodal cortical regions of convergence, particularly bilateral posterior superior temporal sulcus (pSTS), and at 'unimodal' levels of sensory processing. Whether the latter reflects feedback mechanisms or direct crosstalk between auditory and visual cortices is as yet unclear.     

Integrating voice recognition into models of person perception

The results of one empirical study are presented to investigate whether voice recognition might profitably be integrated into a single IAC network for person perception. An identity priming paradigm was used to determine whether face perception and voice perception combined to influence one another. The results revealed within-modality priming of faces by prior presentations of faces, and of voices by prior presentation of voices. Critically, cross-modality priming was also revealed, confirming that the two modalities can be represented within a single system and can influence one another. These results are supported by the results of a simulation, and are discussed in terms of the theoretical development of IAC, and the benefits and future questions that arise from consideration of an integrated multimodal model of person perception.     

Other-race face perception

The other-race effect (or own-race bias or cross-racial identification effect) refers to the finding that recognition memory tends to be better for faces of members of subjects' own race than for faces of members of other races. The current study was designed to test the hypothesis that perceptual skills specific to identifying faces of particular racial groups contribute to this effect. On each of 50 trials, a photograph of a face was tachistoscopically presented for 120 ms, followed by a pattern mask and then a plain-view test pair composed of the previously presented face and a matched foil. As predicted, an other-race effect was obtained on this perceptual task: White subjects performed significantly more poorly on trials involving African American faces than on trials involving White faces, whereas no such difference was obtained among African American subjects.     

Direct perception and the predictive mind

Predictive approaches to the mind claim that perception, cognition, and action can be understood in terms of a single framework: a hierarchy of Bayesian models employing the computational strategy of predictive coding. Proponents of this view disagree, however, over the extent to which perception is direct on the predictive approach. I argue that we can resolve these disagreements by identifying three distinct notions of perceptual directness: psychological, metaphysical, and epistemological. I propose that perception is plausibly construed as psychologically indirect on the predictive approach, in the sense of being constructivist or inferential. It would be wrong to conclude from this, however, that perception is therefore indirect in a metaphysical or epistemological sense on the predictive approach. In the metaphysical case, claims about the inferential properties of constructivist perceptual mechanisms are consistent with both direct and indirect solutions to the metaphysical problem of perception (e.g. naïve realism, representationalism, sense datum theory). In the epistemological case, claims about the inferential properties of constructivist perceptual mechanisms are consistent with both direct and indirect approaches to the justification of perceptual belief. In this paper, I demonstrate how proponents of the predictive approach have conflated these distinct notions of perceptual directness and indirectness, and I propose alternative strategies for developing the philosophical consequences of the approach.     

Configurational information in face perception

A new facial composites technique is demonstrated, in which photographs of the top and bottom halves of different familiar faces fuse to form unfamiliar faces when aligned with each other. The perception of a novel configuration in such composite stimuli is sufficiently convincing to interfere with identification of the constituent parts (experiment 1), but this effect disappears when stimuli are inverted (experiment 2). Difficulty in identifying the parts of upright composites is found even for stimuli made from parts of unfamiliar faces that have only ever been encountered as face fragments (experiment 3). An equivalent effect is found for composites made from internal and external facial features of well-known people (experiment 4). These findings demonstrate the importance of configurational information in face perception, and that configurations are only properly perceived in upright faces.     

Integrating cognitive psychology, neurology and neuroimaging

In the last decade, there has been a dramatic increase in research effectively integrating cognitive psychology, functional neuroimaging, and behavioral neurology. This new work is typically conducting basic research into aspects of the human mind and brain. The present review features as examples of such integrations two series of studies by the author and his colleagues. One series, employing object recognition, mental motor imagery, and mental rotation paradigms, clarifies the nature of a cognitive process, imagined spatial transformations used in shape recognition. Among other implications, it suggests that when recognizing a hand's handedness, imagining one's body movement depends on cerebrally lateralized sensory-motor structures and deciding upon handedness depends on exact match shape confirmation. The other series, using cutaneous, tactile, and auditory pitch discrimination paradigms, elucidates the function of a brain structure, the cerebellum. It suggests that the cerebellum has non-motor sensory support functions upon which optimally fine sensory discriminations depend. In addition, six key issues for this integrative approach are reviewed. These include arguments for the value and greater use of: rigorous quantitative meta-analyses of neuroimaging studies; stereotactic coordinate-based data, as opposed to surface landmark-based data; standardized vocabularies capturing the elementary component operations of cognitive and behavioral tasks; functional hypotheses about brain areas that are consistent with underlying microcircuitry; an awareness that not all brain areas implicated by neuroimaging or neurology are necessarily directly involved in the associated cognitive or behavioral task; and systematic approaches to integrations of this kind.     

What's in a face: Individual differences in face perception

White and black females judged the similarity of all pairs of white and black male faces. An individual difference multidimensional scaling analysis of the similarity judgments indicated that most of the dimensions underlying the perceptions of male faces involved affective (honest, tense, attractive) characteristics rather than simple physical features (eye width, mouth height). The major physical dimension was face shape (long vs. wide). The dimensions were similar for black and white subjects. An individual difference hypothesis that we pay attention to those characteristics that we possess was partially confirmed.     

Impaired face recognition does not preclude intact whole face perception

We studied intact and impaired processes in a prosopagnosic patient (RP). In Experiment 1, RP showed an inversion superiority effect with both faces and objects, with better performance when stimuli were presented upside down than in normal upright orientation. In Experiment 2, we studied the effect of face configuration directly by comparing matching performance with normal vs. scrambled faces. RP was worse with normal than with scrambled faces, whereas normal controls showed an advantage of a good face context. In Experiment 3, RP showed interference from external face features on the evaluation of internal face features. These results indicate, first, that although RP is impaired in face recognition and face matching, he does still encode the whole face rather than relying completely on parts-based procedures. Second, RP has a deficit at the level of the configural processes involved in finding subtle differences between individual faces, as his performance is worse when presented with a normal face configuration than with scrambled or inverted faces.     

Females excel at basic face perception

Females are generally better than males at recognizing facial emotions. However, it is not entirely clear whether and in what way females may also excel at non-affective face recognition. Here, we tested males and females on two perceptual face recognition tasks that involved only neutral expressions: detection and identity discrimination. On face detection (Experiment 1), females were significantly more accurate than males in detecting upright faces. This gender difference was reduced during inverted face detection, and not present during tree detection, suggesting that the magnitude of the gender difference for performance co-varies with the extent to which face processing mechanisms are involved. On facial identity discrimination (Experiment 2), females again outperformed males, particularly when face images were masked by visual noise, or the delay between comparison face images was extended from 0.5 to 3 s. These results reveal a female advantage in processing face-specific information and underscore the role of perceptual factors in socially relevant gender differences.     

Environmental inversion effects in face perception

Visual processing is highly sensitive to stimulus orientation; for example, face perception is drastically worse when faces are oriented inverted vs. upright. However, stimulus orientation must be established in relation to a particular reference frame, and in most studies, several reference frames are conflated. Which reference frame(s) matter in the perception of faces? Here we describe a simple, novel method for dissociating effects of egocentric and environmental orientation on face processing. Participants performed one of two face-processing tasks (expression classification and recognition memory) as they lay horizontally, which served to disassociate the egocentric and environmental frames. We found large effects of egocentric orientation on performance and smaller but reliable effects of environmental orientation. In a follow-up control experiment, we ruled out the possibility that the latter could be explained by compensatory ocular counterroll. We argue that environmental orientation influences face processing, which is revealed when egocentric orientation is fixed.     

Brain mechanisms for predictive control by switching internal models: implications for higher-order cognitive functions

Humans can guide their actions toward the realization of their intentions. Flexible, rapid and precise realization of intentions and goals relies on the brain learning to control its actions on external objects and to predict the consequences of this control. Neural mechanisms that mimic the input–output properties of our own body and other objects can be used to support prediction and control, and such mechanisms are called internal models. We first summarize functional neuroimaging, behavioral and computational studies of the brain mechanisms related to acquisition, modular organization, and the predictive switching of internal models mainly for tool use. These mechanisms support predictive control and flexible switching of intentional actions. We then review recent studies demonstrating that internal models are crucial for the execution of not only immediate actions but also higher-order cognitive functions, including optimization of behaviors toward long-term goals, social interactions based on prediction of others’ actions and mental states, and language processing. These studies suggest that a concept of internal models can consistently explain the neural mechanisms and computational principles needed for fundamental sensorimotor functions as well as higher-order cognitive functions. Electronic supplementary material  The online version of this article (doi:) contains supplementary material, which is available to authorized users.     

Interference resolution in face perception and name retrieval

Selective retrieval is a rather difficult task, and especially so when one attempts to retrieve personal representations such as faces or names. Retrieval of memories under strong competition conditions is pervasive in human memory and some have suggested that inhibitory control is used to overcome interference between competing stimuli. In the present study, we used the retrieval practice paradigm to investigate if competition among personal representations (such as facial features and names) is also resolved by inhibitory mechanisms. This question is theoretically relevant, since personal representations have been said to have a special status on cognition. Moreover, some models of face recognition assume that interference can arise between different representations, but that this interference would be automatically and rapidly solved, with no need for a controlled inhibitory mechanism to act. In two experiments we showed RIF for facial features and familiar names, but only when participants had to actively retrieve some information. This suggests that personal information is subject to mechanisms of inhibitory control, which could help explain everyday life difficulties in processes such as face feature recognition or name retrieval.     

The electrophysiological correlates of integrated face and body-part perception

Previous studies have suggested that the human visual system processes faces and bodies holistically—that is, the different body parts are integrated into a unified representation. However, the time course of this integrative process is less known. In the present study, we investigated this issue by recording event-related potentials evoked by a face and two hands presented simultaneously and in different configurations. When the hands were rotated to obtain a biologically implausible configuration, a reduction of the P2 amplitude was observed relative to the condition in which the face and hands were retained in their veridical configuration and were supplemented with visual cues to highlight further the overall body posture. Our results show that the P2 component is sensitive to manipulations affecting the configuration of face and hand stimuli and suggest that the P2 reflects the operation of perceptual mechanisms responsible for the integrated processing of visually presented body parts.     

Thomas Willis: The faculties and his two cognitive frameworks

Thomas Willis’ 1664 study The anatomy of the brain is widely regarded as one of the first clinical studies of the brain. In The anatomy, Thomas Willis explicitly connected the cognitive faculties and the nerves. Willis’ later, 1672 work, The two discourses concerning the soul of brutes, severely undermined the materialism of Willis’ first study: he affirmed dualism and cognitive immateriality; changed the anatomical locations of cognition; and reasserted a division between the rational and sensitive souls. His exact motive to return to orthodoxy is unclear, but contemporary scholarship of Willis has compounded the confusion with by relying predominantly on The soul of brutes instead of The anatomy. We trace Willis’ career and examine his methodological practices, which help explain the historical practices and pressures. A closer examination of Willis’ Anatomy of the brain reveals a much more materialistic account of the brain, the faculties, and nervous system. In this article, we present our own analysis of Willis’ concept of rationality in the Anatomy and explain its importance for nervous physiology and understanding the analytic techniques for first defining faculty localizations. We then explain the role of the imagination and the immortal soul in the rearticulated anatomical concepts from The soul of brutes.