Telling faces together: Learning new faces through exposure to multiple instances

We are usually able to recognize novel instances of familiar faces with little difficulty, yet recognition of unfamiliar faces can be dramatically impaired by natural within-person variability in appearance. In a card-sorting task for facial identity, different photos of the same unfamiliar face are often seen as different people. Here we report two card-sorting experiments in which we manipulate whether participants know the number of identities present. Without constraints, participants sort faces into many identities. However, when told the number of identities present, they are highly accurate. This minimal contextual information appears to support viewers in “telling faces together”. In Experiment 2 we show that exposure to within-person variability in the sorting task improves performance in a subsequent face-matching task. This appears to offer a fast route to learning generalizable representations of new faces.     

Variability in photos of the same face

Psychological studies of face recognition have typically ignored within-person variation in appearance, instead emphasising differences between individuals. Studies typically assume that a photograph adequately captures a person’s appearance, and for that reason most studies use just one, or a small number of photos per person. Here we show that photographs are not consistent indicators of facial appearance because they are blind to within-person variability. Crucially, this within-person variability is often very large compared to the differences between people. To investigate variability in photos of the same face, we collected images from the internet to sample a realistic range for each individual. In Experiments 1 and 2, unfamiliar viewers perceived images of the same person as being different individuals, while familiar viewers perfectly identified the same photos. In Experiment 3, multiple photographs of any individual formed a continuum of good to bad likeness, which was highly sensitive to familiarity. Finally, in Experiment 4, we found that within-person variability exceeded between-person variability in attractiveness. These observations are critical to our understanding of face processing, because they suggest that a key component of face processing has been ignored. As well as its theoretical significance, this scale of variability has important practical implications. For example, our findings suggest that face photographs are unsuitable as proof of identity.     

Flexible voices: Identity perception from variable vocal signals

Human voices are extremely variable: The same person can sound very different depending on whether they are speaking, laughing, shouting or whispering. In order to successfully recognise someone from their voice, a listener needs to be able to generalize across these different vocal signals (‘telling people together’). However, in most studies of voice-identity processing to date, the substantial within-person variability has been eliminated through the use of highly controlled stimuli, thus focussing on how we tell people apart. We argue that this obscures our understanding of voice-identity processing by controlling away an essential feature of vocal stimuli that may include diagnostic information. In this paper, we propose that we need to extend the focus of voice-identity research to account for both “telling people together” as well as “telling people apart.” That is, we must account for whether, and to what extent, listeners can overcome within-person variability to obtain a stable percept of person identity from vocal cues. To do this, our theoretical and methodological frameworks need to be adjusted to explicitly include the study of within-person variability.

     

The “parts and wholes” of face recognition: A review of the literature

It has been claimed that faces are recognized as a “whole” rather than by the recognition of individual parts. In a paper published in the Quarterly Journal of Experimental Psychology in 1993, Martha Farah and I attempted to operationalize the holistic claim using the part/whole task. In this task, participants studied a face and then their memory presented in isolation and in the whole face. Consistent with the holistic view, recognition of the part was superior when tested in the whole-face condition compared to when it was tested in isolation. The “whole face” or holistic advantage was not found for faces that were inverted, or scrambled, nor for non-face objects, suggesting that holistic encoding was specific to normal, intact faces. In this paper, we reflect on the part/whole paradigm and how it has contributed to our understanding of what it means to recognize a face as a “whole” stimulus. We describe the value of part/whole task for developing theories of holistic and non-holistic recognition of faces and objects. We discuss the research that has probed the neural substrates of holistic processing in healthy adults and people with prosopagnosia and autism. Finally, we examine how experience shapes holistic face recognition in children and recognition of own- and other-race faces in adults. The goal of this article is to summarize the research on the part/whole task and speculate on how it has informed our understanding of holistic face processing.     

Event-related potentials reveal the development of stable face representations from natural variability

Natural variability between instances of unfamiliar faces can make it difficult to reconcile two images as the same person. Yet for familiar faces, effortless recognition occurs even with considerable variability between images. To explore how stable face representations develop, we employed incidental learning in the form of a face sorting task. In each trial, multiple images of two facial identities were sorted into two corresponding piles. Following the sort, participants showed evidence of having learnt the faces performing more accurately on a matching task with seen than with unseen identities. Furthermore, ventral temporal event-related potentials were more negative in the N250 time range for previously seen than for previously unseen identities. These effects appear to demonstrate some degree of abstraction, rather than simple picture learning, as the neurophysiological and behavioural effects were observed with novel images of the previously seen identities. The results provide evidence of the development of facial representations, allowing a window onto natural mechanisms of face learning.     

From Pixels to People: A Model of Familiar Face Recognition

Research in face recognition has largely been divided between those projects concerned with front-end image processing and those projects concerned with memory for familiar people. These perceptual and cognitive programmes of research have proceeded in parallel, with only limited mutual influence. In this paper we present a model of human face recognition which combines both a perceptual and a cognitive component. The perceptual front-end is based on principal components analysis of face images, and the cognitive back-end is based on a simple interactive activation and competition architecture. We demonstrate that this model has a much wider predictive range than either perceptual or cognitive models alone, and we show that this type of combination is necessary in order to analyse some important effects in human face recognition. In sum, the model takes varying images of “known” faces and delivers information about these people.     

Robust representations of individual faces in chimpanzees (<Emphasis Type="Italic">Pan troglodytes</Emphasis>) but not monkeys (<Emphasis Type="Italic">Macaca mulatta</Emphasis>)

Being able to recognize the faces of our friends and family members no matter where we see them represents a substantial challenge for the visual system because the retinal image of a face can be degraded by both changes in the person (age, expression, pose, hairstyle, etc.) and changes in the viewing conditions (direction and degree of illumination). Yet most of us are able to recognize familiar people effortlessly. A popular theory for how face recognition is achieved has argued that the brain stabilizes facial appearance by building average representations that enhance diagnostic features that reliably vary between people while diluting features that vary between instances of the same person. This explains why people find it easier to recognize average images of people, created by averaging multiple images of the same person together, than single instances (i.e. photographs). Although this theory is gathering momentum in the psychological and computer sciences, there is no evidence of whether this mechanism represents a unique specialization for individual recognition in humans. Here we tested two species, chimpanzees (Pan troglodytes) and rhesus monkeys (Macaca mulatta), to determine whether average images of different familiar individuals were easier to discriminate than photographs of familiar individuals. Using a two-alternative forced-choice, match-to-sample procedure, we report a behaviour response profile that suggests chimpanzees encode the faces of conspecifics differently than rhesus monkeys and in a manner similar to humans.     

Finding an unfamiliar face in a line-up: Viewing multiple images of the target is beneficial on target-present trials but costly on target-absent trials

When viewing unfamiliar faces, photographs of the same person often are perceived as belonging to different people and photographs of different people as belonging to the same person. Identity matching of unfamiliar faces is especially challenging when the photographs are of a person whose ethnicity differs from that of the observer. In contrast, matching is trivial when viewing familiar faces, regardless of race. Viewing multiple images of an own-race target identity improves accuracy on a line-up task when the target is known to be present (Dowsett et al., 2016, Q J Exp Psychol, 69, 1), suggesting that exposure to within-person variability in appearance is key to face learning. Across three experiments, we show that viewing multiple images of a target identity also improves accuracy for other-race faces on target-present trials. However, viewing multiple images decreases accuracy (i.e., increases false alarms) on target-absent trials for both own- and other-race faces. We discuss the implications of our findings for models of face recognition and for forensic settings.     

When the face fits: Recognition of celebrities from matching and mismatching faces and voices

The results of two experiments are presented in which participants engaged in a face-recognition or a voice-recognition task. The stimuli were face–voice pairs in which the face and voice were co-presented and were either “matched” (same person), “related” (two highly associated people), or “mismatched” (two unrelated people). Analysis in both experiments confirmed that accuracy and confidence in face recognition was consistently high regardless of the identity of the accompanying voice. However accuracy of voice recognition was increasingly affected as the relationship between voice and accompanying face declined. Moreover, when considering self-reported confidence in voice recognition, confidence remained high for correct responses despite the proportion of these responses declining across conditions. These results converged with existing evidence indicating the vulnerability of voice recognition as a relatively weak signaller of identity, and results are discussed in the context of a person-recognition framework.     

Dissociation of facial attractiveness and distinctiveness processing in congenital prosopagnosia

According to the influential model of Bruce and Young (1986) socially relevant facial information is processed separately from facial information leading to individual face recognition. In recent years functional imaging has identified a network of distinct occipitotemporal cortex areas for the processing of these two kinds of information. Functionally it is not clear at which processing level the “social” and the “recognition” pathways diverge. The study of subjects with a profound face recognition and learning deficit (congenital prosopagnosia—cPA) promises for a better understanding of this issue. We therefore tested the perception of attractiveness (a cue of prime social importance) and distinctiveness (a facial feature related to recognition) in 14 people with cPA. Although attractiveness ratings were highly consistent with controls, cPA subjects' distinctiveness ratings showed random patterns. This dissociation of normal attractiveness processing and impaired distinctiveness processing in cPA helps to specifies the nature of the impairment in this condition while shedding light on the functional architecture of normal face processing.     

Distorting Face Representations in Newborn Brains

What role does experience play in the development of face recognition? A growing body of evidence indicates that newborn brains need slowly changing visual experiences to develop accurate visual recognition abilities. All of the work supporting this “slowness constraint” on visual development comes from studies testing basic-level object recognition. Here, we present the results of controlled-rearing experiments that provide evidence for a slowness constraint on the development of face recognition, a prototypical subordinate-level object recognition task. We found that (1) newborn chicks can rapidly develop view-invariant face recognition and (2) the development of this ability relies on experience with slowly moving faces. When chicks were reared with quickly moving faces, they built distorted face representations that largely lacked invariance to viewpoint changes, effectively “breaking” their face recognition abilities. These results provide causal evidence that slowly changing visual experiences play a critical role in the development of face recognition, akin to basic-level object recognition. Thus, face recognition is not a hardwired property of vision but is learned rapidly as the visual system adapts to the temporal structure of the animal's visual environment.     

Ensemble coding of facial identity is not refined by experience: Evidence from other-race and inverted faces

The ability to recognize identity despite within-person variability in appearance is likely a face-specific skill and shaped by experience. Ensemble coding – the automatic extraction of the average of a stimulus array – has been proposed as a mechanism underlying face learning (allowing one to recognize novel instances of a newly learned face). We investigated whether ensemble encoding, like face learning and recognition, is refined by experience by testing participants with upright own-race faces and two categories of faces with which they lacked experience: other-race faces (Experiment 1) and inverted faces (Experiment 2). Participants viewed four images of an unfamiliar identity and then were asked whether a test image of that same identity had been in the study array. Each test image was a matching exemplar (from the array), matching average (the average of the images in the array), non-matching exemplar (a novel image of the same identity), or non-matching average (an average of four different images of the same identity). Adults showed comparable ensemble coding for all three categories (i.e., reported that matching averages had been present more than non-matching averages), providing evidence that this early stage of face learning is not shaped by face-specific experience.     

Understanding the multiframe caricature advantage for recognizing facial composites

Eyewitnesses often construct a “composite” face of a person they saw commit a crime, a picture that police use to identify suspects. We described a technique (Frowd, Bruce, Ross, McIntyre, & Hancock, 2007) based on facial caricature to facilitate recognition of these images: Correct naming substantially improves when composites are seen with progressive positive caricature, where distinctive information is enhanced, and then with progressive negative caricature, the opposite. Over the course of four experiments, the underpinnings of this mechanism were explored. Positive-caricature levels were found to be largely responsible for improving naming of composites, with some benefit from negative-caricature levels. Also, different frame-presentation orders (forward, reverse, random, repeated) facilitated equivalent naming benefit relative to static composites. Overall, the data indicate that composites are usually constructed as negative caricatures.     

Active and passive-touch during interpersonal multisensory stimulation change self–other boundaries

In the “enfacement” illusion seeing an unfamiliar face being touched at the same time as one’s own face evokes changes in self-face recognition. We investigated the contribution of proprioceptive and motor signals derived from self-generated actions in the sensory-driven malleability of self–other boundaries during the “enfacement” illusion. Changes in self-face recognition during active- and passive-touch interpersonal visuo-tactile stimulation were quantified by means of psychophysical and psychometric tasks. Active- and passive-touch evoked comparable changes in the categorical boundaries of self–other distinction, changing the extent to which the other is assimilated into the mental self-representation. Actively touching or simply feeling touch on one’s own face with concurrent observed touch on someone else’s face seems to elicit comparable changes in self-recognition, suggesting that afferent input might be sufficient for updating one’s body-image, although some components of the experience of self-identification seem to be more affected by passive- than by active-touch.     

Human–Computer Interaction in Face Matching

Automatic facial recognition is becoming increasingly ubiquitous in security contexts such as passport control. Currently, Automated Border Crossing (ABC) systems in the United Kingdom (UK) and the European Union (EU) require supervision from a human operator who validates correct identity judgments and overrules incorrect decisions. As the accuracy of this human–computer interaction remains unknown, this research investigated how human validation is impacted by a priori face‐matching decisions such as those made by automated face recognition software. Observers matched pairs of faces that were already labeled onscreen as depicting the same identity or two different identities. The majority of these labels provided information that was consistent with the stimuli presented, but some were also inconsistent or provided “unresolved” information. Across three experiments, accuracy consistently deteriorated on trials that were inconsistently labeled, indicating that observers’ face‐matching decisions are biased by external information such as that provided by ABCs.     

Robust representations for face recognition: the power of averages

We are able to recognise familiar faces easily across large variations in image quality, though our ability to match unfamiliar faces is strikingly poor. Here we ask how the representation of a face changes as we become familiar with it. We use a simple image-averaging technique to derive abstract representations of known faces. Using Principal Components Analysis, we show that computational systems based on these averages consistently outperform systems based on collections of instances. Furthermore, the quality of the average improves as more images are used to derive it. These simulations are carried out with famous faces, over which we had no control of superficial image characteristics. We then present data from three experiments demonstrating that image averaging can also improve recognition by human observers. Finally, we describe how PCA on image averages appears to preserve identity-specific face information, while eliminating non-diagnostic pictorial information. We therefore suggest that this is a good candidate for a robust face representation.     

How many voices did you hear? Natural variability disrupts identity perception from unfamiliar voices

Our voices sound different depending on the context (laughing vs. talking to a child vs. giving a speech), making within‐person variability an inherent feature of human voices. When perceiving speaker identities, listeners therefore need to not only ‘tell people apart’ (perceiving exemplars from two different speakers as separate identities) but also ‘tell people together’ (perceiving different exemplars from the same speaker as a single identity). In the current study, we investigated how such natural within‐person variability affects voice identity perception. Using voices from a popular TV show, listeners, who were either familiar or unfamiliar with this show, sorted naturally varying voice clips from two speakers into clusters to represent perceived identities. Across three independent participant samples, unfamiliar listeners perceived more identities than familiar listeners and frequently mistook exemplars from the same speaker to be different identities. These findings point towards a selective failure in ‘telling people together’. Our study highlights within‐person variability as a key feature of voices that has striking effects on (unfamiliar) voice identity perception. Our findings not only open up a new line of enquiry in the field of voice perception but also call for a re‐evaluation of theoretical models to account for natural variability during identity perception.     

Image-size disparity reduces difference detection in face matching

Face matching is the act of deciding whether two facial images depict the same person or different people. The real-world face-matching task of checking photo IDs typically occurs under conditions of image-size disparity: A small picture is compared with a life-size face. We examined the effect of image-size disparity on face-matching accuracy. In three experiments, subjects were presented with pairs of equivalently or disparately sized images that depicted the same person or different people. Subjects made same/different judgments and, in two experiments, also reported confidence. Difference detection was significantly poorer given disparate (versus equivalent) image size. Confidence was significantly higher when responses were correct versus incorrect. These findings held whether viewing and decision time was unlimited or limited. Our results raise the practical concern that image-size disparity may undermine difference detection in ID checking, while also indicating that people have some insight into the accuracy of their face-matching judgments.     

TMS to the “occipital face area” affects recognition but not categorization of faces

The human cortical system for face perception is comprised of a network of connected regions including the middle fusiform gyrus (“fusiform face area” or FFA), the inferior occipital cortex (“occipital face area” or OFA), and the superior temporal sulcus. The traditional hierarchical feedforward model of visual processing suggests information flows from early visual cortex to the OFA for initial face feature analysis to higher order regions including the FFA for identity recognition. However, patient data suggest an alternative model. Patients with acquired prosopagnosia, an inability to visually recognize faces, have been documented with lesions to the OFA but who nevertheless show face-selective activation in the FFA. Moreover, their ability to categorize faces remains intact. This suggests that the FFA is not solely responsible for face recognition and the network is not strictly hierarchical, but may be organized in a reverse hierarchical fashion. We used transcranial magnetic stimulation (TMS) to temporarily disrupt processing in the OFA in neurologically-intact individuals and found participants’ ability to categorize intact versus scrambled faces was unaffected, however face identity discrimination was significantly impaired. This suggests that face categorization but not recognition can occur without the “earlier” OFA being online and indicates that “lower level” face category processing may be assumed by other intact face network regions such as the FFA. These results are consistent with the patient data and support a non-hierarchical, global-to-local model with re-entrant connections between the OFA and other face processing areas.     

Adaptation and the perception of facial age

We examined how the perceived age of adult faces is affected by adaptation to younger or older adult faces. Observers viewed images of a synthetic male face simulating ageing over a modelled range from 15 to 65 years. Age was varied by changing shape cues or textural cues. Age level was varied in a staircase to find the observer's subjective category boundary between “old” and “young”. These boundaries were strongly biased by adaptation to the young or old face, with significant aftereffects induced by either shape or textural cues. A further experiment demonstrated comparable aftereffects for photorealistic images of average older or younger adult faces, and found that aftereffects showed some selectivity for a change in gender but also strongly transferred across gender. This transfer shows that adaptation can adjust to the attribute of age somewhat independently of other facial attributes. These findings suggest that perceived age, like many other natural facial dimensions, is highly susceptible to adaptation, and that this adaptation can be carried by both the structural and textural changes that normally accompany facial ageing.