Finding an unfamiliar face in a line-up: Viewing multiple images of the target is beneficial on target-present trials but costly on target-absent trials

When viewing unfamiliar faces, photographs of the same person often are perceived as belonging to different people and photographs of different people as belonging to the same person. Identity matching of unfamiliar faces is especially challenging when the photographs are of a person whose ethnicity differs from that of the observer. In contrast, matching is trivial when viewing familiar faces, regardless of race. Viewing multiple images of an own-race target identity improves accuracy on a line-up task when the target is known to be present (Dowsett et al., 2016, Q J Exp Psychol, 69, 1), suggesting that exposure to within-person variability in appearance is key to face learning. Across three experiments, we show that viewing multiple images of a target identity also improves accuracy for other-race faces on target-present trials. However, viewing multiple images decreases accuracy (i.e., increases false alarms) on target-absent trials for both own- and other-race faces. We discuss the implications of our findings for models of face recognition and for forensic settings.     

The flip side of the other‐race coin: They all look different to me

Poorer recognition of other‐race faces than own‐races faces has been attributed to a problem of discrimination (i.e., telling faces apart). The conclusion that ‘they all look the same to me' is based on studies measuring the perception/memory of highly controlled stimuli, typically involving only one or two images of each identity. We hypothesized that such studies underestimate the challenge involved in recognizing other‐race faces because in the real world, an individual's appearance varies in a number of ways (e.g., lighting, expression, hairstyle), reducing the utility of relying on pictorial cues to identity. In two experiments, Caucasian and East Asian participants completed a perceptual sorting task in which they were asked to sort 40 photographs of two unfamiliar identities into piles such that each pile contained all photographs of a single identity. Participants perceived more identities when sorting other‐race faces than own‐race faces, both when sorting celebrity (Experiment 1) and non‐celebrity (Experiment 2) faces, suggesting that in the real world, ‘they all look different to me'. We discuss these results in the light of models in which each identity is represented as a region in a multidimensional face space; we argue that this region is smaller for other‐race than own‐race faces.     

Evidence for holistic processing of faces viewed as photographic negatives

Inversion and photographic negation both impair face recognition. Inversion seems to disrupt processing of the spatial relationship between facial features ('relational' processing) which normally occurs with upright faces and which facilitates their recognition. It remains unclear why negation affects recognition. To find out if negation impairs relational processing, we investigated whether negative faces are subject to the 'chimeric-face effect'. Recognition of the top half of a composite face (constructed from top and bottom halves of different faces) is difficult when the face is upright, but not when it is inverted. To perform this task successfully, the bottom half of the face has to be disregarded, but the relational processing which normally occurs with upright faces makes this difficult. Inversion reduces relational processing and thus facilitates performance on this particular task. In our experiments, subjects saw pairs of chimeric faces and had to decide whether or not the top halves were identical. On half the trials the two chimeras had identical tops; on the remaining trials the top halves were different. (The bottom halves were always different.) All permutations of orientation (upright or inverted) and luminance (normal or negative) were used. In experiment 1, each pair of 'identical' top halves were the same in all respects. Experiment 2 used differently oriented views of the same person, to preclude matches being based on incidental features of the images rather than the faces displayed within them. In both experiments, similar chimeric-face effects were obtained with both positive and negative faces, implying that negative faces evoke some form of relational processing. It is argued that there may be more than one kind of relational processing involved in face recognition: the 'chimeric-face effect' may reflect an initial 'holistic' processing which binds facial features into a 'Gestalt', rather than being a demonstration of the configurational processing involved in individual recognition.     

Robust representations of individual faces in chimpanzees (<Emphasis Type="Italic">Pan troglodytes</Emphasis>) but not monkeys (<Emphasis Type="Italic">Macaca mulatta</Emphasis>)

Being able to recognize the faces of our friends and family members no matter where we see them represents a substantial challenge for the visual system because the retinal image of a face can be degraded by both changes in the person (age, expression, pose, hairstyle, etc.) and changes in the viewing conditions (direction and degree of illumination). Yet most of us are able to recognize familiar people effortlessly. A popular theory for how face recognition is achieved has argued that the brain stabilizes facial appearance by building average representations that enhance diagnostic features that reliably vary between people while diluting features that vary between instances of the same person. This explains why people find it easier to recognize average images of people, created by averaging multiple images of the same person together, than single instances (i.e. photographs). Although this theory is gathering momentum in the psychological and computer sciences, there is no evidence of whether this mechanism represents a unique specialization for individual recognition in humans. Here we tested two species, chimpanzees (Pan troglodytes) and rhesus monkeys (Macaca mulatta), to determine whether average images of different familiar individuals were easier to discriminate than photographs of familiar individuals. Using a two-alternative forced-choice, match-to-sample procedure, we report a behaviour response profile that suggests chimpanzees encode the faces of conspecifics differently than rhesus monkeys and in a manner similar to humans.     

Ensemble coding of facial identity is not refined by experience: Evidence from other-race and inverted faces

The ability to recognize identity despite within-person variability in appearance is likely a face-specific skill and shaped by experience. Ensemble coding – the automatic extraction of the average of a stimulus array – has been proposed as a mechanism underlying face learning (allowing one to recognize novel instances of a newly learned face). We investigated whether ensemble encoding, like face learning and recognition, is refined by experience by testing participants with upright own-race faces and two categories of faces with which they lacked experience: other-race faces (Experiment 1) and inverted faces (Experiment 2). Participants viewed four images of an unfamiliar identity and then were asked whether a test image of that same identity had been in the study array. Each test image was a matching exemplar (from the array), matching average (the average of the images in the array), non-matching exemplar (a novel image of the same identity), or non-matching average (an average of four different images of the same identity). Adults showed comparable ensemble coding for all three categories (i.e., reported that matching averages had been present more than non-matching averages), providing evidence that this early stage of face learning is not shaped by face-specific experience.     

Recognising faces seen alone or with others: when two heads are worse than one

There is a large body of work investigating face identification, but most of this addresses recognition of a single person. Here, we examine how recognition is affected by the presence of a second face. In Experiments 1 and 2, we demonstrate that memory for an unfamiliar face is severely reduced if it is seen alongside a second person. Sequential presentation of two target faces further reduces accuracy. In Experiments 3 and 4, we demonstrate the same disadvantage for two‐face targets in a matching task, where subjects have no time limits or memory requirement. In matching, the damaging effect of a second face is greatest when the targets are placed close together. Furthermore, there is a general advantage for faces presented to the left. We suggest that it may not be possible to extrapolate results from single‐face studies to experiments (or realistic situations) involving more than one person. Copyright © 2006 John Wiley & Sons, Ltd.     

Social Trait Information in Deep Convolutional Neural Networks Trained for Face Identification

Faces provide information about a person's identity, as well as their sex, age, and ethnicity. People also infer social and personality traits from the face — judgments that can have important societal and personal consequences. In recent years, deep convolutional neural networks (DCNNs) have proven adept at representing the identity of a face from images that vary widely in viewpoint, illumination, expression, and appearance. These algorithms are modeled on the primate visual cortex and consist of multiple processing layers of simulated neurons. Here, we examined whether a DCNN trained for face identification also retains a representation of the information in faces that supports social‐trait inferences. Participants rated male and female faces on a diverse set of 18 personality traits. Linear classifiers were trained with cross validation to predict human‐assigned trait ratings from the 512 dimensional representations of faces that emerged at the top‐layer of a DCNN trained for face identification. The network was trained with 494,414 images of 10,575 identities and consisted of seven layers and 19.8 million parameters. The top‐level DCNN features produced by the network predicted the human‐assigned social trait profiles with good accuracy. Human‐assigned ratings for the individual traits were also predicted accurately. We conclude that the face representations that emerge from DCNNs retain facial information that goes beyond the strict limits of their training.     

Matching Face Images Taken on the Same Day or Months Apart: the Limitations of Photo ID

It is well‐established that matching images of unfamiliar faces is rather error prone. However, there is an important mismatch between face matching in laboratory and realistic settings. All of the currently available face‐matching databases were designed to establish the baseline level of unfamiliar face perception. Therefore, target and test images for each face identity have been taken on the same day, minimizing within‐face variations. In realistic settings, on the other hand, faces do vary, even day to day. This study examined the proficiency of matching images of unfamiliar faces, which were taken on the same day or months apart. In two experiments, same‐day images were matched substantially more accurately and faster than different‐date photographs using the standard 1‐in‐10 and pairwise face‐matching tasks. This suggests that experimental studies on face matching underestimate its difficulty in real‐world situations. Photographs of unfamiliar faces seem to be unreliable proofs of identity, especially if the ID documents do not use very recent images of the holders. Copyright © 2013 John Wiley & Sons, Ltd.     

The Effect of Feedback on Face‐Matching Accuracy

In face matching, observers have to decide if two photographs depict the same person or different people. This is a remarkably difficult task, so the current study investigated whether it can be improved when observers receive feedback for their performance. In five experiments, observers' initial matching performance was recorded before feedback for their accuracy was administered across three blocks. Improvements were then assessed with faces that had been seen previously with or without feedback and with completely new, previously unseen faces. In all experiments, feedback failed to improve face‐matching accuracy. However, trial‐by‐trial feedback helped to maintain accuracy at baseline level after feedback was withdrawn again, even with new faces (Experiments 1–3). By contrast, when no feedback was given throughout the experiment (Experiments 1–3) or when outcome feedback was administered at the end of blocks (Experiments 4 and 5), a continuous decline in matching accuracy was found, whereby observers found it increasingly difficult to tell different facial identities apart. A sixth experiment showed that this decline in accuracy continues throughout when the matching task is prolonged substantially. Together, these findings indicate that observers find it increasingly difficult to differentiate faces in matching tasks over time, but trial‐by‐trial feedback can help to maintain accuracy. The theoretical and practical implications of these findings are discussed. Copyright © 2013 John Wiley & Sons, Ltd.     

Hearing facial identities

While audiovisual integration is well known in speech perception, faces and speech are also informative with respect to speaker recognition. To date, audiovisual integration in the recognition of familiar people has never been demonstrated. Here we show systematic benefits and costs for the recognition of familiar voices when these are combined with time-synchronized articulating faces, of corresponding or noncorresponding speaker identity, respectively. While these effects were strong for familiar voices, they were smaller or nonsignificant for unfamiliar voices, suggesting that the effects depend on the previous creation of a multimodal representation of a person's identity. Moreover, the effects were reduced or eliminated when voices were combined with the same faces presented as static pictures, demonstrating that the effects do not simply reflect the use of facial identity as a “cue” for voice recognition. This is the first direct evidence for audiovisual integration in person recognition.     

Hearing facial identities

While audiovisual integration is well known in speech perception, faces and speech are also informative with respect to speaker recognition. To date, audiovisual integration in the recognition of familiar people has never been demonstrated. Here we show systematic benefits and costs for the recognition of familiar voices when these are combined with time-synchronized articulating faces, of corresponding or noncorresponding speaker identity, respectively. While these effects were strong for familiar voices, they were smaller or nonsignificant for unfamiliar voices, suggesting that the effects depend on the previous creation of a multimodal representation of a person's identity. Moreover, the effects were reduced or eliminated when voices were combined with the same faces presented as static pictures, demonstrating that the effects do not simply reflect the use of facial identity as a “cue” for voice recognition. This is the first direct evidence for audiovisual integration in person recognition.     

Attention to internal face features in unfamiliar face matching

Accurate matching of unfamiliar faces is vital in security and forensic applications, yet previous research has suggested that humans often perform poorly when matching unfamiliar faces. Hairstyle and facial hair can strongly influence unfamiliar face matching but are potentially unreliable cues. This study investigated whether increased attention to the more stable internal face features of eyes, nose, and mouth was associated with more accurate face‐matching performance. Forty‐three first‐year psychology students decided whether two simultaneously presented faces were of the same person or not. The faces were displayed for either 2 or 6 seconds, and had either similar or dissimilar hairstyles. The level of attention to internal features was measured by the proportion of fixation time spent on the internal face features and the sensitivity of discrimination to changes in external feature similarity. Increased attention to internal features was associated with increased discrimination in the 2‐second display‐time condition, but no significant relationship was found in the 6‐second condition. Individual differences in eye‐movements were highly stable across the experimental conditions.     

Variability in photos of the same face

Psychological studies of face recognition have typically ignored within-person variation in appearance, instead emphasising differences between individuals. Studies typically assume that a photograph adequately captures a person’s appearance, and for that reason most studies use just one, or a small number of photos per person. Here we show that photographs are not consistent indicators of facial appearance because they are blind to within-person variability. Crucially, this within-person variability is often very large compared to the differences between people. To investigate variability in photos of the same face, we collected images from the internet to sample a realistic range for each individual. In Experiments 1 and 2, unfamiliar viewers perceived images of the same person as being different individuals, while familiar viewers perfectly identified the same photos. In Experiment 3, multiple photographs of any individual formed a continuum of good to bad likeness, which was highly sensitive to familiarity. Finally, in Experiment 4, we found that within-person variability exceeded between-person variability in attractiveness. These observations are critical to our understanding of face processing, because they suggest that a key component of face processing has been ignored. As well as its theoretical significance, this scale of variability has important practical implications. For example, our findings suggest that face photographs are unsuitable as proof of identity.     

CCTV on trial: Matching video images with the defendant in the dock

The experiments reported in this paper investigated simultaneous identity matching of unfamiliar people physically present in person with moving video images typical of that captured by closed circuit television (CCTV). This simulates the decision faced by a jury in court when the identity of somebody caught on CCTV is disputed. Namely, ‘is the defendant in the dock the person depicted in video’? In Experiment 1, the videos depicted medium‐range views of a number of actor ‘culprits’. Experiment 2 used similar quality images taken a year previously, some of which showed the culprits in disguise. Experiment 3 utilised high‐quality close‐up video images. It was consistently found that in both culprit‐present and culprit‐absent videos and in optimal conditions, matching the identity of a person in video can be highly susceptible to error. Copyright © 2008 John Wiley & Sons, Ltd.     

The sibling familiarity effect: Is within-person facial variability shared across siblings?

Humans are experts at familiar face recognition, but poor at unfamiliar face recognition. Familiarity is created when a face is encountered across varied conditions, but the way in which a person’s appearance varies is identity-specific, so familiarity with one identity does not benefit recognition of other individuals. However, the faces of biological siblings share structural similarities, so we explored whether the benefits of familiarity are shared across siblings. Results show that familiarity with one half of a sibling pair improves kin detection (experiment 1), and that unfamiliar face matching is more accurate when targets are the siblings of familiar versus unfamiliar individuals (experiment 2). PCA applied to facial images of celebrities and their siblings demonstrates that faces are generally better reconstructed in the principal components of a same-sex sibling than those of an unrelated individual. When we encounter the unfamiliar sibling of someone we already know, our pre-existing representation of their familiar relation may usefully inform processing of the unfamiliar face. This can benefit both kin detection and identity processing, but the benefits are constrained by the degree to which facial variability is shared.     

Information leakage in the Response Time‐Based Concealed Information Test

The Response Time‐Based Concealed Information Test (RT‐CIT) can reveal when a person recognizes a relevant (probe) item among other, irrelevant items, based on comparatively slow responses to the probe item. For example, if a person is concealing his or her true identity, one can use the suspected identity details as probes, and other, random details as irrelevants. However, in our study, we show that even when participants are merely informed about such probes (i.e., the relevant identity details) before performing the RT‐CIT, their responses will also be slower to these details. Hence, it is more difficult to distinguish such innocent but pre‐informed persons from actually guilty persons. At the same time, we introduce a CIT version with familiarity‐related inducer stimuli, but with no targets, that elicits probe‐minus‐irrelevant RT differences only among guilty participants but not among informed innocent participants. Implications for the theory and the application of CITs are discussed.     

Focusing on the bodily self: The influence of endogenous attention on visual body processing

The present study explores whether endogenous attention can modulate body perception. A modified version of the Posner paradigm was used to direct participants' attention toward the appearance of distinct body images, which differed only in detailed idiosyncratic features: one's own and another person's hands. Hand stimuli were preceded by symbolic cues that predicted their identity with high probability, which made it possible to compare the processing of expected (valid) and unexpected (invalid) targets. Results revealed that endogenous attention influenced the processing of participants' own hands by speeding participants' responses to valid in contrast to invalid trials. Crucially, no validity effect was found for the hands of another person. These findings cannot be explained in terms of perceptual familiarity, since an optimization of the processing for both familiar and unfamiliar faces by symbolic cues was observed. In light of these results, it is suggested that participants are able to anticipate particular stimuli within the same perceptual category as long as these stimuli appear to be remarkably distinct to them, which is probably the case for particular faces and their own bodies, in contrast to other people's bodies.     

Me, myself, and I: different recognition rates for three photo-IDs of the same person

In all contemporary societies, photo-identity documents are used routinely for person identification, but this process is surprisingly fallible. Here we show that this problem is not limited to the identification of specific photographs of a person, but transcends three identity cards of the same person with different images. These identity cards varied substantially from each other in how well they could be recognised but identification rates were generally poor. We also present a potential solution to this problem by demonstrating that person identification can be improved when several photographs of the same person are made available.     

Asymmetric dependencies in perceiving identity and emotion: Experiments with morphed faces

We investigated whether an asymmetric relationship between the perception of identity and emo-tional expressions in faces (Schweinberger & Soukup, 1998) may be related to differences in the rela-tive processing speed of identity and expression information. Stimulus faces were morphed across identity within a given emotional expression, or were morphed across emotion within a given identity. In Experiment 1, consistent classifications of these images were demonstrated across a wide range of morphing, with only a relatively narrow category boundary. At the same time, classification reaction times (RTs) reflected the increased perceptual difficulty of the morphed images. In Experiment 2, we investigated the effects of variations in the irrelevant dimension on judgments of faces with respect to a relevant dimension, using a Garner-type speeded classification task. RTs for expression classifica-tions were strongly influenced by irrelevant identity information. In contrast, RTs for identity classifi-cations were unaffected by irrelevant expression information, and this held even for stimuli in which identity was more difficult and slower to discriminate than expression. This suggests that differences in processing speed cannot account for the asymmetric relationship between identity and emotion per-ception. Theoretical accounts proposing independence of identity and emotion perception are dis-cussed in the light of these findings.     

Temporal Phase Pluralism*

Some theories of personal identity allow some variation in what it takes for a person to survive from context to context; and sometimes this is determined by the desires of person‐stages or the practices of communities. This leads to problems for decision making in contexts where what is chosen will affect personal identity. ‘Temporal Phase Pluralism’ solves such problems by allowing that there can be a plurality of persons constituted by a sequence of person stages. This illuminates difficult decision making problems when persons have to choose between different life‐altering choices.