Visual speech information for face recognition

Two experiments test whether isolated visible speech movements can be used for face matching. Visible speech information was isolated with a point-light methodology. Participants were asked to match articulating point-light faces to a fully illuminated articulating face in an XAB task. The first experiment tested single-frame static face stimuli as a control. The results revealed that the participants were significantly better at matching the dynamic face stimuli than the static ones. Experiment 2 tested whether the observed dynamic advantage was based on the movement itself or on the fact that the dynamic stimuli consisted of many more static and ordered frames. For this purpose, frame rate was reduced, and the frames were shown in a random order, a correct order with incorrect relative timing, or a correct order with correct relative timing. The results revealed better matching performance with the correctly ordered and timed frame stimuli, suggesting that matches were based on the actual movement itself. These findings suggest that speaker-specific visible articulatory style can provide information for face matching.     

Hearing a face: cross-modal speaker matching using isolated visible speech

An experiment was performed to test whether cross-modal speaker matches could be made using isolated visible speech movement information. Visible speech movements were isolated using a point-light technique. In five conditions, subjects were asked to match a voice to one of two (unimodal) speaking point-light faces on the basis of speaker identity. Two of these conditions were designed to maintain the idiosyncratic speech dynamics of the speakers, whereas three of the conditions deleted or distorted the dynamics in various ways. Some of these conditions also equated video frames across dynamically correct and distorted movements. The results revealed generally better matching performance in the conditions that maintained the correct speech dynamics than in those conditions that did not, despite containing exactly the same video frames. The results suggest that visible speech movements themselves can support cross-modal speaker matching.     

Visual and audiovisual speech perception with color and gray-scale facial images

Research has shown that auditory speech recognition is influenced by the appearance of a talker's face, but the actual nature of this visual information has yet to be established. Here, we report three experiments that investigated visual and audiovisual speech recognition using color, gray-scale, and point-light talking faces (which allowed comparison with the influence of isolated kinematic information). Auditory and visual forms of the syllables /ba/, /bi/, /ga/, /gi/, /va/, and /vi/ were used to produce auditory, visual, congruent, and incongruent audiovisual speech stimuli. Visual speech identification and visual influences on identifying the auditory components of congruent and incongruent audiovisual speech were identical for color and gray-scale faces and were much greater than for point-light faces. These results indicate that luminance, rather than color, underlies visual and audiovisual speech perception and that this information is more than the kinematic information provided by point-light faces. Implications for processing visual and audiovisual speech are discussed.     

Prior knowledge about display inversion in biological motion perception

Display inversion severely impedes veridical perception of point-light biological motion (Pavlova and Sokolov, 2000 Perception & Psychophysics 62 889-899; Sumi, 1984 Perception 13 283-286). Here, by using a spontaneous-recognition paradigm, we ask whether prior information about display orientation improves biological motion perception. Participants were shown a set of 180 degrees inverted point-light stimuli depicting a human walker and quadrupeds (dogs). In experiment 1, one group of observers was not aware of the orientation of stimuli, whereas the other group was told beforehand that stimuli will be presented upside down. In experiment 2, independent groups of participants informed about stimulus orientation saw the same set of stimuli, in each of which either a moving or a static background line was inserted. The findings indicate that information about display inversion is insufficient for reliable recognition of inverted point-light biological motion. Instead, prior information facilitates display recognition only when it is complemented by additional contextual elements. It appears that visual impressions from inverted point-light stimuli remain impenetrable with respect to one's knowledge about display orientation. The origins of orientation specificity in biological motion perception are discussed in relation to the recent neuroimaging data obtained with point-light stimuli and fragmented Mooney faces.     

Perception of elliptic biological motion

We tested the ability of the mature visual system for discrimination between types of elliptic biological motion on the basis of event kinematics. Healthy adult volunteers were presented with point-light displays depicting elliptic motion when only a single dot, a moving point-light arm, or a whole point-light human figure was visible. The displays were created in accordance with the two-thirds power kinematic law (natural motion), whereas the control displays violated this principle (unnatural motion). On each trial, participants judged whether the display represented natural or unnatural motion. The findings indicate that adults are highly sensitive to violation of the two-thirds power kinematic law. Notably, participants can easily discriminate between natural and unnatural motions without recognising the stimuli, which suggests that people implicitly use kinematic information. Most intriguing, event recognition seems to diminish the capacity to judge whether event kinematics is unnatural. We discuss possible ways for a cross-talk between perception and production of biological movement, and the brain mechanisms involved in biological motion processing.     

It's not all in the face: reduced face visibility does not modulate social segmentation

Humans rely on social information to parse environmental content into social and nonsocial events. Here, we assessed if information conveyed by faces was necessary for this process. We asked participants to view a video clip depicting a social interaction and mark social and nonsocial events while actors’ faces were either visible or blurred. Keypress and eye-movement data were collected. Participants consistently identified social and nonsocial event boundaries independently of face availability, with greater agreement and less variability in keypresses for social relative to nonsocial events. Eye-tracking revealed that participants attended more to actors’ faces when they were visible and more to bodies when faces were blurred. Thus, face information is not necessary for social segmentation, which appears to be a flexible process that depends on a range of information conveyed by both faces and bodies.     

Child and Adult Sensitivity to Gender Information in Patterns of Facial Motion

Sixty 5-year-olds and 120 adults participated in research that examined the development of sensitivity to gender information in patterns of facial motion. Subjects were asked to identify the gender of static or dynamic versions of point-light stimulus faces. The dynamic facial displays were filmed either while the stimulus recited the alphabet or while they were engaged in an interaction. Although adults' levels of identification accuracy were greater than those obtained by children, both age groups were able to identify the gender of dynamic facial displays at greater than chance levels. However, adults were able to identify the gender of both reciting and interacting faces, whereas children could only discriminate sex at greater than chance levels when observing interacting faces.     

Perceiving human locomotion: priming effects in direction discrimination

During the perception of biological motion, the available stimulus information is confined to a small number of lights attached to the major joints of a moving actor. Despite this drastic impoverishment of the stimulus, the human visual apparatus organizes the swarm of moving dots in a vivid percept of a human figure. In addition, observers effortlessly identify the action the figure is involved in. After a historical introduction and a short walk through the literature, data from a priming experiment are presented. In a serial two-choice reaction-time task, participants were presented with a point-light walker, facing either to the right or to the left and walking either forward or backward on a treadmill. Subjects had to identify the direction of articulatory movements. Reliable priming effects were established in consecutive trials, but these effects were tempered by the relation between priming and primed walker. The reaction time to a walker was shorter when the walker in the preceding trial moved in the same direction and was facing in the same direction. The findings are discussed in relation to recent data from neuropsychological case studies, neuroimaging, and single-cell recording.     

Human biological and nonbiological point-light movements: Creation and validation of the dataset

Human action perception is so powerful that people can identify movement efficiently in the absence of pictorial information, such as in point-light displays. Interest is growing in this type of stimulus for research in neuroscience. This interest stems from the advantage of separating the component of pure human action kinematics from other pictorial information, such as facial expression and muscle contraction. Although several groups have previously developed datasets of human point-light actions, due to the lack of datasets composed of daily actions with short durations, we developed 20 biological and 40 control (scrambled) point-light movements by using the technique of recording people wearing reflector patches. The videos are about 1 s long. Subsequently, we performed a judgment task in which 100 participants (50 male and 50 female) evaluated each video according to three categories: human action resemblance, performed action, and gender of actor. We present the mean scores of each evaluation for each video, and further propose a selection of the most suitable videos to be used as human point-light action displays and scrambled point-light displays for control. Finally, we discuss our findings on the gender attributions of the point-light displays.     

Different cues of personality and health from the face and gait of women

Redundant cues for attractiveness in humans have been identified, but the idea of multiple systems displaying different socially-relevant traits has yet to be extensively examined. We compared the accuracy with which observers could identify socially-relevant information of female targets, both from static images of their faces, and from point-light displays of their gait. Perception of extraversion was at chance. However, agreeableness and sociosexuality were more accurately perceived from the face than gait, while physical health showed the opposite pattern. This double dissociation suggests different information can be carried in different modalities. In addition, partial correlation analyses suggested that even when both modalities allowed accurate trait identification, the information content was different. Our results demonstrate that cues of different socially-relevant traits are communicated more effectively through different modalities, and these modality-specific cues contain distinctive information, supporting a "multiple messages" hypothesis.     

Biological motion cues trigger reflexive attentional orienting

The human visual system is extremely sensitive to biological signals around us. In the current study, we demonstrate that biological motion walking direction can induce robust reflexive attentional orienting. Following a brief presentation of a central point-light walker walking towards either the left or right direction, observers' performance was significantly better on a target in the walking direction compared with that in the opposite direction even when participants were explicitly told that walking direction was not predictive of target location. Interestingly, the effect disappeared when the walker was shown upside-down. Moreover, the reflexive attentional orienting could be extended to motions of other biological entities but not inanimate objects, and was not due to the viewpoint effect of the point-light figure. Our findings provide strong evidence that biological motion cues can trigger reflexive attentional orienting, and highlight the intrinsic sensitivity of the human visual attention system to biological signals.     

Person identification from biological motion: effects of structural and kinematic cues

Human observers are able to identify a person based on his or her gait. However, little is known about the underlying mechanisms and the kind of information used to accomplish such a task. In this study, participants learned to discriminate seven male walkers shown as point-light displays from frontal, half-profile, or profile view. The displays were gradually normalized with respect to size, shape, and walking frequency, and identification performance was measured. All observers quickly learned to discriminate the walkers, but there was an overall advantage in favor of the frontal view. No effect of size normalization was found, but performance deteriorated when shape or walking frequency was normalized. Presenting the walkers from novel viewpoints resulted in a further decrease in performance. However, even after applying all normalization steps and rotating the walker by 90 degrees, recognition performance was still nearly three times higher than chance level.     

Recognition of biological motion from blurred natural scenes

Biological-motion perception can be regarded as a template-matching process. We are concerned with the visual cues in this template. Biological-motion perception is usually studied with point-light displays similar to the point-light displays invented by Johansson (1973 Perception and Psychophysics 14 201 - 211). These stimuli are in some ways abstract. In order to use more natural stimuli, we recorded movies of different actions in natural scenes. By blurring the scenes we modified the visual cues, particularly the local form and motion information. Observers were asked to identify the action portrayed. Our results demonstrate that templates for biological-motion recognition combine global form and motion cues. Reductions of local form and local motion information by blurring can be compensated by global form change and global motion. Local motion information is also used for segmentation.     

The effect of grammatical gender on object categorization

In 3 experiments, we investigated the effect of grammatical gender on object categorization. Participants were asked to judge whether 2 objects, whose names did or did not share grammatical gender, belonged to the same semantic category by pressing a key. Monolingual speakers of English (Experiment 1), Italian (Experiments 1 and 2), and Spanish (Experiments 2 and 3) were tested in their native language. Italian and Spanish participants responded faster to pairs of stimuli sharing the same gender, whereas no difference was observed for English participants. In Experiment 2, the pictures were chosen in such a way that the grammatical gender of the names was opposite in Italian and Spanish. Therefore, the same pair of stimuli gave rise to different patterns depending on the gender congruency of the names in the languages. In Experiment 3, Spanish speakers performed the same task under an articulatory suppression condition, showing no grammatical gender effect. The locus where meaning and gender interact can be located at the level of the lexical representation that specifies syntactic information: Nouns sharing the same grammatical gender activate each other, thus facilitating their processing and speeding up responses, either to semantically related pairs or to semantically unrelated pairs.     

Schematic knowledge changes what judgments of learning predict in a source memory task

Source monitoring can be influenced by information that is external to the study context, such as beliefs and general knowledge (Johnson, Hashtroudi, & Lindsay, 1993). We investigated the extent to which metamnemonic judgments predict memory for items and sources when schematic information about the sources is or is not provided at encoding. Participants made judgments of learning (JOLs) to statements presented by two speakers and were informed of the occupation of each speaker either before or after the encoding session. Replicating earlier work, prior knowledge decreased participants’ tendency to erroneously attribute statements to schematically consistent but episodically incorrect speakers. The origin of this effect can be understood by examining the relationship between JOLs and performance: JOLs were equally predictive of item and source memory in the absence of prior knowledge, but were exclusively predictive of source memory when participants knew of the relationship between speakers and statements during study. Background knowledge determines the information that people solicit in service of metamnemonic judgments, suggesting that these judgments reflect control processes during encoding that reduce schematic errors.     

Comparing solid-body with point-light animations

The movement of faces provides useful information for a variety of tasks and is now an active area of research. We compare here two ways of presenting face motion in experiments: as solid-body animations and as point-light displays. In the first experiment solid-body and point-light animations, based on the same motion-captured marker data, produced similar levels of performance on a sex-judgment task. The trend was for an advantage for the point-light displays, probably in part because of residual spatial cues available in such stimuli. In the second experiment we compared spatially normalised point-light displays of marker data with solid-body animations and pseudorandom point-light animations. Performance with solid-body animations and normalised point-light displays was similar and above chance, while performance with the pseudorandom point-light stimuli was not above chance. We conclude that both relatively few well-placed points and solid-body animations provide useful information about facial motion, but that a greater number of randomly placed points does not support above-chance performance. Solid-body animations have the methodological advantages of reducing the importance of marker placement and are more effective in isolating motion information, even if they are subsequently rendered as point-light displays.     

Suprasegmental information affects processing of talking faces at birth

From birth, newborns show a preference for faces talking a native language compared to silent faces. The present study addresses two questions that remained unanswered by previous research: (a) Does the familiarity with the language play a role in this process and (b) Are all the linguistic and paralinguistic cues necessary in this case? Experiment 1 extended newborns’ preference for native speakers to non-native ones. Given that fetuses and newborns are sensitive to the prosodic characteristics of speech, Experiments 2 and 3 presented faces talking native and nonnative languages with the speech stream being low-pass filtered. Results showed that newborns preferred looking at a person who talked to them even when only the prosodic cues were provided for both languages. Nonetheless, a familiarity preference for the previously talking face is observed in the “normal speech” condition (i.e., Experiment 1) and a novelty preference in the “filtered speech” condition (Experiments 2 and 3). This asymmetry reveals that newborns process these two types of stimuli differently and that they may already be sensitive to a mismatch between the articulatory movements of the face and the corresponding speech sounds.     

I know your face but not where I saw you: Context memory is impaired for other-race faces

People are more likely to falsely identify a face of another race than a face of their own race. When witnesses make identifications, they often need to remember where they have previously encountered a face. Failure to remember the context of an encounter can result in unconscious transference and lead to misidentifications. Forty-five White participants were shown White and Black faces, each presented on one of five backgrounds. The participants had to identify these faces in an old/new recognition test. If participants stated that they had seen a face, they had to identify the context in which the face had originally appeared. Participants made more context errors with Black faces than with White faces. This shows that the own-race bias extends to context memory.     

面孔知觉中特征、结构和整体加工策略的眼动研究

考察面孔知觉中特征加工、结构加工和整体加工三种不同加工方式对应的眼动模式。实验1中, 将以特征信息为主的错乱面孔和以结构信息为主的模糊面孔作为线索刺激, 引发对测试面孔的特征加工和结构加工, 眼动分析表明：特征加工表现为对面孔各特征内更长的凝视时间, 结构加工表现为对面孔各特征间高频的眼跳。实验2采用相同的研究范式, 将完整面孔、轻度错乱面孔和低水平模糊面孔作为线索刺激, 引发对测试面孔除特征加工和结构加工外的另一种加工方式-- 整体加工, 表现为注视点更多地落在测试面孔中央区的鼻子部位以扩大注视范围, 进而把握整张面孔信息。本研究揭示了三种不同面孔加工方式眼动模式的差异。     

The effect of familiarity on face adaptation

Face aftereffects can provide information on how faces are stored by the human visual system (eg Leopold et al, 2001 Nature Neuroscience 4 89-94), but few studies have used robustly represented (highly familiar) faces. In this study we investigated the influence of facial familiarity on adaptation effects. Participants were adapted to a series of distorted faces (their own face, a famous face, or an unfamiliar face). In experiment 1, figural aftereffects were significantly smaller when participants were adapted to their own face than when they were adapted to the other faces (ie their own face appeared significantly less distorted than a famous or unfamiliar face). Experiment 2 showed that this 'own-face' effect did not occur when the same faces were used as adaptation stimuli for participants who were unfamiliar with them. Experiment 3 replicated experiment 1, but included a pre-adaptation baseline. The results highlight the importance of considering facial familiarity when conducting research on face aftereffects.