Evaluation and integration of speech and pointing gestures during referential understanding

Two experiments investigated the relative influence of speech and pointing gesture information in the interpretation of referential acts. Children averaging 3 and 5 years of age and adults viewed a videotape containing the independent manipulation of speech and gestural forms of reference. A man instructed the subjects to choose a ball or a doll by vocally labeling the referent and/or pointing to it. A synthetic speech continuum between two alternatives was crossed with the pointing gesture in a factorial design. Based on research in other domains, it was predicted that all age groups would utilize gestural information, although both speech and gestures were predicted to influence children less than adults. The main effects and interactions of speech and gesture in combination with quantitative models of performance showed the following similarities in information processing between preschoolers and adults: (1) referential evaluation of gestures occurs independently of the evaluation of linguistic reference; (2) speech and gesture are continuous, rather than discrete, sources of information; (3) 5-year-olds and adults combine the two types of information in such a way that the least ambiguous source has the most impact on the judgment. Greater discriminability of both speech and gesture information for adults compared to preschoolers indicated small quantitative progressions with development in the ability to extract and utilize referential signals.     

Scores on the State-Trait Anxiety Inventory for children in a longitudinal study of pubertal Spanish youth.

Changes in scores on the State-Trait Anxiety Inventory for Children for a sample of 534 pubertal adolescents were longitudinally studied over a period of 4 years. Biological development and other variables were investigated. For the 310 boys, A-trait scores decreased significantly with age. The 224 girls showed significantly higher scores than boys in all years of the study. No significant relationship was found between anxiety and pubertal maturity. The year-to-year persistence in symptomatology of anxiety ranged between 55.6% and 66.1% for girls and between 48.5% and 53.1% for boys. Girls showed bi- and tri-annual persistence between 31% and 50%, and boys between 13% and 27%. Neuroticism scores formed the best predictor of A-trait scores. Symptomatology of anxiety is not transient during early adolescence, so personality may provide a risk indicator in the development of anxiety.     

Perceiving affect from the voice and the face

This experiment examines how emotion is perceived by using facial and vocal cues of a speaker. Three levels of facial affect were presented using a computer-generated face. Three levels of vocal affect were obtained by recording the voice of a male amateur actor who spoke a semantically neutral word in different simulated emotional states. These two independent variables were presented to subjects in all possible permutations—visual cues alone, vocal cues alone, and visual and vocal cues together—which gave a total set of 15 stimuli. The subjects were asked to judge the emotion of the stimuli in a two-alternative forced choice task (either HAPPY or ANGRY). The results indicate that subjects evaluate and integrate information from both modalities to perceive emotion. The influence of one modality was greater to the extent that the other was ambiguous (neutral). The fuzzy logical model of perception (FLMP) fit the judgments significantly better than an additive model, which weakens theories based on an additive combination of modalities, categorical perception, and influence from only a single modality.     

Perceiving speech from inverted faces

We examined whether the orientation of the face influences speech perception in face-to-face communication. Participants identified auditory syllables, visible syllables, and bimodal syllables presented in an expanded factorial design. The syllables were /ba/, /va/, /δa/, or /da/. The auditory syllables were taken from natural speech whereas the visible syllables were produced by computer animation of a realistic talking face. The animated face was presented either as viewed in normal upright orientation or inverted orientation (180° frontal rotation). The central intent of the study was to determine if an inverted view of the face would change the nature of processing bimodal speech or simply influence the information available in visible speech. The results with both the upright and inverted face views were adequately described by the fuzzy logical model of perception (FLMP). The observed differences in the FLMP’s parameter values corresponding to the visual information indicate that inverting the view of the face influences the amount of visible information but does not change the nature of the information processing in bimodal speech perception     

Models of integration given multiple sources of information

Several models of information integration are developed and analyzed within the context of a proto-typical pattern-recognition task. The central concerns are whether the models prescribe maximally efficient (optimal) integration and to what extent the models are psychologically valid. Evaluation, integration, and decision processes are specified for each model. Important features are whether evaluation is noisy, whether integration follows Bayes's theorem, and whether decision consists of a criterion rule or a relative goodness rule. Simulations of the models and predictions of the results by the same models are carried out to provide a measure of identifiability or the extent to which the models can be distinguished from one another. The models are also contrasted against empirical results from tasks with 2 and 4 response alternatives and with graded responses.     

Perceptual processing in dichotic listening.

Subjects heard two lists of 4 items each presented simultaneously to the two ears at a rate of four pairs of items per sec. A recall cue presented immediately after the test list signalled report of 4 of the 8 items. In recall by spatial location, the cue indicated whether the items on the right ear on left ear should be recalled. In recall by category name, the cue indicated the superset category (e.g., letters or words) of the items to be recalled. Recall by spatial location was not significantly different than recall by category name. This results argues against the idea of a preperceptual auditory storage that holds information along spatial channels for 1 or 2 sec. The final experiment showed that recall by spatial location is significantly better than recall by category name when the report cue is given before, not after, the list presentation. These results show that spatial location can be used to enhance semantic processing and/or memory of 1 of 2 simultaneous items, but only if the relevant location is known at the time of the item presentation.