Facilitating referential communication among children: The listener as teacher

This study was designed to test the hypothesis that systematic variations in listener behavior can have an important influence on both speaker behavior and communicative success among children. In particular, we investigated the idea that systematic variations in listener behavior might have not only within-trial effects on the adequacy of speakers' messages and the accuracy of communication among children, but also cumulative effects on speakers' initial messages across trials. Effects of stimulus complexity were also examined. Pairs of 7- and 9-year-old children participated in a referential communication game, with the younger child serving as speaker and the older one as listener. Half of the listeners were given a plan for effective listening which emphasized the importance of asking questions if the speakers' messages were ambiguous. Replicating earlier findings, the plan manipulation was successful in encouraging listeners to ask questions when necessary. The major result was that listener questions not only had the expected trial-by-trial effect on message adequacy and communicative accuracy, but also showed a cumulative effect on speaker performance. When exposed to systematic listener feedback, speakers improved their initial messages over trials. Stimulus complexity was not a major determinant of performances. These findings suggest that provision of systematic listener feedback may be an effective method for teaching speaker skills to young children.     

Degree of Language Experience Modulates Visual Attention to Visible Speech and Iconic Gestures During Clear and Degraded Speech Comprehension

Visual information conveyed by iconic hand gestures and visible speech can enhance speech comprehension under adverse listening conditions for both native and non‐native listeners. However, how a listener allocates visual attention to these articulators during speech comprehension is unknown. We used eye‐tracking to investigate whether and how native and highly proficient non‐native listeners of Dutch allocated overt eye gaze to visible speech and gestures during clear and degraded speech comprehension. Participants watched video clips of an actress uttering a clear or degraded (6‐band noise‐vocoded) action verb while performing a gesture or not, and were asked to indicate the word they heard in a cued‐recall task. Gestural enhancement was the largest (i.e., a relative reduction in reaction time cost) when speech was degraded for all listeners, but it was stronger for native listeners. Both native and non‐native listeners mostly gazed at the face during comprehension, but non‐native listeners gazed more often at gestures than native listeners. However, only native but not non‐native listeners' gaze allocation to gestures predicted gestural benefit during degraded speech comprehension. We conclude that non‐native listeners might gaze at gesture more as it might be more challenging for non‐native listeners to resolve the degraded auditory cues and couple those cues to phonological information that is conveyed by visible speech. This diminished phonological knowledge might hinder the use of semantic information that is conveyed by gestures for non‐native compared to native listeners. Our results demonstrate that the degree of language experience impacts overt visual attention to visual articulators, resulting in different visual benefits for native versus non‐native listeners.     

Guess who? Children use prosody to infer intended listeners

This study examined the relative influence of prosody and semantic content in children's inferences about intended listeners. Children (n = 72), who ranged in age from 5 to 10 years, heard greetings with prosody and content that was either infant or adult directed and chose the intended listener from amongst an infant or an adult. While content affected all children's choices, the effect of prosody was stronger (at least, for children aged 7–10 years). For conditions in which prosodic cues were suggestive of one listener, and content cues, another, children aged 7–10 years chose the listener according to prosody. In contrast, the youngest age group (5‐ to 6‐year‐olds) chose listeners at chance levels in these incongruent conditions. While prosodic cues were most influential in determining children's choices, their ratings of how certain they felt about their choices indicated that content nonetheless influenced their thinking about the intended listener. Results are the first to show the unique influence of prosody in children's thinking about appropriate speech styles. Findings add to work showing children's ability to use prosody to make inferences about speakers' communicative intentions.     

Investigating joint attention mechanisms through spoken human-robot interaction

Referential gaze during situated language production and comprehension is tightly coupled with the unfolding speech stream ( [Griffin, 2001] , [Meyer et al., 1998] and [Tanenhaus et al., 1995] ). In a shared environment, utterance comprehension may further be facilitated when the listener can exploit the speaker’s focus of (visual) attention to anticipate, ground, and disambiguate spoken references. To investigate the dynamics of such gaze-following and its influence on utterance comprehension in a controlled manner, we use a human–robot interaction setting. Specifically, we hypothesize that referential gaze is interpreted as a cue to the speaker’s referential intentions which facilitates or disrupts reference resolution. Moreover, the use of a dynamic and yet extremely controlled gaze cue enables us to shed light on the simultaneous and incremental integration of the unfolding speech and gaze movement.We report evidence from two eye-tracking experiments in which participants saw videos of a robot looking at and describing objects in a scene. The results reveal a quantified benefit-disruption spectrum of gaze on utterance comprehension and, further, show that gaze is used, even during the initial movement phase, to restrict the spatial domain of potential referents. These findings more broadly suggest that people treat artificial agents similar to human agents and, thus, validate such a setting for further explorations of joint attention mechanisms.     

Social eye gaze modulates processing of speech and co-speech gesture

In human face-to-face communication, language comprehension is a multi-modal, situated activity. However, little is known about how we combine information from different modalities during comprehension, and how perceived communicative intentions, often signaled through visual signals, influence this process. We explored this question by simulating a multi-party communication context in which a speaker alternated her gaze between two recipients. Participants viewed speech-only or speech + gesture object-related messages when being addressed (direct gaze) or unaddressed (gaze averted to other participant). They were then asked to choose which of two object images matched the speaker’s preceding message. Unaddressed recipients responded significantly more slowly than addressees for speech-only utterances. However, perceiving the same speech accompanied by gestures sped unaddressed recipients up to a level identical to that of addressees. That is, when unaddressed recipients’ speech processing suffers, gestures can enhance the comprehension of a speaker’s message. We discuss our findings with respect to two hypotheses attempting to account for how social eye gaze may modulate multi-modal language comprehension.     

Analyzing nonverbal listener responses using parallel recordings of multiple listeners

In this paper, we study nonverbal listener responses on a corpus with multiple parallel recorded listeners. These listeners were meant to believe that they were the sole listener, while in fact there were three persons listening to the same speaker. The speaker could only see one of the listeners. We analyze the impact of the particular setup of the corpus on the behavior and perception of the two types of listeners: the listeners that could be seen by the speaker and the listeners that could not be seen. Furthermore, we compare the nonverbal listening behaviors of these three listeners to each other with regard to timing and form. We correlate these behaviors with behaviors of the speaker, like pauses and whether the speaker is looking at the listeners or not.     

Analyzing nonverbal listener responses using parallel recordings of multiple listeners

In this paper, we study nonverbal listener responses on a corpus with multiple parallel recorded listeners. These listeners were meant to believe that they were the sole listener, while in fact there were three persons listening to the same speaker. The speaker could only see one of the listeners. We analyze the impact of the particular setup of the corpus on the behavior and perception of the two types of listeners: the listeners that could be seen by the speaker and the listeners that could not be seen. Furthermore, we compare the nonverbal listening behaviors of these three listeners to each other with regard to timing and form. We correlate these behaviors with behaviors of the speaker, like pauses and whether the speaker is looking at the listeners or not.     

Listener perceptions along a fluency--disfluency continuum: a phenomenological analysis

The purpose of this study was to conduct a phenomenological analysis (a qualitative research method) of unbiased listeners' perceptions to six speech samples across a fluency-disfluency continuum. A sample of 60 individuals heard only one sample chosen from three levels of fluent or three levels of disfluent speech. Listeners were interviewed following the presentation of the speech sample and their comments were analyzed with respect to the perception of the speaker's communicative effectiveness. Communicative effectiveness was supported by three phenomenological categories: speaker attributes, listener attributes, and story attributes. Five theme clusters further supported these categories: speech production, context, speaker identity, listener comfort, and story comprehension. The results showed that listener perceptions within theme clusters varied across the six speech samples. The results also showed that listeners differentially respond to a broad array of information in the speech signal (not simply fluency or disfluency). These findings support Traunmuller's (1994) modulation theory associated with information that can be obtained from the speech signal. Implications for the treatment of stuttering are also discussed. EDUCATIONAL OBJECTIVES: The reader will learn (1) how listeners may have multiple and varying perceptual experiences depending upon where along a fluency-disfluency continuum a speech sample is heard; (2) how perceptual experiences are influenced by speaker, listener, and story attributes; and (3) how phenomenological analysis may expand our understanding of multifactorial issues associated with stuttering.     

The influence of speaker gaze on listener comprehension: Contrasting visual versus intentional accounts

Previous research has shown that listeners follow speaker gaze to mentioned objects in a shared environment to ground referring expressions, both for human and robot speakers. What is less clear is whether the benefit of speaker gaze is due to the inference of referential intentions (Staudte and Crocker, 2011) or simply the (reflexive) shifts in visual attention. That is, is gaze special in how it affects simultaneous utterance comprehension? In four eye-tracking studies we directly contrast speech-aligned speaker gaze of a virtual agent with a non-gaze visual cue (arrow). Our findings show that both cues similarly direct listeners’ attention and that listeners can benefit in utterance comprehension from both cues. Only when they are similarly precise, however, does this equality extend to incongruent cueing sequences: that is, even when the cue sequence does not match the concurrent sequence of spoken referents can listeners benefit from gaze as well as arrows. The results suggest that listeners are able to learn a counter-predictive mapping of both cues to the sequence of referents. Thus, gaze and arrows can in principle be applied with equal flexibility and efficiency during language comprehension.     

Intimacy in response to direct gaze

Believing that they were participating in a study of encounter groups, male and female subjects were encouraged to deliver a personally revealing monologue to a same-sex listener whose visual behavior had been “programmed” by the experimenter. In the four experimental conditions, designed to explore the role of gaze as a stimulus, as a potential reinforcer, and as a source of feedback, the listener provided (a) continuous direct gaze; (b) direct gaze contingent upon intimate statements; (c) continuous gaze aversion; or (d) gaze aversion contingent upon intimate statements. The intimacy of the speaker's monologue was rated by a “blind” observer, the listener, and the speaker himself. Direct gaze, whether constant or contingent, appeared to promote intimacy between females and reticence between males, while gaze avoidance had the opposite effects. The only raters who disagreed with this assessment were the male speakers, who felt that they had been most intimate in the two direct gaze conditions. Positive feelings including liking for the listener and task satisfaction also were associated both with self-revelation and with direct gaze among females but not among males.     

The ecology of self-monitoring effects on memory of verbal productions: Does speaking to someone make a difference?

Experiments involving verbal self-monitoring show that memory for spoken words varies with types of sensory feedback: memory is better when words are spoken aloud than when they are lip-synched or covertly produced. Such effects can be explained by the Central Monitoring Theory (CMT) via a process that matches a forward model reflecting expected sensory effects of practiced forms and sensory information during speech. But CMT oversees factors of shared attention as achieved by speaker–listener gaze, and implies that sensory feedback may not affect the learning of unpracticed forms (non-words). These aspects of CMT were examined in two experiments of self-monitoring focusing on oro-sensory feedback. In Experiment 1 we show that varying feedback creates differential effects on memory for spoken words and that speaker–listener gaze alters these effects. Using non-words, Experiment 2 shows the absence of differential feedback effects. The results confirm CMT but suggest the need to refine the theory in terms of processes that mediate attention.     

Communicative Context Affects Use of Referential Prosody

The current study assessed the extent to which the use of referential prosody varies with communicative demand. Speaker–listener dyads completed a referential communication task during which speakers attempted to indicate one of two color swatches (one bright, one dark) to listeners. Speakers' bright sentences were reliably higher pitched than dark sentences for ambiguous (e.g., bright red versus dark red) but not unambiguous (e.g., bright red versus dark purple) trials, suggesting that speakers produced meaningful acoustic cues to brightness when the accompanying linguistic content was underspecified (e.g., “Can you get the red one?”). Listening partners reliably chose the correct corresponding swatch for ambiguous trials when lexical information was insufficient to identify the target, suggesting that listeners recruited prosody to resolve lexical ambiguity. Prosody can thus be conceptualized as a type of vocal gesture that can be recruited to resolve referential ambiguity when there is communicative demand to do so.     

Listener response to stuttering modification techniques

In two investigations, nonprofessional listeners used a 25-item bi-polar adjective scale to evaluate an adult male speaker who stuttered. In investigation one, 24 listeners evaluated the speaker during a Stuttering Only and a Stuttering + Cancellation condition. In investigation two, 50 listeners evaluated the speaker in a Stuttering Only and a Stuttering + Pullout condition. Listeners in the first investigation did not assign significantly different ratings to the Stuttering Only and Stuttering + Cancellation conditions. In the second investigation listeners rated the Stuttering Only condition more positively than the Stuttering + Pullout condition (p < 0.05). Analysis of listener response to four open-ended questions indicated significantly more positive reaction to the Stuttering Only condition than either of the Stuttering + Modification conditions. Finally, listeners rated the speaker using the cancellation and pullout techniques as being significantly more handicapped than when he was stuttering only. The findings provided preliminary evidence indicating that everyday listeners may react less favorably to an adult male speaker who is modifying his stuttered speech than when this same speaker is simply stuttering.     

Acoustic differences between humorous and sincere communicative intentions

Previous studies indicate that the acoustic features of speech discriminate between positive and negative communicative intentions, such as approval and prohibition. Two studies investigated whether acoustic features of speech can discriminate between two positive communicative intentions: humour and sweet‐sincerity, where sweet‐sincerity involved being sincere in a positive, warm‐hearted way. In Study 1, 22 mothers read a book containing humorous, sweet‐sincere, and neutral‐sincere images to their 19‐ to 24‐month‐olds. In Study 2, 41 mothers read a book containing humorous or sweet‐sincere sentences and images to their 18‐ to 24‐month‐olds. Mothers used a higher mean F0 to communicate visual humour as compared to visual sincerity. Mothers used greater F0 mean, range, and standard deviation; greater intensity mean, range, and standard deviation; and a slower speech rate to communicate verbal humour as compared to verbal sweet‐sincerity. Mothers used a rising linear contour to communicate verbal humour, but used no specific contour to express verbal sweet‐sincerity. We conclude that speakers provide acoustic cues enabling listeners to distinguish between positive communicative intentions.     

Judgment of infant cry: The roles of acoustic characteristics and sociodemographic characteristics

Adult judgments of infant cry are determined by both acoustic properties of the cry and listener sociodemographic characteristics. The main purpose of this research was to investigate how these two sources shape adult judgments of infant cry. We systematically manipulated both the acoustic properties of infant cries and contrasted listener sociodemographic characteristics. Then, we asked participants to listen to several acoustic manipulations of infant cries and to judge the level of distress the infant was expressing and the level of distress participants felt when listening. Finally, as a contrasting condition, participants estimated the age of the crying infant. Using tree‐based models, we found that judgments of the level of distress the infant was expressing as well as the level of distress listeners felt are mainly accounted for by select acoustic properties of infant cry (proportion of sound/pause, fundamental frequency, and number of utterances), whereas age estimates of a crying infant are determined mainly by listener sociodemographic characteristics (gender and parental status). Implications for understanding infant cry and its effects as well as early caregiver‐infant interactions are discussed.     

The development and flexibility of gaze alternations in bonobos and chimpanzees

Infants’ early gaze alternations are one of their first steps towards a sophisticated understanding of the social world. This ability, to gaze alternate between an object of interest and another individual also attending to that object, has been considered foundational to the development of many complex social‐cognitive abilities, such as theory of mind and language. However, to understand the evolution of these abilities, it is important to identify whether and how gaze alternations are used and develop in our closest living relatives, bonobos (Pan paniscus) and chimpanzees (Pan troglodytes). Here, we evaluated the development of gaze alternations in a large, developmental sample of bonobos (N = 17) and chimpanzees (N = 35). To assess the flexibility of ape gaze alternations, we tested whether they produced gaze alternations when requesting food from a human who was either visually attentive or visually inattentive. Similarly to human infants, both bonobos and chimpanzees produced gaze alternations, and did so more frequently when a human communicative partner was visually attentive. However, unlike humans, who gaze alternate frequently from early in development, chimpanzees did not begin to gaze alternate frequently until adulthood. Bonobos produced very few gaze alternations, regardless of age. Thus, it may be the early emergence of gaze alternations, as opposed gaze alternations themselves, that is derived in the human lineage. The distinctively early emergence of gaze alternations in humans may be a critical underpinning for the development of complex human social‐cognitive abilities.     

The effects of self-disclosure and non self-disclosure of stuttering on listeners' perceptions of a person who stutters

The aim of this study was to examine listener perceptions of an adult male person who stutters (PWS) who did or did not disclose his stuttering. Ninety adults who do not stutter individually viewed one of three videotaped monologues produced by a male speaker with severe stuttering. In one monologue, 30 listeners heard the speaker disclose stuttering at the beginning and in another monologue, 30 listeners heard the speaker disclose stuttering at the end. A third group of 30 listeners viewed a monologue where no disclosure of stuttering occurred. After listeners viewed a monologue, they were asked to rate a set of six Likert scale statements and answer three open-ended questions. The results showed that only one of six Likert statements was significantly different across the three conditions. The only statement that was different was that the speaker was perceived to be significantly more friendly when disclosing stuttering at the end of the monologue than when not disclosing stuttering. There were no significant differences between the percentage of positive and negative comments made by listeners across the three conditions. Listeners' comments to each open-ended question showed they were comfortable listening to stuttering with or without disclosure and slightly more than half of the listeners believed their perceptions of the speaker did not change when he disclosed stuttering. The results also showed that the speaker who disclosed stuttering at the beginning of the monologue received significantly more positive listener comments than when he disclosed stuttering at the end of the monologue. Results are discussed relative to comparisons with the study, the clinical relevance of acknowledging stuttering as a component of treatment, and future research on the self-disclosure of stuttering. Educational objectives: The reader will be able to: (1) describe how different groups of listeners perceive and respond to two conditions of self-disclosure of stuttering and one condition involving non self-disclosure of stuttering; (2) summarize the range of listener responses to and benefits of self-disclosure of stuttering; and (3) describe the value of self-disclosure of stuttering for the listener and the speaker.     

Communicative functions of directional verbal probabilities: Speaker’s choice,listener’s inference,and reference points

Verbal probabilities have directional communicative functions, and most can be categorized as positive (e.g., “it is likely”) or negative (e.g., “it is doubtful”). We examined the communicative functions of verbal probabilities based on the reference point hypothesis According to this hypothesis, listeners are sensitive to and can infer a speaker’s reference points based on the speaker’s selected directionality. In four experiments (two of which examined speakers’ choice of directionality and two of which examined listeners’ inferences about a speaker’s reference point), we found that listeners could make inferences about speakers’ reference points based on the stated directionality of verbal probability. Thus, the directionality of verbal probabilities serves the communicative function of conveying information about a speaker’s reference point.     

Abstraction and the (Misnamed) Language Familiarity Effect

Talkers are recognized more accurately if they are speaking the listeners’ native language rather than an unfamiliar language. This “language familiarity effect” has been shown not to depend upon comprehension and must instead involve language sound patterns. We further examine the level of sound‐pattern processing involved, by comparing talker recognition in foreign languages versus two varieties of English, by (a) English speakers of one variety, (b) English speakers of the other variety, and (c) non‐native listeners (more familiar with one of the varieties). All listener groups performed better with native than foreign speech, but no effect of language variety appeared: Native listeners discriminated talkers equally well in each, with the native variety never outdoing the other variety, and non‐native listeners discriminated talkers equally poorly in each, irrespective of the variety's familiarity. The results suggest that this talker recognition effect rests not on simple familiarity, but on an abstract level of phonological processing.