Degree of Language Experience Modulates Visual Attention to Visible Speech and Iconic Gestures During Clear and Degraded Speech Comprehension

Visual information conveyed by iconic hand gestures and visible speech can enhance speech comprehension under adverse listening conditions for both native and non‐native listeners. However, how a listener allocates visual attention to these articulators during speech comprehension is unknown. We used eye‐tracking to investigate whether and how native and highly proficient non‐native listeners of Dutch allocated overt eye gaze to visible speech and gestures during clear and degraded speech comprehension. Participants watched video clips of an actress uttering a clear or degraded (6‐band noise‐vocoded) action verb while performing a gesture or not, and were asked to indicate the word they heard in a cued‐recall task. Gestural enhancement was the largest (i.e., a relative reduction in reaction time cost) when speech was degraded for all listeners, but it was stronger for native listeners. Both native and non‐native listeners mostly gazed at the face during comprehension, but non‐native listeners gazed more often at gestures than native listeners. However, only native but not non‐native listeners' gaze allocation to gestures predicted gestural benefit during degraded speech comprehension. We conclude that non‐native listeners might gaze at gesture more as it might be more challenging for non‐native listeners to resolve the degraded auditory cues and couple those cues to phonological information that is conveyed by visible speech. This diminished phonological knowledge might hinder the use of semantic information that is conveyed by gestures for non‐native compared to native listeners. Our results demonstrate that the degree of language experience impacts overt visual attention to visual articulators, resulting in different visual benefits for native versus non‐native listeners.     

Early understanding of the socially mediated representational function of pictures

The present two studies investigated whether toddlers’ ability to use pictures in problem solving could be facilitated by making available the social context in which pictures were created. To make this social context available, we introduced an Experimental treatment in which the creator was intentionally drawing different objects. In Experiment-1, due to this treatment, children performed better in the Test in which the Experimenter was drawing the Test-pictures of the retrieval task. In Experiment-2, after the same Experimental treatment the facilitative effect was replicated although the social context was removed in the Test phase by using pre-drawn Test-pictures without any drawing action in the retrieval task. The results suggest that a treatment which offers the opportunity to understand the socially mediated representational function of pictures enables children to perform better when contextualizing pictures in current reality. These two experiments revealed a novel way to facilitate children’s picture comprehension by identifying an underlying factor that can explain children’s difficulty in picture comprehension.     

语篇阅读中时间参照加工的似时间转换特点

为了在语篇的层面上从浅层的语词加工和深层的情境加工两个角度考察时间参照加工的特点,采用单因素4水平重复测量实验设计,使用EyelinkⅡ型眼动仪比较了时间参照与时间转换的眼动指标。结果显示,时间词区:远参照与短时转换的眼动指标接近,近参照与长时转换的指标接近;关键事件区:远参照与长时转换的指标接近,近参照与短时转换的指标接近。结果表明时间参照表现出类似时间转换的特点,其特点与能否维系语篇的因果连贯性相关。     

Single-Stage Prediction Models Do Not Explain the Magnitude of Syntactic Disambiguation Difficulty

The disambiguation of a syntactically ambiguous sentence in favor of a less preferred parse can lead to slower reading at the disambiguation point. This phenomenon, referred to as a garden-path effect, has motivated models in which readers initially maintain only a subset of the possible parses of the sentence, and subsequently require time-consuming reanalysis to reconstruct a discarded parse. A more recent proposal argues that the garden-path effect can be reduced to surprisal arising in a fully parallel parser: words consistent with the initially dispreferred but ultimately correct parse are simply less predictable than those consistent with the incorrect parse. Since predictability has pervasive effects in reading far beyond garden-path sentences, this account, which dispenses with reanalysis mechanisms, is more parsimonious. Crucially, it predicts a linear effect of surprisal: the garden-path effect is expected to be proportional to the difference in word surprisal between the ultimately correct and ultimately incorrect interpretations. To test this prediction, we used recurrent neural network language models to estimate word-by-word surprisal for three temporarily ambiguous constructions. We then estimated the slowdown attributed to each bit of surprisal from human self-paced reading times, and used that quantity to predict syntactic disambiguation difficulty. Surprisal successfully predicted the existence of garden-path effects, but drastically underpredicted their magnitude, and failed to predict their relative severity across constructions. We conclude that a full explanation of syntactic disambiguation difficulty may require recovery mechanisms beyond predictability.     

Parsing as a Cue-Based Retrieval Model

This paper develops a novel psycholinguistic parser and tests it against experimental and corpus reading data. The parser builds on the recent research into memory structures, which argues that memory retrieval is content-addressable and cue-based. It is shown that the theory of cue-based memory systems can be combined with transition-based parsing to produce a parser that, when combined with the cognitive architecture ACT-R, can model reading and predict online behavioral measures (reading times and regressions). The parser's modeling capacities are tested against self-paced reading experimental data (Grodner & Gibson, 2005), eye-tracking experimental data (Staub, 2011), and a self-paced reading corpus (Futrell et al., 2018).     

Dynamic relationships between phonological memory and reading: A five year longitudinal study from age 4 to 9

We reconcile competing theories of the role of phonological memory in reading development, by uncovering their dynamic relationship during the first 5 years of school. Phonological memory, reading and phoneme awareness were assessed in 780 phonics‐educated children at age 4, 5, 6 and 9. Confirmatory factor analyses demonstrated that phonological memory loaded onto two factors: verbal short‐term memory (verbal STM; phonological tasks that loaded primarily on serial order memory) and nonword repetition. Using longitudinal structural equation models, we found that verbal STM directly predicted early word‐level reading from age 4 to 6, reflecting the importance of serial‐order memory for letter‐by‐letter decoding. In contrast, reading had no reciprocal influence on the development of verbal STM. The relationship between nonword repetition and reading was bidirectional across the 5 years of study: nonword repetition and reading predicted each other both directly and indirectly (via phoneme awareness). Indirect effects from nonword repetition (and verbal STM) to reading support the view that phonological memory stimulates phonemically detailed representations through repeated encoding of complex verbal stimuli. Similarly, the indirect influence of reading on nonword repetition suggests that improved reading ability promotes the phoneme‐level specificity of phonological representations. Finally, the direct influence from reading to nonword repetition suggests that better readers use orthographic cues to help them remember and repeat new words accurately. A video abstract of this article can be viewed at https://www.youtube.com/watch?v=70LZfTR0BjE .     

Evaluating driver comprehension of the roadway environment to retain accountability of safety during driving automation

Driver comprehension is a substantial component of situation awareness that involves the ability of an individual to understand the significance of an object, traffic sign, or hazard while driving. An increase in crashes related to autonomous driving systems has raised a concern regarding the safety of other roadway users due to the diminishing accountability resulting from a general lack of understanding of the limitations or disregard of the safety protocols by users. To keep drivers vigilant when engaged in partial automated systems, a methodology to monitor real-time driver comprehension was proposed. A driving simulator study consisting of 90 participants, equally split between males and females, was executed to establish driver comprehension in six different variations of driving difficulty. Joint probability density functions were created by considering percent time spent gazing, answers to probe questions, and driving performance. Based on these density functions, five levels of comprehension were devised and assigned thresholds. Overall, as task difficulty increased, a non-linear deterioration in driving speed along with an increase in total gaze duration was observed before comprehension was attained. A two-step validation protocol was also proposed to ensure similar levels of comprehension to non-automated driving from the human driver, when engaged in early forms of automation. The proposed real-time driver comprehension monitoring constitutes a first step toward developing a methodology to reinstate the accountability of safety of other roadway users when engaged in driver-in-the-loop automation systems.     

口语句子的韵律边界：窥探言语理解的秘窗

韵律边界加工与言语理解紧密相关, 最近十几年来逐渐成为心理学和语言学的研究焦点。韵律系统包含若干由小到大的韵律单位, 不同单位的韵律成分其边界强度不同, 表现在音高、延宕和停顿三个声学线索上的参数也不同。句子的听力理解过程中, 听话人运用声学线索感知权重策略对韵律边界的声学线索进行加工。从神经层面上来看, 对于韵律边界的加工, 大脑显示出独立且特异性的神经机制。韵律边界的加工能力在婴儿出生后随年龄的增长而发展, 到了老年阶段则逐渐退化, 而且似乎能够对二语迁移。未来, 需要扩大对韵律边界声学表现的考查范围, 进一步明确韵律边界的加工过程, 进一步厘清韵律边界加工和句法加工之间的关系, 进一步关注二语者韵律边界加工能力的发展。     

输入方式对学语前聋生语篇理解的影响：文本和测题类型的调节效应

为了考察信息输入方式、文本类型和测题类型对聋生语篇理解的影响,研究通过录像呈现,以口语、手语、书面语三种输入方式向学语前极重度七、八、九年级聋生呈现叙述性和描述性两种文本类型的语篇信息,并要求其完成理解测试。结果显示：输入方式对聋生语篇理解的效率存在影响,书面语成绩最高且速度最快,依次为手语、口语,文本类型和测题类型对该影响有调节作用。结果表明输入方式是聋生语篇理解的主要影响因素,文本和测题类型是调节因素。