汉语句子-图片信息整合的眼动测量

采用眼球追踪-分析法考察了“句子-图片”验证范式下句图信息整合过程中句式和语义关系类型的作用。实验结果表明：（1）汉语句子-图片信息整合过程中，句图继时呈现没有消除语义失配效应，注视次数、总注视时间显著增加，瞳孔直径扩大。语义失配会显著干扰句法难度低的句式的句图信息整合。（2）句式影响句子-图片信息的整合过程，注视次数、总注视时间、瞳孔直径随句法难度增加而增加。（3）汉语句图信息整合过程中，语义失配效应受句式影响，表现出任务的特异性，这说明成分比较模型不具有一般性

Sentence-Picture Integration in Chinese:Evidence from the Measurement of Eye Movement

Carpenter and Just (1975) proposed the Constituent Comparison Model (CCM) to translate how a sentence and picture were processed and integrated. They found that semantic incongruency yielded a mismatch effect, and participants experienced greater difficulty in processing and verifying a negation than they did in the case of an affirmation. When the sentence and picture were displayed one after the other, Underwood et al. (2004) did not observe the mismatch effect. However, Knoeferle et al. (2005) found the effect in a sentence comprehension study with serial picture-sentence presentation. The validity of the CCM for other languages has not been confirmed. The following questions remain unanswered: Does syntax interfered with the verification task and mismatch effect? Does the mismatch effect disappear when sentences are presented before the pictures in Chinese? Forty undergraduates were paid to participate in the experiment. All were native speakers of Mandarin Chinese and had normal or corrected-to-normal vision. There were three different types of simple sentences, and each type consisted of four sentences. Each sentence was paired with one of the following two pictures: an asterisk above the plus sign and the plus sign above an asterisk. The pairings of sentences and pictures yielded 24 different displays. Eye movements were recorded using a SensoMotoric Instruments (SMI) iView Hi-Speed system. Each trial started with the appearance of a cross on the left side of the screen, and terminated when the participant pressed a response key. Subsequently, a sentence appeared in the center of the screen, followed by a picture. The task of the participants was to decide whether or not the sentence and picture corresponded with each other, and to indicate their decision by pressing one of the two response keys. The results revealed the main effects of sentence type and semantic relation. The difference between affirmation and double negation was not significant in the sentence-picture task. Participants found it more difficult to process and verify a negation than a double negation. The mismatch effect was evident on the total fixation time, fixation times, and pupil diameters. Furthermore, the interaction of sentence type and semantic relation was significant. The negative sentence-picture was believed to be difficult because participants required more fixation time and total fixation time to complete the tasks of verification and integration in the mismatch condition. Further, the mismatch effect was also found in the case of Chinese. Semantic mismatches tended to increase the participants’ cognitive load, and the syntax influenced the mismatch effect. It is possible that the participants had adopted a special strategy for representing double negative sentences in which double negative verbs canceled each other. Therefore, we should be cautious with regard to the applicability of the CCM as a more general model of sentence-picture tasks in other languages