Processing strategies in picture-sentence verification tasks

The encoding and comparison strategies used by observers in matching pictures and sentences were investigated. The location of the mismatch or difference between a sentence and a picture was varied, and the latencies of "same"-"different" responses were compared as a function of whether the mismatch occurred in the sentence subject, verb, or object. Sentences were presented auditorily and varied in both voice and reversibility. Pictures either preceded or were presented simultaneously with the sentences. In all cases, Ss adopted a serial self-terminating comparison strategy and a surface structure encoding strategy: The comparison was terminated as soon as a mismatch was encountered, and actives were processed in the order subject-verb-object, whereas passives were processed in the order object-verb-subject.     

Syntax in pictures

It was hypothesised that pictures drawn to represent active and passive sentences would reflect this syntactic difference. Specifically, it was predicted that the reversal of the order in which the logical subject (LS) and the logical object (LO) occur in the change from active to passive form of sentence will affect the order in which they occur in their pictorial representations. A significant directional preference was found for placing the LO to the left of the LS in the pictorial representations of passive but not active sentences. This was discussed in terms of the marked status of the passive in relation to the active. In the case of sentences differing only in lexical marking there was no asymmetric directionality effect corresponding to that found for actives and passives. A significant tendency to match the surface order of the sentences in the pictures was found for both lexically marked and unmarked sentences. The possibility that other spatial or physical dimensions might be used to express syntactic and semantic factors in the pictorial representation of sentences is considered.     

Linguistic structures and mental operations

The internal representations and mental operations used in verifying a sentence against a pictorial referent are examined in an experiment where the location of the false constituent (subject, verb, or object) varied. The latencies for false sentences were ordered: verb, grammatical subject, grammatical object, with verb mismatches being detected the fastest. An information-processing analysis indicated that both comparison and search operations are employed, and that sentences are represented, in general, as a list of propositions with case-relational information. Context and task demands impose order on the list and lead to different operations on these structures. The depth of processing sentences varies from simple lexical comparisons up to full encoding and usage of syntactic and semantic features and case relations.     

Comprehending quantified sentences: The relation between sentence-picture and semantic memory verification

People were timed as they decided whether quantified sentences like All (some) of the round figures are red were true or false of an accompanying picture. The response latency was a function of the quantifier and the relation between the sets mentioned in the subject and predicate of the sentence. The pattern of latencies was similar to the pattern found for sentences that refer to concepts in semantic memory, e.g., All (some) dogs are animals (Meyer, 1970). This result suggests that the same process may be operating in both domains. Two alternative models of the process are considered. One model assumes that the common process consists of computing the relation between the two sets mentioned in the sentence. The other model assumes that the common process consists of comparing the representation of the sentence to the representation of information computed from the second source. Both models are integrated with broader theories of performance in various comprehension tasks.     

ASSOCIATIVE ENCODING OF WORDS IN PASSIVE SENTENCES

Subjects were presented with short passive sentences and timed to respond in a recognition test involving a series of single words. Items to be rejected included synonymous associates of critical words and associates which would fit sensibly into the sentence frame. Rejection latencies varied as a function of the position of the critical word in the sentence, the time to reject associates of the logical subject (the final noun in the passive sentences) being much slower than associates of the object.     

Long-term recognition memory for sentences

The present studies examined immediate and one-week delayed recognition performance for active and passive sentences. Recognition tests included distractor sentences wherein the logical subject, verb, of logical object of a sentence was replaced by a semantically confusable, visually confusable, or unrelated distractor. Results indicated that logical subjects, verbs, and logical objects of sentences were recognized equally well during immediate testing. During delayed testing, changes in the logical object or recipient of the action were not detected as well as verb changes of logical subject changes. The implications of these results for current sentence memory models were discussed.     

Sentence understanding and knowledge of the world: Evidences from a sentence-picture matching task performed by aphasic patients

The abilities of aphasic patients to perform a sentence-picture matching task were investigated on simple declarative reversible sentences. Two factors were found to differentially affect Broca's versus Wernicke's aphasics' performance: sentence plausibility and spatial arrangement. Sentence plausibility, that is to say, the liability of two given nouns in a sentence to act as agent and recipient according to normal subjects' expectancies, had a significant effect on the frequency of correct responses of Broca's aphasics but not on those of Wernicke's. Conversely, the latter were found to be more sensitive to a match or mismatch in the left-to-right spatial arrangement of the grammatical subject and object in a sentence on the one hand, and the order of the corresponding personages in the picture on the other. The results are discussed in terms of “frame selection” and the handling of ordered structure strategies contributing to normal system, but selectively disturbed in the case of brain damage.     

The effects of thematic context and presentation mode on memory for sentence voice

A sentence in discourse may appear in the passive voice to emphasize the logical object rather than the logical subject when it is thematically more important. Two experiments are reported that explore the impact of this textual function of voice on sentence memory. The first experiment required subjects to listen to prose passages and then recall them. Sentences were recalled predominantly in the active voice regardless of voice or thematic focus in the prose passage, showing that the English-language bias for the active voice was a more important determinant of sentence reconstruction than was the experimental manipulation of thematic context. The second experiment required subjects to listen to or read either prose passages or lists of unrelated sentences and then to try to recognize key sentences that were either unchanged or changed lexically, semantically, or in voice. Recognition, both overall and specifically for voice, was better for sentences that were read than for those that were heard, and recognition for semantic change was consistently higher than for any other. Only when passages were read was there evidence in support of a thematic textual influence on memory for sentence voice.This research was supported in part by NIMH Grant 32063.     

On the process of comparing sentences against pictures

The present study outlines a theory of how people compare sentences against pictures. This theory was tested in four experiments in which Ss were timed as they judged whether a sentence (e.g., Star isn't above plus) was true or false of a picture (e.g., ⁺_∗). The latencies in these tasks were consistent with the thesis that: (1) sentences are represented in terms of elementary propositions; (2) pictures are encoded in the same interpretive format; (3) these two codes are compared in an algorithmic series of mental operations, each of which contributes additively to the response latency; and (4) sentence encoding, picture encoding, comparing, and responding are four serially ordered stages, and their component latencies are additive. From these results, it was also possible to rule out certain explanations based on visual imagery, conversion (e.g., converting isn't above into is below), reading time, normative word frequencies, and other factors. Finally, it was shown that this theory is consistent with previous studies on sentence comprehension, sentence verification, concept verification, and other related phenomena.     

Evidence for a predication effect in deciding on the personal significance of abstract word meanings

Two experiments were conducted on college students (combinedN=240) to test for the effect of sentence predication on the independent judgement of word significance. Students judge which of two nouns was personally more significant to them. They also employed these nouns in a task which required them to place one word in the subject location and the other in the predicate location of an incomplete sentence. Administration order of these two experimental tasks was counterbalanced. Experiment I demonstrated that when the sentence-completion task is taken first-in which a predication is necessarily framed between the two nouns-the student will subsequently be more likely to judge the noun placed in the subject location of the sentence as more significant than its counterpart (p<.025). Experiment II provided a cross-validation of these findings and also demonstrated that the location of the more significant noun in the sentence can vary between subject and object location depending on whether the verb relation in the sentence unites the two nouns positively or negatively (p<.001).     

The comparative judgements of generally and specifically gifted children

Different groups of children were compared on sentence verification tasks. The children were either academically, musically or artistically gifted, and there were two forms of the task. In one, a picture was followed by a sentence, and in another, one sentence was followed by another. Subjects had to decide as quickly as possible whether or not the second proposition logically confirmed the first. In the picture-sentence condition results from all groups could be fitted to the constituent comparison model for sentence verification proposed by Carpenter and Just (1975). For the sentence-sentence condition, however, the observed results diverged from those predicted by the model. The results are explained in terms of different degrees of linguistic processing capacities of the subjects, and they demonstrate the importance which verbal-logical congruence has for children. Artistically able children had difficulties in processing subject/object incongruence in sentence pairs whereas musically able children had more problems in processing above/below incongruence.     

Remembering complex sentences

Recall of complex sentences at two retention intervals was examined, using sentences which varied simultaneously in three ways, being either active or passive, of low or high Yngve depth, and predictable or unpredictable. Recall of any particular sentence was cued with either the logical subject, logical object, verb or adverbial phrase noun. In general, unpredictable sentences were recollected better than predictable ones, low Yngve depth sentences were recalled better than high Yngve depth ones, and passive sentences were retained better than active ones. The most effective cue was the object, followed in turn by the subject and the adverbial phrase noun, with the verb being least effective. Implications of these findings are discussed.     

Click monitoring revisited: An on-line study of sentence comprehension

Spoken sentence comprehension is based upon rapid and complex psychological processes, yielding a constantly fluctuating cognitive load. The aim of this study was to evaluate on-line click monitoring, a classical but poorly exploited experimental method, which should allow for an easy measurement of processing load at any chosen point of experimental sentences. In Experiments 1 and 2, we obtained longer latencies to clicks located at the boundary of reversible object relative clauses than to clicks identically located in subject relatives and to clicks located earlier within object relatives. Experiment 3 further revealed that this effect of syntactic type was specific to transposed object relatives and did not occur with normal object relatives. In Experiment 4, we observed longer latencies with semantically reversible than with irreversible sentences, but no difference between actives and passives. These results were obtained under strict control of potential lexical and phonological biases, and suggest that on-line click monitoring may be one useful tool in the study of sentence comprehension.     

汉语口语句子产生中句法启动效应的眼动研究

研究利用眼动技术考察了汉语口语句子产生中的句法启动效应及其眼动模式。实验采用双宾语句(主语+谓语+间接宾语+直接宾语)、介宾语句(主语+谓语+直接宾语+间接宾语)和简单句三种启动句类型,听觉呈现启动句后呈现图片要求被试用句子进行描述,目标句类型为双宾语句或介宾语句。结果发现,启动句类型影响了目标句的句法选择及对图片中不同客体的注视时间比率及注视顺序。无论产生的目标句是否与启动句句法结构一致,双宾语句启动下对间接宾语的客体注视时间比率更高,且更多地优先注视间接宾语,介宾语句启动下对直接宾语的客体注视时间比率更高且更多地优先注视直接宾语,而在简单句启动条件下,眼动注视模式则会受到所产生目标句句法结构的影响。行为结果和眼动指标都支持了两阶段理论中结构选择和内容计划两阶段的观点。     

Reading and listening comprehension in Broca's aphasia: Lexical versus syntactical errors

Ten agrammatic Broca's aphasics were presented with a series of four picture plates together with a spoken or written sentence stimulus. All sentence stimuli were of the structure, the + N + is/are + V + ing + the + N. The four pictures on each stimulus plate represented (a) the correct response, (b) a reversal of the stimulus sentence subject and object, (c) a change in the number of the subject of the stimulus sentence, and (d) a change in one of the major lexical items of the stimulus sentence. Subjects selected the correct picture most often. When they erred, they usually selected a subject-object reversal. Number errors were less frequent, and the patients seldom selected a change in major lexical item. This pattern occurred with both written and spoken sentences. These results were interpreted as reflecting the dependence of agrammatic Broca's aphasics on the semantic interpretation of the lexicon for decoding sentences in the face of deficits in syntactical-grammatical interpretation, irrespective of comprehension modality.     

Different mechanisms for role relations versus verb–action congruence effects: Evidence from ERPs in picture–sentence verification

Extant accounts of visually situated language processing do make general predictions about visual context effects on incremental sentence comprehension; these, however, are not sufficiently detailed to accommodate potentially different visual context effects (such as a scene–sentence mismatch based on actions versus thematic role relations, e.g., (Altmann & Kamide, 2007; Knoeferle & Crocker, 2007; Taylor & Zwaan, 2008; Zwaan & Radvansky, 1998)). To provide additional data for theory testing and development, we collected event-related brain potentials (ERPs) as participants read a subject–verb–object sentence (500 ms SOA in Experiment 1 and 300 ms SOA in Experiment 2), and post-sentence verification times indicating whether or not the verb and/or the thematic role relations matched a preceding picture (depicting two participants engaged in an action). Though incrementally processed, these two types of mismatch yielded different ERP effects. Role–relation mismatch effects emerged at the subject noun as anterior negativities to the mismatching noun, preceding action mismatch effects manifest as centro-parietal N400s greater to the mismatching verb, regardless of SOAs. These two types of mismatch manipulations also yielded different effects post-verbally, correlated differently with a participant's mean accuracy, verbal working memory and visual-spatial scores, and differed in their interactions with SOA. Taken together these results clearly implicate more than a single mismatch mechanism for extant accounts of picture–sentence processing to accommodate.     

汉语句子理解中物体隐含颜色心理模拟的ERPs研究

当前研究采用事件相关电位(ERPs)技术,同时沿用经典的句图匹配范式,考察句子理解中物体典型颜色与非典型颜色的心理加工过程在脑电活动上的反映。实验中被试先阅读句子再判断句子后呈现的图片物体是否在句子中出现过,句子中隐含的物体颜色或是关键物体的典型颜色或是非典型颜色。实验结果发现,典型颜色句子版本下句图不匹配条件比句图匹配条件引发了更大的N400效应;而非典型颜色句子版本下两者N400差异不显著。研究结果表明,人们在汉语句子理解过程中会实时对隐含的物体颜色信息进行心理模拟。并且,句子隐含物体颜色的典型性是造成匹配易化或不匹配易化的关键因素之一。     

The neural mechanisms of word order processing revisited: electrophysiological evidence from Japanese

We present two ERP studies on the processing of word order variations in Japanese, a language that is suited to shedding further light on the implications of word order freedom for neurocognitive approaches to sentence comprehension. Experiment 1 used auditory presentation and revealed that initial accusative objects elicit increased processing costs in comparison to initial subjects (in the form of a transient negativity) only when followed by a prosodic boundary. A similar effect was observed using visual presentation in Experiment 2, however only for accusative but not for dative objects. These results support a relational account of word order processing, in which the costs of comprehending an object-initial word order are determined by the linearization properties of the initial object in relation to the linearization properties of possible upcoming arguments. In the absence of a prosodic boundary, the possibility for subject omission in Japanese renders it likely that the initial accusative is the only argument in the clause. Hence, no upcoming arguments are expected and no linearization problem can arise. A prosodic boundary or visual segmentation, by contrast, indicate an object-before-subject word order, thereby leading to a mismatch between argument "prominence" (e.g. in terms of thematic roles) and linear order. This mismatch is alleviated when the initial object is highly prominent itself (e.g. in the case of a dative, which can bear the higher-ranking thematic role in a two argument relation). We argue that the processing mechanism at work here can be distinguished from more general aspects of "dependency processing" in object-initial sentences.     

Neural dynamics of animacy processing in language comprehension: ERP evidence from the interpretation of classifier-noun combinations

An event-related potential experiment was conducted to investigate the temporal neural dynamics of animacy processing in the interpretation of classifier-noun combinations. Participants read sentences that had a non-canonical structure, object noun + subject noun + verb + numeral-classifier + adjective. The object noun and its classifier were either (a) congruent, (b) incongruent, but matching in animacy, or (c) incongruent, mismatching in animacy. An N400 effect was observed for both incongruent conditions, but not for additional mismatch in animacy. When only data from participants who accepted the non-canonical structure were analyzed, the animacy mismatch elicited a P600 but still no N400. These findings suggest that animacy information is not used immediately for semantic integration of nouns and their classifiers, but is used in a later analysis reflected by P600. Thus, the temporal neural dynamics of animacy processing in sentence comprehension may be modulated by the relevance of animacy to thematic interpretation.     

Age Preservation of the Syntactic Processor in Production

Two experiments are reported on the influence of cognitive aging on grammatical choice in language production. In both experiments, participants from two age-groups (young and old) produced sentences in a formulation task (V. Ferreira, 1996) that contrasted conditions allowing a choice between alternative sentence arrangements (i.e., double object or prepositional dative) or that permitted no choice (i.e., prepositional dative only). Experiment 1 showed that older adults were able to formulate the alternative sentence arrangements with the same speed and fluency as young adults. Experiment 2 showed that cueing attention to one of the two object nouns to be included in the sentence resulted in the earlier expression of the cued noun in choice conditions, but with little evidence of a response time or dysfluency cost in the no-choice condition. As in Experiment 1, there were no substantive age differences in latencies or dysfluencies. These results support existing models for the mechanisms that choose between grammatical alternatives and bind phrases to available argument positions and provide evidence that older adults are not impaired in their use of these mechanisms.