Tracing the time course of picture--word processing

A number of independent lines of research have suggested that semantic and articulatory information become available differentially from pictures and words. The first of the experiments reported here sought to clarify the time course by which information about pictures and words becomes available by considering the pattern of interference generated when incongruent pictures and words are presented simultaneously in a Stroop-like situation. Previous investigators report that picture naming is easily disrupted by the presence of a distracting word but that word naming is relatively immune to interference from an incongruent picture. Under the assumption that information available from a completed process may disrupt an ongoing process, these results suggest that words access articulatory information more rapidly than do pictures. Experiment 1 extended this paradigm by requiring subjects to verify the category of the target stimulus. In accordance with the hypothesis that picture access the semantic code more rapidly than words, there was a reversal in the interference pattern: Word categorization suffered considerable disruption, whereas picture categorization was minimally affected by the presence of an incongruent word. Experiment 2 sought to further test the hypothesis that access to semantic and articulatory codes is different for pictures and words by examining memory for those items following naming or categorization. Categorized words were better recognized than named words, whereas the reverse was true for pictures, a result which suggests that picture naming involves more extensive processing than picture categorization. Experiment 3 replicated this result under conditions in which viewing time was held constant. The last experiment extended the investigation of memory differences to a situation in which subjects were required to generate the superordinate category name. Here, memory for categorized pictures was as good as memory for named pictures. Category generation also influenced memory for words, memory performance being superior to that following a yes--no verification of category membership. These experiments suggest a model of information access whereby pictures access semantic information were readily than name information, with the reverse being true for words. Memory for both pictures and words was a function of the amount of processing required to access a particular type of information as well as the extent of response differentiation necessitated by the task.     

The role of visual similarity in picture categorization

Categorization is usually assumed to require access to a concept's meaning. When pictures are categorized faster than words, they are assumed to be understood faster than words. However, pictures from the same category are more similar than pictures from different categories. The present article argues that the use of visual similarity as a cue to category membership may produce the picture advantage. The visual similarity hypothesis was tested in two experiments. In the first experiment, pictures showed a disadvantage for the visually similar categories of fruits and vegetables, but showed their usual advantage for the visually dissimilar categories of fruits and animals. In the second experiment, with a mixed list design, pictures were slower only for visually similar different decisions, but showed the usual advantage for all other decisions. The reliability of visual similarity as a cue to the decision accounted well for these results. Because visual similarity can be shown to have large effects on picture categorization, the use of categorization to compare speed of understanding of pictures and words is questionable.     

Words, pictures, and priming: on semantic activation, conscious identification, and the automaticity of information processing

We investigated the encoding mechanisms involved in the perceptual recognition of words and pictures. Latencies in naming word and picture targets were analyzed as a function of several characteristics of a preceding prime, including whether it was a word or a picture, its duration of exposure, the interval between the prime and target onset, and whether or not the prime was consciously identified and reported by the subject. Results indicated that a common semantic code is available that can represent the meaning of either a word or a picture. This semantic representation, however, appears to be more easily activated by picture primes than by word primes and seems to benefit the naming of picture targets more than the naming of word targets. Despite the advantage for pictures with respect to semantic activation, overall processing in the naming task was slower and more attention demanding for pictures than for words. Comparison of our data with data on classification, in which an opposite pattern occurs (overall processing appears to be slower and more attention demanding for words than for pictures), suggests that, on the average, pictures have faster and more automatic access to their meanings than to their names but that words have faster and more automatic access to their names than to their meanings. This conclusion concerning the relative ability of stimuli to activate different kinds of internal representations has implications for a theory of the basis and development of automaticity.     

Semantic component of a cross-modal Stroop-like task

Three experiments showed that the pattern of interference of single-modality Stroop tests also exists cross-modally. Distractors and targets were either pictures or auditory words. In a naming task (Experiment 1), word distractors from the same semantic category as picture targets interfered with picture naming more than did semantically unrelated distractors; the semantic category of picture distractors did not differentially affect word naming. In a categorization task (Experiment 2), this Stroop-like effect was reversed: Picture distractors from the same semantic category as word targets interfered less with word categorization than picture distractors that were semantically unrelated; the semantic category of word distractors did not differentially affect picture categorization. Experiment 3 replicated these effects when each subject performed both tasks; the task, naming or categorizing, determined the pattern of interference between pictures and auditory words. The results thus support the existence of a semantic component of a cross-modal Stroop-like effect.     

Effects of a concurrent memory task on hemispheric asymmetries in categorization

Recent research on the division of processing between the two cerebral hemispheres has often employed two concurrent tasks to investigate the dynamic nature of hemispheric asymmetries. The experiment reported here explored the effects of two concurrent high-level cognitive tasks (memory retention and semantic categorization) on the direction and magnitude of hemispheric differences in the processing of words and pictures. Subjects were required to categorize words and pictures presented to either the left visual field-right hemisphere (LVF-RH) or the right visual field-left hemisphere (RVF-LH). The categorization could be performed while holding either verbal material in memory (digit span), pictorial material in memory (serial nonsense figure recognition), or with no concurrent memory task. The effects produced hemisphere-specific, material-nonspecific interference. The verbal task removed a RVF-LH advantage at word categorization and enhanced a LVF-RH advantage on picture categorization; the pictorial task interfered with picture categorization in the LVF-RH, while enhancing a RVF-LH advantage at word categorization. The results are discussed in terms of multiple resource models of hemisphere function, capacity limitations, and the functional locus of processing required to produce various dynamic hemispheric effects.     

Right-hemisphere interactions in picture-word processing

The processing of pictures was investigated in three experiments which eliminated the response effects involved in naming. When a categorization task was used, clear advantages in response latency and accuracy were observed for left visual-field (LVF) presentations. This was in contrast to previous investigations which have used a naming task and which have reported right visual-field (RVF) advantages. When a distracting word was added to the display, the pattern of influence also changed from that reported previously. The use of naming tasks has indicated predominantly left-hemisphere effects, with demonstrations of interactions between pictures and words in the RVF. With a categorization task in Experiment 2, however, the only effective words were those related in meaning to the picture, and only when they were projected to the right hemisphere. The third experiment confirmed the LVF advantage for picture processing with masked displays, but found no reliable asymmetry with unmasked presentations. The pattern of semantic facilitation was also confirmed with the masked displays, but when the mask was removed an inhibition effect replaced the facilitation effect. These effects are interpreted as indicating that picture recognition is localized within the right cerebral hemisphere. It is suggested that the facilitating effect of related words is restricted to the left hemisphere because it is an effect upon recognition processes, whereas the inhibition effect reflects response competition. It is also suggested that previous reports of left-hemisphere interference effects are due to effects of response competition in naming tasks.     

A comparison of conscious and automatic memory processes for picture and word stimuli: a process dissociation analysis

Four experiments were conducted to evaluate explanations of picture superiority effects previously found for several tasks. In a process dissociation procedure (Jacoby, 1991) with word stem completion, picture fragment completion, and category production tasks, conscious and automatic memory processes were compared for studied pictures and words with an independent retrieval model and a generate-source model. The predictions of a transfer appropriate processing account of picture superiority were tested and validated in "process pure" latent measures of conscious and unconscious, or automatic and source, memory processes. Results from both model fits verified that pictures had a conceptual (conscious/source) processing advantage over words for all tasks. The effects of perceptual (automatic/word generation) compatibility depended on task type, with pictorial tasks favoring pictures and linguistic tasks favoring words. Results show support for an explanation of the picture superiority effect that involves an interaction of encoding and retrieval processes.     

Semantic priming effects on picture and word processing

The effects of semantic priming on picture and word processing were assessed under conditions in which subjects were required simply to identify stimuli (label pictures or read words) as rapidly as possible. Stimuli were presented in pairs (a prime followed by a target), with half of the pairs containing members of the same semantic category and half containing unrelated concepts. Semantic relatedness was found to facilitate the identification of both pictures (Experiment 1) and words (Experiment 2), and obtained interactions of semantic relatedness and stimulus quality in both experiments suggested that semantic priming affects the initial encoding of both types of stimuli. In Experiment 3, subjects received pairs of pictures, pairs of words, and mixed pairs composed of a picture and a word or of a word and a picture. Significant priming effects were obtained on mixed as well as unmixed pairs, supporting the assumption that pictures and words access semantic information from a common semantic store. Of primary interest was the significantly greater priming obtained in picture-picture pairs than in word-word or mixed pairs. This suggests that, in addition to priming that is mediated by the semantic system, priming may occur in picture-picture pairs that results from the overlap in visual features common to the pictorial representations of objects from the same semantic category.     

Picture superiority in conceptual memory: Dissociative effects of encoding and retrieval tasks

We examined the role of encoding processes for picture superiority in explicit and conceptual-implicit memory. The nature of encoding instruction (naming or semantic categorization) yielded dissociative effects on picture and word memory on one explicit test, category-cued recall, and two conceptual-implicit tests, category-cued generation and category-cued verification. Category-cued recall was greater for pictures than for words following naming, but it did not differ for pictures and words following semantic categorization. Category-cued generation priming was greater for pictures than for words following naming, but it was greater for words than for pictures following semantic categorization. In contrast, category-cued verification priming did not differ for pictures and words following either naming or semantic categorization. Thus, picture superiority can be eliminated or reversed depending on the type of conceptual encoding task and conceptual-retrieval test.     

Interhemispheric categorization of pictures and words

Earlier studies suggest that interhemispheric processing increases the processing power of the brain in cognitively complex tasks as it allows the brain to divide the processing load between the hemispheres. We report two experiments suggesting that this finding does not generalize to word-picture pairs: they are processed at least as efficiently when processed by a single hemisphere as compared to processing occurring between the two hemispheres. We examined whether dividing the stimuli between the visual fields/hemispheres would be more advantageous than unilateral stimulus displays in the semantic categorization of simultaneously presented pictures, words, and word-picture pairs. The results revealed that within-domain stimuli (semantically related picture pairs or word pairs) were categorized faster in bilateral than in unilateral displays, whereas cross-domain stimuli (word-picture pairs) were not categorized faster in bilateral than in unilateral displays. It is suggested that interhemispheric sharing of word-picture stimuli is not advantageous as compared to unilateral processing conditions because words and pictures use different access routes, and therefore, it may be possible to process in parallel simultaneously displayed word-picture stimuli within a single hemisphere.     

Electrophysiological differences in the processing of affective information in words and pictures

It is generally assumed that affective picture viewing is related to higher levels of physiological arousal than is the reading of emotional words. However, this assertion is based mainly on studies in which the processing of either words or pictures has been investigated under heterogenic conditions. Positive, negative, relaxing, neutral, and background (stimulus fragments) words and pictures were presented to subjects in two experiments under equivalent experimental conditions. In Experiment 1, neutral words elicited an enhanced late positive component (LPC) that was associated with an increased difficulty in discriminating neutral from background stimuli. In Experiment 2, high-arousing pictures elicited an enhanced early negativity and LPC that were related to a facilitated processing for these stimuli. Thus, it seems that under some circumstances, the processing of affective information captures attention only with more biologically relevant stimuli. Also, these data might be better interpreted on the basis of those models that postulate a different access to affective information for words and pictures.     

Context effects in stroop-like word and picture processing

Presents a series of 6 experiments in which Stroop-like effects were generated by modally pure color-color, picture-picture, and word-word stimuli instead of the usual modally mixed color-word or picture-word stimuli. Naming, reading, and categorization tasks were applied. The Stroop inhibition was preserved with these stimuli but unexpectedly showed a semantic gradient only in the naming and not in the reading task. Word categorizing was slower and more interference prone than picture categorizing. These and other results can be captured by a model with two main assumptions: (a) semantic memory and the lexicon are separate, and (b) words have privileged access to the lexicon, whereas pictures and colors have privileged access to the semantic network. Such a model is developed and put to an initial test.     

The picture superiority effect in associative recognition

The picture superiority effect has been well documented in tests of item recognition and recall. The present study shows that the picture superiority effect extends to associative recognition. In three experiments, students studied lists consisting of random pairs of concrete words and pairs of line drawings; then they discriminated between intact (old) and rearranged (new) pairs of words and pictures at test. The discrimination advantage for pictures over words was seen in a greater hit rate for intact picture pairs, but there was no difference in the false alarm rates for the two types of stimuli. That is, there was no mirror effect. The same pattern of results was found when the test pairs consisted of the verbal labels of the pictures shown at study (Experiment 4), indicating that the hit rate advantage for picture pairs represents an encoding benefit. The results have implications for theories of the picture superiority effect and models of associative recognition.     

The fate of originally presented surface information following recall errors in sentence memory tasks

Pictures are remembered better than their names. This picture superiority effect in episodic memory has been attributed either to the greater sensory distinctiveness of pictures or to their greater conceptual distinctiveness. Weldon and Coyote (1996) tested the conceptual distinctiveness hypothesis by comparing how well pictures as opposed to words primed in two conceptual implicit memory tasks (category production and word association). They found no picture superiority in priming and concluded that the basis of the picture superiority effect must then be pictures' greater sensory distinctiveness. Using the same logic, we compared how well pictures as opposed to words primed in a perceptual implicit memory task (picture and word fragment identification). The sensory distinctiveness theory would predict that pictures should prime picture fragment identification better than words prime word fragment identification, a result we call the picture superiority in within-form priming. Across three experiments which manipulated the encoding task at study, only one showed picture superiority in within-form priming. In contrast, in all three experiments there was robust picture superiority in recall, and exposure to pictures and words at study and test produced independent effects in which both study and test exposure to pictures was more effective for recall than exposure to words. We consider how these results might be reconciled by differences in retrieval demands between recall and fragment identification.     

The activation of word class information during speech production

In 5 picture-word interference experiments the activation of word class information was investigated. The first experiment, in which subjects used bare nouns to describe the pictures, failed to reveal any interference effect of noun distractor words as opposed to closed-class distractor words. In the next 4 experiments the pictures were named by using a definite determiner and the noun completing a sentence fragment. The data demonstrate that noun distractors interfere more strongly with picture naming than do non-noun distractors. This held for both visual and auditory presentation of the distractor words. The interference effect showed up in a time window where semantic interference can usually be observed, supporting the assumption that at an early stage of lexical access semantic and syntactic activation processes overlap.     

Faster might not be better: Pictures may not elicit a stronger unconscious priming effect than words when modulated by semantic similarity

It has been suggested that unconscious semantic processing is stimulus-dependent, and that pictures might have privileged access to semantic content. Those findings led to the hypothesis that unconscious semantic priming effect for pictorial stimuli would be stronger as compared to verbal stimuli. This effect was tested on pictures and words by manipulating the semantic similarity between the prime and target stimuli. Participants performed a masked priming categorization task for either words or pictures with three semantic similarity conditions: strongly similar, weakly similar, and non-similar. Significant differences in reaction times were only found between strongly similar and non-similar and between weakly similar and non-similar, for both pictures and words, with faster overall responses for pictures as compared to words. Nevertheless, pictures showed no superior priming effect over words. This could suggest the hypothesis that even though semantic processing is faster for pictures, this does not imply a stronger unconscious priming effect.     

Semantic interference in picture naming during dual-task performance does not vary with reading ability

Previous dual-task studies examining the locus of semantic interference of distractor words in picture naming have obtained diverging results. In these studies, participants manually responded to tones and named pictures while ignoring distractor words (picture–word interference, PWI) with varying stimulus onset asynchrony (SOA) between tone and PWI stimulus. Whereas some studies observed no semantic interference at short SOAs, other studies observed effects of similar magnitude at short and long SOAs. The absence of semantic interference in some studies may perhaps be due to better reading skill of participants in these than in the other studies. According to such a reading-ability account, participants’ reading skill should be predictive of the magnitude of their interference effect at short SOAs. To test this account, we conducted a dual-task study with tone discrimination and PWI tasks and measured participants’ reading ability. The semantic interference effect was of similar magnitude at both short and long SOAs. Participants’ reading ability was predictive of their naming speed but not of their semantic interference effect, contrary to the reading ability account. We conclude that the magnitude of semantic interference in picture naming during dual-task performance does not depend on reading skill.     

Age of acquisition effects in vocabulary learning

Two experiments examined whether the age of acquisition (AoA) of a concept influences the speed at which native English speakers are able to name pictures using a newly acquired second language (L2) vocabulary. In Experiment 1, participants were taught L2 words associated with pictures. In Experiment 2 a second group of participants were taught the same words associated with L1 translations. Following training both groups performed a picture naming task in which they were asked to name pictures using the newly acquired words. Significant AoA effects were observed only in Experiment 1, in that participants were faster at naming pictures representing early acquired relative to late acquired concepts. The results suggest that the AoA of a concept can exert influence over processing which is independent of the AoA of the word form. The results also indicate that different training methods may lead to qualitative differences in the nature of the links formed between words and concepts during the earliest stages of second language learning.     

Cerebral lateralization of language in deaf and hearing people

In Experiment 1 neither hearing nor prelingually deaf signing adolescents showed marked lateralization for lexical decision but, unlike the hearing, the deaf were not impaired by the introduction of pseudohomophones. In Experiment 2 semantic categorization produced a left hemisphere advantage in the hearing for words but not pictures whereas in the deaf words and signs but not pictures showed a right hemisphere advantage. In Experiment 3 the lexical decision and semantic categorization findings were confirmed and both groups showed a right hemisphere advantage for a face/nonface decision task. The possible effect of initial language acquisition on the development of hemispheric lateralization for language is discussed.     

Presentation format effects in a levels-of-processing task

Three experiments were conducted to examine better performance in long-term memory when stimulus items are pictures or spoken words compared to printed words. Hypotheses regarding the allocation of attention to printed words, the semantic link between pictures and processing, and a rich long-term representation for pictures were tested. Using levels-of-processing tasks eliminated format effects when no memory test was expected and processing was deep (El), and when study and test formats did not match (E3). Pictures produced superior performance when a memory test was expected (El & 2) and when study and test formats were the same (E3). Results of all experiments support the attenuation of attention model and that picture superiority is due to a more direct access to semantic processing and a richer visual code. General principles to guide the processing of stimulus information are discussed.