Episodic encoding and recognition of pictures and words: role of the human medial temporal lobes

In the present PET study, we examined brain activity related to processing of pictures and printed words in episodic memory. Our goal was to determine how the perceptual format of objects (verbal versus pictorial) is reflected in the neural organization of episodic memory for common objects. We investigated this issue in relation to encoding and recognition with a particular focus on medial temporal-lobe (MTL) structures. At encoding, participants saw pictures of objects or their written names and were asked to make semantic judgments. At recognition, participants made yes-no recognition judgments in four different conditions. In two conditions, target items were pictures of objects; these objects had originally been encoded either in picture or in word format. In two other conditions, target items were words; they also denoted objects originally encoded either as pictures or as words. Our data show that right MTL structures are differentially involved in picture processing during encoding and recognition. A posterior MTL region showed higher activation in response to the presentation of pictures than of words across all conditions. During encoding, this region may be involved in setting up a representation of the perceptual information that comprises the picture. At recognition, it may play a role in guiding retrieval processes based on the perceptual input, i.e. the retrieval cue. Another more anterior right MTL region was found to be differentially involved in recognition of objects that had been encoded as pictures, irrespective of whether the retrieval cue provided was pictorial or verbal in nature; this region may be involved in accessing stored pictorial representations. Our results suggest that left MTL structures contribute to picture processing only during encoding. Some regions in the left MTL showed an involvement in semantic encoding that was picture specific; others showed a task-specific involvement across pictures and words. Together, our results provide evidence that the involvement of some but not all MTL regions in episodic encoding and recognition is format specific.     

Depictions of motion devised by a blind person

A blind man (E.A.) was asked to draw pictures suggesting wheels in various kinds of motion. Six pictures were drawn by E.A. The pictures were shown to sighted subjects, who were asked to assign labels to the pictures, in a multiple-choice format. The labels were assigned at a rate above chance. We argue that the pictures are metaphoric and that pictorial metaphor relies on common properties of the static picture and the kinetic referent.     

The verbal nature of representations of the canonical colors of objects

In selecting the canonical colors of color-specific objects, children may use verbal mediation, a cognitive process whereby an object and its color are matched using verbal rather than pictorial representation [British Journal of Developmental Psychology 14 (1996) 339]. To investigate this process, 108 2- to 5-year-old children were asked to identify 11 colors and to choose crayons to color pictures of color-specific objects. Canonical color choice was significantly predicted by color-labeling skill above the variance portion predicted by age alone. Children were also asked to explain their color choices. Children who knew their colors and who colored canonically provided explanations for their color choices consistent with verbal mediation. However, a high proportion of children who did not know a color, and thus could not use verbal mediation, also gave sophisticated explanations for their color choices if they colored the objects canonically. These findings provide modest support for the idea that identification of canonical colors of objects is a verbal process.     

Expertise in pictorial perception: eye-movement patterns and visual memory in artists and laymen

In two sessions with free scanning and memory instructions, eye-movement patterns from nine artists were compared with those of nine artistically untrained participants viewing 16 pictures representing a selection of categories from ordinary scenes to abstraction: 12 pictures were made to accommodate an object-oriented viewing mode (selection of recognisable objects), and a pictorial viewing mode (selection of more structural features), and 4 were abstract. The artistically untrained participants showed preference for viewing human features and objects, while the artists spent more scanning time on structural/abstract features. A group by session interaction showed a change of viewing strategy in the artists, who viewed more objects and human features in the memory task session. A verbal test of recall memory showed no overall difference in the number of pictures remembered, but the number of correctly remembered pictorial features was significantly higher for artists than for the artistically untrained viewers, irrespective of picture type. No differences in fixation frequencies/durations were found between groups across sessions, but a significant task-dependent-group by session interaction of fixation frequency/duration showed that the artistically untrained participants demonstrated repetition effects in fewer, longer fixations with repeated viewing, while the opposite pattern obtained for the artists.     

On the significance of “double encoding” for the superior recall of pictures to names

Ten-year-old children who were shown pictures of objects immediately preceded by the object's name recalled the material no better than those exposed to the names of the stimuli alone. Both conditions yielded significantly poorer retention than those in which pictures alone were presented or pictures followed by their names. A second study replicated this result. In addition this demonstrated, by a picture and name recognition task, that the effects could not be due to subjects in the “name prior to picture” condition ignoring the pictorial component. These results were interpreted as contradicting the “double encoding” explanation of the superiority of pictures to names in free recall. Parallel visual and verbal encoding of a pictured object does not facilitate retention unless the verbal cue is actively elicited from the subject by the stimulus. The implications of this result for other studies which have employed either simultaneous or sequential presentation of pictures and names are briefly discussed.     

French norms of imagery for pictures, for concrete and abstract words]

This paper deals with French norms for mental image versus picture agreement for 138 pictures and the imagery value for 138 concrete words and 69 abstract words. The pictures were selected from Snodgrass et Vanderwart's norms (1980). The concrete words correspond to the dominant naming response to the pictorial stimuli. The abstract words were taken from verbal associative norms published by Ferrand (2001). The norms were established according to two variables: 1) mental image vs. picture agreement, and 2) imagery value of words. Three other variables were controlled: 1) picture naming agreement; 2) familiarity of objects referred to in the pictures and the concrete words, and 3) subjective verbal frequency of words. The originality of this work is to provide French imagery norms for the three kinds of stimuli usually compared in research on dual coding. Moreover, these studies focus on figurative and verbal stimuli variations in visual imagery processes.     

On the construction of thinking

In this study the author attempts to widen our understanding of language and thought construction by using Chinese ideograms as a model. This allows us to understand how concrete internal objects can coalesce to form abstract internal objects, e.g. concepts, ideas, symbols, and metaphors. One can establish a pictorial language that antedates verbal language. This is the case in the dream work that we routinely study. Thus, this study delves into the physiology of the alpha function. To this end the author not only relies on psychoanalytic concepts but also on concepts from philosophy and from language itself. The author presents the ideograms not only for discussion but also for their visual impact. If we really want to understand pictorial images, they must be seen and not just discussed from a theoretical point of view. Based on this understanding the author advances a proposal for the technique used in treating patients who cannot establish mental representations for their affect. And the technical proposal is illustrated in two clinical cases.     

Differential weighting of common and distinctive components

We investigated possible explanations of the finding that the relative weight (W) of common components in similarity judgments is higher for verbal than for pictorial stimuli. A serial presentation of stimulus components had no effect on verbal stimuli; it increased the impact of both common and distinctive components of pictorial stimuli but did not affect their relative weight. On the other hand, W was increased by manipulations that reduced the cohesiveness of composite pictures, such as separating, scrambling, and mixing their components. Furthermore, W was decreased by manipulations that enhanced the cohesiveness of composite verbal stimuli by imposing structure on their components. Verbal and pictorial representations of the same stimuli yielded no systematic differences in W.     

A normative set of 98 pairs of nonsensical pictures (droodles)

Our purpose in the present study is to provide a normative set of nonsensical pictures known as droodles and to demonstrate the role of semantic comprehension in facilitating recall of pictorial stimuli. The set consists of 98 pairs of droodles. Experiment 1 standardized these pictorial stimuli with respect to several variables, such as appropriateness of verbal labels, relationship between two droodles, and correct recall. Appropriateness of verbal labels was rated higher for pictures presented in pairs than for pictures presented singly. Experiment 2 used the standardized set of droodles in a recall experiment similar to those of Bower, Karlin, and Dueck (1975) and others. As we expected, semantic interpretation can strongly facilitate recall. Multiple regression analysis showed that several measures had significant power of explanation for recall performance. The full set of norms and pictures from this article may be downloaded from http://brm.psychonomic-journals.org/content/supplemental.     

A MODEL FOR THE USE OF ATTITUDE SCALES ACROSS CULTURES

To minimize the verbal content of the attitude measurement situation with Papua New Guinea students, a set of pictorial items was selected for use in the place of verbal items. Initially, 150 pictures related to concepts in the Wilson-Patterson Attitude Inventory (WPAI) were described by a group of 50 students. Seventy of these items, those which were described in a consistent way, were presented to 227 students along with a Papua New Guinea version of the WPAI. A final set of 25 pictures was selected on the basis of their discriminative power, internal consistency, and their content validity. Reliability and validity measures and the rotated factor structure of the 25 item pictorial scale were consistent with the WPAI.     

Exploring the superadditive effects of scent and pictures on verbal recall: An extension of dual coding theory

This research extends the dual coding theory of memory retrieval (Paivio 1969, 2007) beyond its traditional focus on verbal and pictorial information to olfactory information. We manipulate the presence or absence of olfactory and pictorial stimuli at the time of encoding (study 1) or retrieval (study 2) and measure the impact on verbal recall. After a time delay, scent enhances recall of verbal information, and scent-based retrieval cues potentiate the facilitative effect of pictures on recall. These results cannot be attributed merely to increased elaboration at the time of exposure.     

How are pictures registered in memory?

Three experiments tested the hypothesis that pictorial memory is much less dependent on rehearsal than is verbal memory. Experiment I examined incidental learning since this is assumed to reflect learning with little or no rehearsal. Following a classification task, intentional and incidental learning for pictures and for words was compared. The superiority of pictorial memory was especially marked in incidental learning. Experiment II showed that this result was not due to differences in the amount of processing required to classify pictures and words. RTs to classify words and pictures did not differ, and incidental learning was again superior for pictures. In Experiment III rehearsal opportunity was restricted by a concurrent task during presentation of word and picture lists, and the decrement was very much greater for word learning than for picture learning. It was concluded that manipulation of rehearsal opportunity has relatively little effect on pictorial memory.     

Pictorial and conceptual representation of glimpsed pictures

Pictures seen in a rapid sequence are remembered briefly, but most are forgotten within a few seconds (M. C. Potter. A. Staub, J. Rado. & D. H. O'Connor. 2002). The authors investigated the pictorial and conceptual components of this fleeting memory by presenting 5 pictured scenes and immediately testing recognition of verbal titles (e.g., people at a table) or recognition of the pictures themselves. Recognition declined during testing, but initial performance was higher and the decline steeper when pictures were tested. A final experiment included test decoy pictures that were conceptually similar to but visually distinct from the original pictures. Yeses to decoys were higher than yeses to other distractors. Fleeting memory for glimpsed pictures has a strong conceptual component (conceptual short-term memory), but there is additional highly volatile pictorial memory (pictorial short-term memory) that is not tapped hy a gist title or decoy picture.     

Observing spontaneous metaphor in children

The study considered whether apparent metaphors are a frequent part of child language and whether the child recognizes the metaphoric relation created. Seventy-three nursery and kindergarten children (2 to 6 years of age) were observed for one or two half-hour periods of free play. Naturally occurring utterances in unconventional uses were recorded. The children were then questioned about their possible metaphoric creations to determine their awareness and understanding of their utterances. Results suggest that metaphoric processes exist quite early in development, as exemplified by a high frequency of spontaneous metaphor in the free play of young children. The semantic extensions were often deliberate and used appropriately. On some occasions the child was able to articulate the rationale for the verbal substitution. The content and cognitive features of the figures are discussed. Several hypotheses are offered for the developmental trend of decline in frequency of metaphor use with age.     

Role of referent concreteness in the acquisition of passive sentence comprehension through abstract modeling

The present experiment tested the hypothesis that development of syntactic comprehension through verbal modeling is enhanced by referent concreteness as a contextual influence. Young children heard a model narrate a series of events in passive form while the model either performed the corresponding activities, showed pictures portraying the same activities, or displayed no referential aids. In accord with prediction, verbal modeling with enactive referents produced higher levels of comprehension of passives than modeling with pictorial referents or modeling without referential aids. Modeling with pictorial referents and modeling without referents did not differ in overall efficacy. However, modeling alone produced results that were less consistent across different measures of comprehension. Children who lacked understanding of passives were more dependent on concrete referents than those who had some initial comprehension of the linguistic form. The results suggest that verbal modeling with pictorial referents and verbal modeling alone facilitate comprehension of passives, whereas verbal modeling with enactive referents promotes learning. Findings of a supplemental experiment reveal that the effects of verbal modeling on comprehension are enhanced when syntactic forms occur in a meaningful verbal context.     

Memory for modality of presentation: Within-modality discrimination

Two experiments demonstrated that Ss are capable of making within-modality memory discriminations in both visual and auditory modalities. In Experiment I Ss studied mixed lists of pictures and labels representing common objects and were subsequently required to judge whether the original presentation was pictorial or verbal The high level of performance achieved on this task was unaffected by degree of categorical relatedness of items within method of presentation or by instructions to produce visual images when items were presented verbally. In Experiment II Ss demonstrated the ability to remember whether a sentence was originally presented by a male or a female speaker. Some strategies by which within-modality discrimination in memory might be accomplished are discussed.     

Recognition memory for pictorial material in subclinical depression

Depression has been associated with impaired recollection of episodic details in tests of recognition memory that use verbal material. In two experiments, the remember/know procedure was employed to investigate the effects of dysphoric mood on recognition memory for pictorial materials that may not be subject to the same processing limitations found for verbal materials in depression. In Experiment 1, where the recognition test took place two weeks after encoding, subclinically depressed participants reported fewer know judgements which were likely to be at least partly due to a remember-to-know shift. Although pictures were accompanied by negative or neutral captions at encoding, no effect of captions on recognition memory was observed. In Experiment 2, where the recognition test occurred soon after viewing the pictures, subclinically depressed participants reported fewer remember judgements. All participants reported more remember judgements for pictures of emotionally negative content than pictures of neutral content. Together, these findings demonstrate that recognition memory for pictorial stimuli is compromised in dysphoric individuals in a way that is consistent with a recollection deficit for episodic detail and also reminiscent of that previously reported for verbal materials. These findings contribute to our developing understanding of how mood and memory interact.     

Learning 10000 pictures

Four experiments are reported which examined memory capacity and retrieval speed for pictures and for words. Single-trial learning tasks were employed throughout, with memory performance assessed by forced-choice recognition, recall measures or choice reaction-time tasks. The main experimental findings were: (1) memory capacity, as a function of the amount of material presented, follows a general power law with a characteristic exponent for each task; (2) pictorial material obeys this power law and shows an overall superiority to verbal material. The capacity of recognition memory for pictures is almost limitless, when measured under appropriate conditions; (3) when the recognition task is made harder by using more alternatives, memory capacity stays constant and the superiority of pictures is maintained; (4) picture memory also exceeds verbal memory in terms of verbal recall; comparable recognition/recall ratios are obtained for pictures, words and nonsense syllables; (5) verbal memory shows a higher retrieval speed than picture memory, as inferred from reaction-time measures. Both types of material obey a power law, when reaction-time is measured for various sizes of learning set, and both show very rapid rates of memory search.

From a consideration of the experimental results and other data it is concluded that the superiority of the pictorial mode in recognition and free recall learning tasks is well established and cannot be attributed to methodological artifact.