Multisensory recognition of actively explored objects.

Shape recognition can be achieved through vision or touch, raising the issue of how this information is shared across modalities. Here we provide a short review of previous findings on cross-modal object recognition and we provide new empirical data on multisensory recognition of actively explored objects. It was previously shown that, similar to vision, haptic recognition of objects fixed in space is orientation specific and that cross-modal object recognition performance was relatively efficient when these views of the objects were matched across the sensory modalities (Newell, Ernst, Tjan, & Bülthoff, 2001). For actively explored (i.e., spatially unconstrained) objects, we now found a cost in cross-modal relative to within-modal recognition performance. At first, this may seem to be in contrast to findings by Newell et al. (2001). However, a detailed video analysis of the visual and haptic exploration behaviour during learning and recognition revealed that one view of the objects was predominantly explored relative to all others. Thus, active visual and haptic exploration is not balanced across object views. The cost in recognition performance across modalities for actively explored objects could be attributed to the fact that the predominantly learned object view was not appropriately matched between learning and recognition test in the cross-modal conditions. Thus, it seems that participants naturally adopt an exploration strategy during visual and haptic object learning that involves constraining the orientation of the objects. Although this strategy ensures good within-modal performance, it is not optimal for achieving the best recognition performance across modalities.     

Multisensory identification of natural objects in a two-way crossmodal priming paradigm

The question of how vision and audition interact in natural object identification is currently a matter of debate. We developed a large set of auditory and visual stimuli representing natural objects in order to facilitate research in the field of multisensory processing. Normative data was obtained for 270 brief environmental sounds and 320 visual object stimuli. Each stimulus was named, categorized, and rated with regard to familiarity and emotional valence by N=56 participants (Study 1). This multimodal stimulus set was employed in two subsequent crossmodal priming experiments that used semantically congruent and incongruent stimulus pairs in a S1-S2 paradigm. Task-relevant targets were either auditory (Study 2) or visual stimuli (Study 3). The behavioral data of both experiments expressed a crossmodal priming effect with shorter reaction times for congruent as compared to incongruent stimulus pairs. The observed facilitation effect suggests that object identification in one modality is influenced by input from another modality. This result implicates that congruent visual and auditory stimulus pairs were perceived as the same object and demonstrates a first validation of the multimodal stimulus set.     

Reasoning and mental representation

Four experiments exploring the effects of the coherence of a mental representation of material on reasoning performance are presented. Each employs a simple task that allows most subjects at some stage to solve the problem. We postulate that the crucial factor influencing performance is a unified representation of the material. In Experiment 1 we use an authorization of a kind familiar in daily life, and in Experiments 2, 3 and 4 we use sentences describing simple objects in different ways. In each case performance was enhanced when the material could be given a unified representation.     

Multisensory cortical processing of object shape and its relation to mental imagery

Here, we used functional magnetic resonance imaging to investigate the multisensory processing of object shape in the human cerebral cortex and explored the role of mental imagery in such processing. Regions active bilaterally during both visual and haptic shape perception, relative to texture perception in the respective modality, included parts of the superior parietal gyrus, the anterior intraparietal sulcus, and the lateral occipital complex. Of these bimodal regions, the lateral occipital complexes preferred visual over haptic stimuli, whereas the parietal areas preferred haptic over visual stimuli. Whereas most subjects reported little haptic imagery during visual shape perception, experiences of visual imagery during haptic shape perception were common. Across subjects, ratings of the vividness of visual imagery strongly predicted the amount of haptic shape-selective activity in the right, but not in the left, lateral occipital complex. Thus, visual imagery appears to contribute to activation of some, but not all, visual cortical areas during haptic perception.     

The mental representation of countries

Three studies were conducted to describe adults' knowledge of the countries of the world. Undergraduates from a selective liberal arts college recognised nearly two-thirds of nations, and about one-third were produced in a free-recall task. The use of information related to countries' locations was an important determinant of order of recall. Contiguity in free-recall output, responses in a free-association task, and the prediction of recall from recognition also provided evidence of other relations between countries, including semantic and phonetic links and membership in groups of countries. The number of times a country was mentioned in the news was a consistent predictor of recognition and recall. The relative prosperity, population density, and geographic area of countries also influenced recall.     

Medieval theories of mental representation

Throughout most of the Middle ages, it was generally held that stored mental representations of perceived objects or events preserved the forms or species of such objects. This belief was consistent with a metaphor used by Plato. It was also consistent with the medieval belief that a number of cognitive processes took place in the ventricles of the brain and with the phenomenology of afterimages and imagination itself. In the 14th century, William of Ockham challenged this belief by claiming that mental representations are not stored but instead constructed in the basis of past learned experiences.     

The mental representation of countries

Three studies were conducted to describe adults’ knowledge of the countries of the world. Undergraduates from a selective liberal arts college recognised nearly two-thirds of nations, and about one-third were produced in a free-recall task. The use of information related to countries’ locations was an important determinant of order of recall. Contiguity in free-recall output, responses in a free-association task, and the prediction of recall from recognition also provided evidence of other relations between countries, including semantic and phonetic links and membership in groups of countries. The number of times a country was mentioned in the news was a consistent predictor of recognition and recall. The relative prosperity, population density, and geographic area of countries also influenced recall.     

Mental representation of three-dimensional objects in visual problem solving and recognition

Subjects inspected sets of flat, separated orthographic projections of surfaces of potential three-dimensional objects. After solving problems based on these orthographic views, subjects discriminated between isometric views of the same objects and drawings of distractor structures. Recognition of the isometrics, which had never been shown during the problem solving phase of the experiment, was excellent. In addition, recognition of isometrics corresponding to problems that had been solved correctly when presented in orthographic form was significantly superior to recognition of isometrics based on problems solved incorrectly. In Experiment 2, conditions were included in which either orthographic or isometric views functioned as problem solving or recognition displays. Only in the case of orthographic problem solving followed by isometric recognition (Experiment 1) was the superiority of recognition for correctly-solved problems over incorrectly-solved problems obtained. The pattern of results suggests that viewers construct mental representations embodying structural information about integrated, three-dimensional objects when asked to reason about flat, disconnected projections.     

The spatiotopic representation of visual objects across time

Each eye movement introduces changes in the retinal location of objects. How a stable spatiotopic representation emerges from such variable input is an important question for the study of vision. Researchers have classically probed human observers' performance in a task requiring a location judgment about an object presented at different locations across a saccade. Correct performance on this task requires realigning or remapping retinal locations to compensate for the saccade. A recent study showed that performance improved with longer presaccadic viewing time, suggesting that accurate spatiotopic representations take time to build up. The first goal of the study was to replicate that finding. Two experiments, one an exact replication and the second a modified version, failed to replicate improved performance with longer presaccadic viewing time. The second goal of this study was to examine the role of attention in constructing spatiotopic representations, as theoretical and neurophysiological accounts of remapping have proposed that only attended targets are remapped. A third experiment thus manipulated attention with a spatial cueing paradigm and compared transsaccadic location performance of attended versus unattended targets. No difference in spatiotopic performance was found between attended and unattended targets. Although only negative results are reported, they might nevertheless suggest that spatiotopic representations are relatively stable over time.     

Visual mental rotation of possible and impossible objects

Participants were tested on two visual mental rotation tasks using three-dimensional "possible" and "impossible" shapes. Both types of stimuli can be easily encoded by their parts and how they are spatially organized. However, while possible shapes can also be easily encoded as a global image, it is more difficult to encode impossible shapes in such a way. Participants visually rotated both types of stimuli at comparable rates, reflecting that local representations were used in the process of visual mental rotation.     

Distinct mechanisms for the representation of moving and static objects

The visual system has been suggested to integrate different views of an object in motion. We investigated differences in the way moving and static objects are represented by testing for priming effects to previously seen ("known") and novel object views. We showed priming effects for moving objects across image changes (e.g., mirror reversals, changes in size, and changes in polarity) but not over temporal delays. The opposite pattern of results was observed for objects presented statically; that is, static objects were primed over temporal delays but not across image changes. These results suggest that representations for moving objects are: (1) updated continuously across image changes, whereas static object representations generalize only across similar images, and (2) more short-lived than static object representations. These results suggest two distinct representational mechanisms: a static object mechanism rather spatially refined and permanent, possibly suited for visual recognition, and a motion-based object mechanism more temporary and less spatially refined, possibly suited for visual guidance of motor actions.     

Depth representation of moving 3-D objects in apparent-motion path

Apparent motion is perceived when two objects are presented alternately at different positions. The internal representations of apparently moving objects are formed in an apparent-motion path which lacks physical inputs. We investigated the depth information contained in the representation of 3-D moving objects in an apparent-motion path. We examined how probe objects-briefly placed in the motion path-affected the perceived smoothness of apparent motion. The probe objects comprised 3-D objects which were defined by being shaded or by disparity (convex/concave) or 2-D (flat) objects, while the moving objects were convex/concave objects. We found that flat probe objects induced a significantly smoother motion perception than concave probe objects only in the case of the convex moving objects. However, convex probe objects did not lead to smoother motion as the flat objects did, although the convex probe objects contained the same depth information for the moving objects. Moreover, the difference between probe objects was reduced when the moving objects were concave. These counterintuitive results were consistent in conditions when both depth cues were used. The results suggest that internal representations contain incomplete depth information that is intermediate between that of 2-D and 3-D objects.     

Conceptual structures, representation of objects and object relations]

This article provides an overview of psychological studies of object concepts, highlighting the more recent re-conceptualizations, and the latest developments in research in this field. These developments have tended to focus on the notion of context, as well as on the notion of causal relations between features. Our theoretical analysis of this field is backed up by experimental illustrations. We complete with an examination of "category-specific impairments" studies, in the light of the evolution of concept theories.