A sharp image or a sharp knife: norms for the modality-exclusivity of 774 concept-property items

According to recent embodied cognition theories, mental concepts are represented by modality-specific sensory-motor systems. Much of the evidence for modality-specificity in conceptual processing comes from the property-verification task. When applying this and other tasks, it is important to select items based on their modality-exclusivity. We collected modality ratings for a set of 387 properties, each of which was paired with two different concepts, yielding a total of 774 concept-property items. For each item, participants rated the degree to which the property could be experienced through five perceptual modalities (vision, audition, touch, smell, and taste). Based on these ratings, we computed a measure of modality exclusivity, the degree to which a property is perceived exclusively through one sensory modality. In this paper, we briefly sketch the theoretical background of conceptual knowledge, discuss the use of the property-verification task in cognitive research, provide our norms and statistics, and validate the norms in a memory experiment. We conclude that our norms are important for researchers studying modality-specific effects in conceptual processing.     

Verifying different-modality properties for concepts produces switching costs

According to perceptual symbol systems, sensorimotor simulations underlie the representation of concepts. It follows that sensorimotor phenomena should arise in conceptual processing. Previous studies have shown that switching from one modality to another during perceptual processing incurs a processing cost. If perceptual simulation underlies conceptual processing, then verifying the properties of concepts should exhibit a switching cost as well. For example, verifying a property in the auditory modality (e.g., BLENDER-loud) should be slower after verifying a property in a different modality (e.g., CRANBERRIES-tart) than after verifying a property in the same modality (e.g., LEAVES-rustling). Only words were presented to subjects, and there were no instructions to use imagery. Nevertheless, switching modalities incurred a cost, analogous to the cost of switching modalities in perception. A second experiment showed that this effect was not due to associative priming between properties in the same modality. These results support the hypothesis that perceptual simulation underlies conceptual processing.     

An Information Space for Partial Length Perception in Dynamic Touch

Two experiments evaluated change in the perception of an environmental property (object length) in each of 3 perceptual modalities (vision, audition, and haptics) when perceivers were provided with the opportunity to experience the same environmental property by means of an additional perceptual modality (e.g., haptics followed by vision, vision followed by audition, or audition followed by haptics). Experiment 1 found that (a) posttest improvements in perceptual consistency occurred in all 3 perceptual modalities, regardless of whether practice included experience in an additional perceptual modality and (b) posttest improvements in perceptual accuracy occurred in haptics and audition but only when practice included experience in an additional perceptual modality. Experiment 2 found that learning curves in each perceptual modality could be accommodated by a single function in which auditory perceptual learning occurred over short time scales, haptic perceptual learning occurred over middle time scales, and visual perceptual learning occurred over long time scales. Analysis of trial-to-trial variability revealed patterns of long-term correlations in all perceptual modalities regardless of whether practice included experience in an additional perceptual modality.     

Modality exclusivity norms for 400 nouns: The relationship between perceptual experience and surface word form

We present modality exclusivity norms for 400 randomly selected noun concepts, for which participants provided perceptual strength ratings across five sensory modalities (i.e., hearing, taste, touch, smell, and vision). A comparison with previous norms showed that noun concepts are more multimodal than adjective concepts, as nouns tend to subsume multiple adjectival property concepts (e.g., perceptual experience of the concept baby involves auditory, haptic, olfactory, and visual properties, and hence leads to multimodal perceptual strength). To show the value of these norms, we then used them to test a prediction of the sound symbolism hypothesis: Analysis revealed a systematic relationship between strength of perceptual experience in the referent concept and surface word form, such that distinctive perceptual experience tends to attract distinctive lexical labels. In other words, modality-specific norms of perceptual strength are useful for exploring not just the nature of grounded concepts, but also the nature of form–meaning relationships. These norms will be of benefit to those interested in the representational nature of concepts, the roles of perceptual information in word processing and in grounded cognition more generally, and the relationship between form and meaning in language development and evolution.     

Perceptual and motor attribute ratings for 559 object concepts

To understand how and when object knowledge influences the neural underpinnings of language comprehension and linguistic behavior, it is critical to determine the specific kinds of knowledge that people have. To extend the normative data currently available, we report a relatively more comprehensive set of object attribute rating norms for 559 concrete object nouns, each rated on seven attributes corresponding to sensory and motor modalities??color, motion, sound, smell, taste, graspability, and pain??in addition to familiarity (376 raters, M = 23 raters per item). The mean ratings were subjected to principal-components analysis, revealing two primary dimensions plausibly interpreted as relating to survival. We demonstrate the utility of these ratings in accounting for lexical and semantic decision latencies. These ratings should prove useful for the design and interpretation of experimental tests of conceptual and perceptual object processing.     

A taste of words: linguistic context and perceptual simulation predict the modality of words

Previous studies have shown that object properties are processed faster when they follow properties from the same perceptual modality than properties from different modalities. These findings suggest that language activates sensorimotor processes, which, according to those studies, can only be explained by a modal account of cognition. The current paper shows how a statistical linguistic approach of word co-occurrences can also reliably predict the category of perceptual modality a word belongs to (auditory, olfactory-gustatory, visual-haptic), even though the statistical linguistic approach is less precise than the modal approach (auditory, gustatory, haptic, olfactory, visual). Moreover, the statistical linguistic approach is compared with the modal embodied approach in an experiment in which participants verify properties that share or shift modalities. Response times suggest that fast responses can best be explained by the linguistic account, whereas slower responses can best be explained by the embodied account. These results provide further evidence for the theory that conceptual processing is both linguistic and embodied, whereby less precise linguistic processes precede precise simulation processes.     

ARE EMOTIONS PERCEPTUAL EXPERIENCES OF VALUE?

A number of emotion theorists hold that emotions are perceptions of value. In this paper I say why they are wrong. I claim that in the case of emotion there is nothing that can provide the perceptual modality that is needed if the perceptual theory is to succeed (where by ‘perceptual modality’ I mean the particular manner in which something is perceived). I argue that the five sensory modalities are not possible candidates for providing us with ‘emotional perception’. But I also say why the usual candidate offered – namely feeling or affectivity – does not give us the sought‐after perceptual modality. I conclude that as there seems to be nothing else that can provide the needed perceptual modality, we should reject the perceptual theory of emotion. 1     

Sensory load incurs conceptual processing costs

Theories of grounded cognition propose that modal simulations underlie cognitive representation of concepts [Barsalou, L. W. (1999). Perceptual symbol systems. Behavioral and Brain Sciences, 22(4), 577-660; Barsalou, L. W. (2008). Grounded cognition. Annual Review of Psychology, 59, 617-645]. Based on recent evidence of modality-specific resources in perception, we hypothesized that verifying properties of concepts encoded in different modalities are hindered more by perceptual short-term memory load to the same versus different sensory modality as that used to process the property. We manipulated load to visual and auditory modalities by having participants store one or three items in short-term memory during property verification. In the high (but not low) load condition, property verification took longer when the property (e.g., yellow) involved the same modality as that used by the memory load (e.g., pictures). Interestingly, similar interference effects were obtained on the conceptual verification and on the memory task. These findings provide direct support for the view that conceptual processing relies on simulation in modality-specific systems.     

Similarity and categorization: from vision to touch

Even though human perceptual development relies on combining multiple modalities, most categorization studies so far have focused on the visual modality. To better understand the mechanisms underlying multisensory categorization, we analyzed visual and haptic perceptual spaces and compared them with human categorization behavior. As stimuli we used a three-dimensional object space of complex, parametrically-defined objects. First, we gathered similarity ratings for all objects and analyzed the perceptual spaces of both modalities using multidimensional scaling analysis. Next, we performed three different categorization tasks which are representative of every-day learning scenarios: in a fully unconstrained task, objects were freely categorized, in a semi-constrained task, exactly three groups had to be created, whereas in a constrained task, participants received three prototype objects and had to assign all other objects accordingly. We found that the haptic modality was on par with the visual modality both in recovering the topology of the physical space and in solving the categorization tasks. We also found that within-category similarity was consistently higher than across-category similarity for all categorization tasks and thus show how perceptual spaces based on similarity can explain visual and haptic object categorization. Our results suggest that both modalities employ similar processes in forming categories of complex objects.     

Modality switching costs emerge in concept creation as well as retrieval

Theories of embodied cognition hold that the conceptual system uses perceptual simulations for the purposes of representation. A strong prediction is that perceptual phenomena should emerge in conceptual processing, and, in support, previous research has shown that switching modalities from one trial to the next incurs a processing cost during conceptual tasks. However, to date, such research has been limited by its reliance on the retrieval of familiar concepts. We therefore examined concept creation by asking participants to interpret modality-specific compound phrases (i.e., conceptual combinations). Results show that modality switching costs emerge during the creation of new conceptual entities: People are slower to simulate a novel concept (e.g., auditory jingling onion) when their attention has already been engaged by a different modality in simulating a familiar concept (e.g., visual shiny penny). Furthermore, these costs cannot be accounted for by linguistic factors alone. Rather, our findings support the embodied view that concept creation, as well as retrieval, requires situated perceptual simulation.     

Contour-based object identification and segmentation: stimuli, norms and data, and software tools.

We summarize five studies of our large-scale research program, in which we examined aspects of contour-based object identification and segmentation, and we report on the stimuli we used, the norms and data we collected, and the software tools we developed. The stimuli were outlines derived from the standard set of line drawings of everyday objects by Snodgrass and Vanderwart (1980). We used contour curvature as a major variable in all the studies. The total number of 1,500 participants produced very solid, normative identification rates of silhouettes and contours, straight-line versions, and fragmented versions, and quite reliable benchmark data about saliency of points and object segmentation into parts. We also developed several software tools to generate stimuli and to analyze the data in nonstandard ways. Our stimuli, norms and data, and software tools have great potential for further exploration of factors influencing contour-based object identification, and are also useful for researchers in many different disciplines (including computer vision) on a wide variety of research topics (e.g., priming, agnosia, perceptual organization, and picture naming). The full set of norms, data, and stimuli may be downloaded from www.psychonomic.org/archive/.     

Contour-based object identification and segmentation: Stimuli,norms and data,and software tools

We summarize five studies of our large-scale research program, in which we examined aspects of contour-based object identification and segmentation, and we report on the stimuli we used, the norms and data we collected, and the software tools we developed. The stimuli were outlines derived from the standard set of line drawings of everyday objects by Snodgrass and Vanderwart (1980). We used contour curvature as a major variable in all the studies. The total number of 1,500 participants produced very solid, normative identification rates of silhouettes and contours, straight-line versions, and fragmented versions, and quite reliable benchmark data about saliency of points and object segmentation into parts. We also developed several software tools to generate stimuli and to analyze the data in nonstandard ways. Our stimuli, norms and data, and software tools have great potential for further exploration of factors influencing contour-based object identification, and are also useful for researchers in many different disciplines (including computer vision) on a wide variety of research topics (e.g., priming, agnosia, perceptual organization, and picture naming). The full set of norms, data, and stimuli may be downloaded fromwww.psychonomic.org/archive/.     

Transfer between vision and haptics: Memory for 2-D patterns and 3-D objects

Explicit memory tests such as recognition typically access semantic, modality-independent representations, while perceptual implicit memory tests typically access presemantic, modality-specific representations. By demonstrating comparable cross- and within-modal priming using vision and haptics with verbal materials (Easton, Srinivas, & Greene, 1997), we recently questioned whether the representations underlying perceptual implicit tests were modality specific. Unlike vision and audition, with vision and haptics verbal information can be presented in geometric terms to both modalities. The present experiments extend this line of research by assessing implicit and explicit memory within and between vision and haptics in the nonverbal domain, using both 2-D patterns and 3-D objects. Implicit test results revealed robust cross-modal priming for both 2-D patterns and 3-D objects, indicating that vision and haptics shared abstract representations of object shape and structure. Explicit test results for 3-D objects revealed modality specificity, indicating that the recognition system keeps track of the modality through which an object is experienced.     

Sensorimotor simulations underlie conceptual representations: Modality-specific effects of prior activation

According to the perceptual symbols theory (Barsalou, 1999), sensorimotor simulations underlie the representation of concepts. Simulations are componential in the sense that they vary with the context in which the concept is presented. In the present study, we investigated whether representations are affected by recent experiences with a concept. Concept names (e.g., APPLE) were presented twice in a property verification task with a different property on each occasion. The two properties were either from the same perceptual modality (e.g.,green, shiny) or from different modalities (e.g.,tart, shiny). All stimuli were words. There was a lag of several intervening trials between the first and second presentation. Verification times and error rates for the second presentation of the concept were higher if the properties were from different modalities than if they were from the same modality.     

Perceptual simulation in property verification

If people represent concepts with perceptual simulations, two predictions follow in the property verification task (e.g., Isface a property of GORILLA?). First, perceptual variables such as property size should predict the performance of neutral subjects, because these variables determine the ease of processing properties in perceptual simulations (i.e., perceptual effort). Second, uninstructed neutral subjects should spontaneously construct simulations to verify properties and therefore perform similarly to imagery subjects asked explicitly to use images (i.e., instructional equivalence). As predicted, neutral subjects exhibited both perceptual effort and instructional equivalence, consistent with the assumption that they construct perceptual simulations spontaneously to verify properties. Notably, however, this pattern occurred only when highly associated false properties prevented the use of a word association strategy. In other conditions that used unassociated false properties, the associative strength between concept and property words became a diagnostic cue for true versus false responses, so that associative strength became a better predictor of verification than simulation. This pattern indicates that conceptual tasks engender mixtures of simulation and word association, and that researchers must deter word association strategies when the goal is to assess conceptual knowledge.     

Variations in characteristic perceptual asymmetry: modality specific and modality general components.

The current study investigates sources of between-subjects variation in asymmetry scores on visual half-field and dichotic listening tasks. For each of these presentation modalities, subjects were given multiple laterality tasks. Results indicate that about 50% of the between-subjects variations in asymmetry scores in each modality is attributable to individual differences in perceptual asymmetries that are not stimulus-specific, referred to as "characteristic perceptual asymmetries" in this paper. In addition, the question of whether individual differences in characteristic perceptual asymmetries are modality specific, modality general, or both was investigated by entering subjects' asymmetry scores from visual half-field and dichotic listening tasks into a principal component analysis. The results of this analysis indicate that there are both modality specific and modality general influences on subjects' characteristic perceptual asymmetries.     

Transfer across modality in perceptual implicit memory

We examined whether words studied in one modality (visual or auditory) would prime performance in the opposite modality in five different perceptual implicit memory tests: auditory perceptual identification, auditory stem completion, visual perceptual identification, visual stem completion, and visual fragment completion. Significant transfer across modality was observed in all five tasks. However, a large proportion of the subjects reported using explicit retrieval strategies during the implicit tests. Those subjects who claimed not to have used explicit retrieval processes during the test phase demonstrated transfer across modalities in the stem completion tests and the perceptual identification tests, but not in the fragment completion test. The results indicate that implicit visual word-fragment completion is unique, in the sense that it relies exclusively on perceptual memory processes, whereas the other tasks rely, in part, on nonperceptual memory processes.     

Effect of practice on the perception of temporal patterns

The effect of practice on the organization and the rate of identification of temporal patterns was investigated. The patterns involved a set of eight dichotomous left-right elements, repeated without interruption until the subject was able to identify the patterns. The patterns were presented in the auditory, tactual, or visual modalities, or in pairs of these modalities. In some conditions, all the pattern elements were presented in one modality; in other conditions, four elements were presented in one modality and the remaining four elements were presented in the second modality.

The results demonstrated that when all pattern elements were presented in one modality, naive subjects organized the sequence into a well-structured pattern; practiced subjects organized the sequence into a pattern beginning at the starting element. These patterns may be poorly-structured. When four pattern elements were presented in each of two modalities, naive subjects organized the elements in each modality separately; practiced subjects disregarded the modality structure and organized the sequence into a well-structured pattern.

These changes in organization suggest a hierarchy of perceptual modes; perception by modality (i.e. by sensations) is least complex, perception by pattern structure is intermediate, and perception by start point is most complex. Changes in the rate of pattern identification confirm this hierarchy. Furthermore, the changes in organization and identification found for highly practiced patterns were also found for novel patterns.     

The role of preparation and cue-modality in crossmodal task switching

The present study examined the effects of cue-based preparation and cue-target modality mapping in crossmodal task switching. In two experiments, we randomly presented lateralized visual and auditory stimuli simultaneously. Subjects were asked to make a left/right judgment for a stimulus in only one of the modalities. Prior to each trial, the relevant stimulus modality was indicated by a visual or auditory cue. The cueing interval was manipulated to examine preparation. In Experiment 1, we used a corresponding mapping of cue-modality and stimulus modality, whereas in Experiment 2 the mapping of cue and stimulus modalities was reversed. We found reduced modality-switch costs with a long cueing interval, showing that attention shifts to stimulus modalities can be prepared, irrespective of cue-target modality mapping. We conclude that perceptual processing in crossmodal switching can be biased in a preparatory way towards task-relevant stimulus modalities.