Perceptual processing affects conceptual processing

According to the Perceptual Symbols Theory of cognition ( Barsalou, 1999 ), modality-specific simulations underlie the representation of concepts. A strong prediction of this view is that perceptual processing affects conceptual processing. In this study, participants performed a perceptual detection task and a conceptual property-verification task in alternation. Responses on the property-verification task were slower for those trials that were preceded by a perceptual trial in a different modality than for those that were preceded by a perceptual trial in the same modality. This finding of a modality-switch effect across perceptual processing and conceptual processing supports the hypothesis that perceptual and conceptual representations are partially based on the same systems.     

Modality exclusivity norms for 423 object properties

Recent work has shown that people routinely use perceptual information during language comprehension and conceptual processing, from single-word recognition to modality-switching costs in property verification. In investigating such links between perceptual and conceptual representations, the use of modality-specific stimuli plays a central role. To aid researchers working in this area, we provide a set of norms for 423 adjectives, each describing an object property, with mean ratings of how strongly that property is experienced through each of five perceptual modalities (visual, haptic, auditory, olfactory, and gustatory). The data set also contains estimates of modality exclusivity—that is, a measure of the extent to which a particular property may be considered unimodal (i.e., perceived through one sense alone). Although there already exists a number of sets of word and object norms, we provide the first set to categorize words describing object properties along the dimensions of the five perceptual modalities. We hope that the norms will be of use to researchers working at the interface between linguistic, conceptual, and perceptual systems. The modality exclusivity norms may be downloaded as supplemental materials for this article from brm.psychonomic-journals.org/ content/supplemental.     

Sensory load incurs conceptual processing costs

Theories of grounded cognition propose that modal simulations underlie cognitive representation of concepts [Barsalou, L. W. (1999). Perceptual symbol systems. Behavioral and Brain Sciences, 22(4), 577-660; Barsalou, L. W. (2008). Grounded cognition. Annual Review of Psychology, 59, 617-645]. Based on recent evidence of modality-specific resources in perception, we hypothesized that verifying properties of concepts encoded in different modalities are hindered more by perceptual short-term memory load to the same versus different sensory modality as that used to process the property. We manipulated load to visual and auditory modalities by having participants store one or three items in short-term memory during property verification. In the high (but not low) load condition, property verification took longer when the property (e.g., yellow) involved the same modality as that used by the memory load (e.g., pictures). Interestingly, similar interference effects were obtained on the conceptual verification and on the memory task. These findings provide direct support for the view that conceptual processing relies on simulation in modality-specific systems.     

Modality exclusivity norms for 400 nouns: The relationship between perceptual experience and surface word form

We present modality exclusivity norms for 400 randomly selected noun concepts, for which participants provided perceptual strength ratings across five sensory modalities (i.e., hearing, taste, touch, smell, and vision). A comparison with previous norms showed that noun concepts are more multimodal than adjective concepts, as nouns tend to subsume multiple adjectival property concepts (e.g., perceptual experience of the concept baby involves auditory, haptic, olfactory, and visual properties, and hence leads to multimodal perceptual strength). To show the value of these norms, we then used them to test a prediction of the sound symbolism hypothesis: Analysis revealed a systematic relationship between strength of perceptual experience in the referent concept and surface word form, such that distinctive perceptual experience tends to attract distinctive lexical labels. In other words, modality-specific norms of perceptual strength are useful for exploring not just the nature of grounded concepts, but also the nature of form–meaning relationships. These norms will be of benefit to those interested in the representational nature of concepts, the roles of perceptual information in word processing and in grounded cognition more generally, and the relationship between form and meaning in language development and evolution.     

Concepts are not represented by conscious imagery

According to theories of grounded cognition, conceptual representation and perception share processing mechanisms. We investigated whether this overlap is due to conscious perceptual imagery. Participants filled out questionnaires to assess the vividness of their imagery (Questionnaire on Mental Imagery) and the extent to which their imagery was object oriented and spatially oriented (Object-Spatial Imagery Questionnaire), and they performed a mental rotation task. One week later, they performed a verbal property verification task. In this task, involvement of modality-specific systems is evidenced by the modality-switch effect, the finding that performance on a target trial (e.g., apple—green) is better after a same-modality trial (e.g., diamond—sparkle) than after a different-modality trial (e.g., airplane—noisy). Results showed a modality-switch effect, but there was no systematic relation between imagery scores and modality switch. We conclude that conscious mental imagery is not fundamental to conceptual representation.     

Modality switching costs emerge in concept creation as well as retrieval

Theories of embodied cognition hold that the conceptual system uses perceptual simulations for the purposes of representation. A strong prediction is that perceptual phenomena should emerge in conceptual processing, and, in support, previous research has shown that switching modalities from one trial to the next incurs a processing cost during conceptual tasks. However, to date, such research has been limited by its reliance on the retrieval of familiar concepts. We therefore examined concept creation by asking participants to interpret modality-specific compound phrases (i.e., conceptual combinations). Results show that modality switching costs emerge during the creation of new conceptual entities: People are slower to simulate a novel concept (e.g., auditory jingling onion) when their attention has already been engaged by a different modality in simulating a familiar concept (e.g., visual shiny penny). Furthermore, these costs cannot be accounted for by linguistic factors alone. Rather, our findings support the embodied view that concept creation, as well as retrieval, requires situated perceptual simulation.     

Verifying different-modality properties for concepts produces switching costs

According to perceptual symbol systems, sensorimotor simulations underlie the representation of concepts. It follows that sensorimotor phenomena should arise in conceptual processing. Previous studies have shown that switching from one modality to another during perceptual processing incurs a processing cost. If perceptual simulation underlies conceptual processing, then verifying the properties of concepts should exhibit a switching cost as well. For example, verifying a property in the auditory modality (e.g., BLENDER-loud) should be slower after verifying a property in a different modality (e.g., CRANBERRIES-tart) than after verifying a property in the same modality (e.g., LEAVES-rustling). Only words were presented to subjects, and there were no instructions to use imagery. Nevertheless, switching modalities incurred a cost, analogous to the cost of switching modalities in perception. A second experiment showed that this effect was not due to associative priming between properties in the same modality. These results support the hypothesis that perceptual simulation underlies conceptual processing.     

Verifying properties of concepts spontaneously requires sharing resources with same-modality percept

In the present experiments, participants had to verify properties of concepts but, depending on the trial condition, concept-property pairs were presented via headphones or on the screen. The results showed that participants took longer and were less accurate at verifying conceptual properties when the channel used to present the CONCEPT-property pair and the type of property matched in sensory modality (e.g., LEMON-yellow on screen; BLENDER-loud in headphones) compared to when properties and channel did not match (e.g., LEMON-yellow in headphones; BLENDER-loud on screen). Such interference is consistent with theories of embodied cognition holding that knowledge is grounded in modality-specific systems (Barsalou in Behav Brain Sci 22:577–660, 1999). When the resources of one modality are burdened during the task, processing costs are incurred in a conceptual task (Vermeulen et al. in Cognition 109:287–294, 2008).     

The Toronto Word Pool: Norms for imagery,concreteness, orthographic variables,and grammatical usage for 1,080 words

Imagery and concreteness norms and percentage noun usage were obtained on the 1,080 verbal items from the Toronto Word Pool. Imagery was defined as the rated ease with which a word aroused a mental image, and concreteness was defined in relation to level of abstraction. The degree to which a word was functionally a noun was estimated in a sentence generation task. The mean and standard deviation of the imagery and concreteness ratings for each item are reported together with letter and printed frequency counts for the words and indications of sex differences in the ratings. Additional data in the norms include a grammatical function code derived from dictionary definitions, a percent noun judgment, indexes of statistical approximation to English, and an orthographic neighbor ratio. Validity estimates for the imagery and concreteness ratings are derived from comparisons with scale values drawn from the Paivio, Yuille, and Madigan (1968) noun pool and the Toglia and Battig (1978) norms.     

Modality-specific auditory and visual temporal processing deficits

We studied the attentional blink (AB) and the repetition blindness (RB) effects using an audio-visual presentation procedure designed to overcome several potential methodological confounds in previous cross-modal research. In Experiment 1, two target digits were embedded amongst letter distractors in two concurrent streams (one visual and the other auditory) presented from the same spatial location. Targets appeared in either modality unpredictably at different temporal lags, and the participants' task was to recall the digits at the end of the trial. We evaluated both AB and RB for pairs of targets presented in either the same or different modalities. Under these conditions both AB and RB were observed in vision, AB but not RB was observed in audition, and there was no evidence of AB or RB cross-modally from audition to vision or vice versa. In Experiment 2, we further investigated the AB by including Lag 1 items and observed Lag 1 sparing, thus ruling out the possibility that the observed effects were due to perceptual and/or conceptual masking. Our results support a distinction between a modality-specific interference at the attentional selection stage and a modality-independent interference at later processing stages. They also provide a new dissociation between the AB and RB.     

Category typicality effects in children's memory performance: Qualitative and quantitative differences in the processing of category information

Kindergarten, third, and sixth graders (6, 9, and 12 years of age, respectively) received a cue-at-input/cue-at-output recall task, using category typical and atypical items that were based on either (1) children's conceptions of item typicality, or (2) adults' conceptions of item typicality. At each grade level, recall was greater with the child-defined lists than with the adult-defined lists, and typical items were recalled to a greater extent than atypical items. Further analyses items were recalled to a greater extent than atypical items. Further analyses revealed that the recall of typical items varied as a function of children's typicality ratings of items, and that the “typicality effect” in the adult-norm condition was due primarily to the childrennot realizing that many of the atypical items were appropriate category exemplars. In contrast, typicality effects in the child-norm condition were attributed to qualitative differences in the judged “goodness of example” of the typical and atypical items. The results were discussed in terms of the appropriateness of typicality as a dimension of children's natural language concepts, the role of age differences in knowledge base in affecting performance on a cognitive task, and of the importance of using child-generated norms in studies of children's processing of category information.     

Language Processing Differences Between Blind and Sighted Individuals and the Abstract Versus Concrete Concept Difference

In the property listing task (PLT), participants are asked to list properties for a concept (e.g., for the concept dog, “barks,” and “is a pet” may be produced). In conceptual property norming (CPNs) studies, participants are asked to list properties for large sets of concepts. Here, we use a mathematical model of the property listing process to explore two longstanding issues: characterizing the difference between concrete and abstract concepts, and characterizing semantic knowledge in the blind versus sighted population. When we apply our mathematical model to a large CPN reporting properties listed by sighted and blind participants, the model uncovers significant differences between concrete and abstract concepts. Though we also find that blind individuals show many of the same processing differences between abstract and concrete concepts found in sighted individuals, our model shows that those differences are noticeably less pronounced than in sighted individuals. We discuss our results vis-a-vis theories attempting to characterize abstract concepts.     

Multisensory executive functioning

To better understand the prefrontal circuitry that putatively supports executive functions, such as those involved in switching tasks, we asked whether a current task set is open equally to receiving information from any sensory modality or if it is to some degree modality-specific. Subjects were presented with a sequence of digits to be classified as either odd/even or greater/less than five. The digits were either auditory or visual, with the modality varying randomly. Results demonstrated a reaction time (RT) cost associated with switching between tasks and also an unexpected cost of switching between modalities. When both modality and task switched, the two costs were greater than either of the costs alone, but significantly less than predicted simply by summing the two costs together (i.e., they were underadditive). These data indicate that the frontal mechanisms that allow for a switch in task are only partially modality-specific. Current theories of executive function must be adapted to account for this finding. We also suggest that the present paradigm is amenable to future research aimed at determining precisely how modalities are linked within a task set.     

Information-processing modules and their relative modality specificity

This research uses fMRI to understand the role of eight cortical regions in a relatively complex information-processing task. Modality of input (visual versus auditory) and modality of output (manual versus vocal) are manipulated. Two perceptual regions (auditory cortex and fusiform gyrus) only reflected perceptual encoding. Two motor regions were involved in information rehearsal as well as programming of overt actions. Two cortical regions (parietal and prefrontal) performed processing (retrieval and representational change) independent of input and output modality. The final two regions (anterior cingulate and caudate) were involved in control of cognition independent of modality of input or output and content of the material. An information-processing model, based on the ACT-R theory, is described that predicts the BOLD response in these regions. Different modules in the theory vary in the degree to which they are modality-specific and the degree to which they are involved in central versus peripheral cognitive processes.     

Switching between sensory and affective systems incurs processing costs

Recent models of the conceptual system hold that concepts are grounded in simulations of actual experiences with instances of those concepts in sensory-motor systems (e.g., Barsalou, 1999, 2003; Solomon & Barsalou, 2001). Studies supportive of such a viewhave shown that verifying a property of a concept in one modality, and then switching to verify a property of a different concept in a different modality generates temporal processing costs similar to the cost of switching modalities in perception. In addition to non-emotional concepts, the present experiment investigated switching costs in verifying properties of positive and negative (emotional) concepts. Properties of emotional concepts were taken from vision, audition, and the affective system. Parallel to switching costs in neutral concepts, the study showed that for positive and negative concepts, verifying properties from different modalities produced processing costs such that reaction times were longer and error rates were higher. Importantly, this effect was observed when switching from the affective system to sensory modalities, and vice-versa. These results support the embodied cognition view of emotion in humans.     

Levels-of-processing effects in subject-performed tasks

In memory for subject-performed tasks (SPTs), subjects encode a list of simple action phrases (e.g., thumb through a book, knock at the door) by performing these actions during learning. In three experiments, we investigated the size of the levels-of-processing effects in SPTs as compared with those in standard verbal learning tasks (VTs). Subjects under SPT and VT conditions learned lists of action phrases in a surface or a conceptual orienting task. Under both encoding conditions, the subjects recalled fewer items with surface orienting tasks than with conceptual orienting tasks, but the levels-of-processing effects were strongly reduced in the SPT condition. In the SPT condition, items that were encoded in a surface orienting task were still substantially recalled. The items were recalled almost as well as the conceptually encoded items in the VT condition. The distinct reduction of the levels-of-processing effect is caused by the fact that, in SPT encoding even with a verbal surface orienting task, subjects process conceptual information in order to perform the denoted action. We attribute the small conceptual advantage, which remains with SPT despite the conceptual processing for performing, to the fact that items are not as well integrated into memory as they are when conceptual processing is focused on the action component, rather than on the semantic contexts. This lower integration reduces the accessibility of items in the verbal surface task, even with SPT encoding.     

The Multilevel Modality-Switch Effect: What Happens When We See the Bees Buzzing and Hear the Diamonds Glistening

Previous studies demonstrated that the sequential verification of different sensory modality properties for concepts (e.g., BLENDER-loud; BANANA-yellow) brings about a processing cost, known as the modality-switch effect. We report an experiment designed to assess the influence of the mode of presentation (i.e., visual, aural) of stimuli on the modality-switch effect in a property verification and lexical decision priming paradigm. Participants were required to perform a property verification or a lexical decision task on a target sentence (e.g., “a BEE buzzes”, “a DIAMOND glistens”) presented either visually or aurally after having been presented with a prime sentence (e.g., “the LIGHT is flickering”, “the SOUND is echoing”) that could either share both, one or none of the target’s mode of presentation and content modality. Results show that the mode of presentation of stimuli affects the conceptual modality-switch effect. Furthermore, the depth of processing required by the task modulates the complex interplay of perceptual and semantic information. We conclude that the MSE is a task-related, multilevel effect which can occur on two different levels of information processing (i.e., perceptual and semantic).     

Automatic Extraction of Property Norm‐Like Data From Large Text Corpora

Traditional methods for deriving property‐based representations of concepts from text have focused on either extracting only a subset of possible relation types, such as hyponymy/hypernymy (e.g., car is‐a vehicle ) or meronymy/metonymy (e.g., car has wheels ), or unspecified relations (e.g., car — petrol ). We propose a system for the challenging task of automatic, large‐scale acquisition of unconstrained, human‐like property norms from large text corpora, and discuss the theoretical implications of such a system. We employ syntactic, semantic, and encyclopedic information to guide our extraction, yielding concept‐relation‐feature triples (e.g., car be fast , car require petrol , car cause pollution ), which approximate property‐based conceptual representations. Our novel method extracts candidate triples from parsed corpora (Wikipedia and the British National Corpus) using syntactically and grammatically motivated rules, then reweights triples with a linear combination of their frequency and four statistical metrics. We assess our system output in three ways: lexical comparison with norms derived from human‐generated property norm data, direct evaluation by four human judges, and a semantic distance comparison with both WordNet similarity data and human‐judged concept similarity ratings. Our system offers a viable and performant method of plausible triple extraction: Our lexical comparison shows comparable performance to the current state‐of‐the‐art, while subsequent evaluations exhibit the human‐like character of our generated properties.