Sensorimotor simulations underlie conceptual representations: Modality-specific effects of prior activation

According to the perceptual symbols theory (Barsalou, 1999), sensorimotor simulations underlie the representation of concepts. Simulations are componential in the sense that they vary with the context in which the concept is presented. In the present study, we investigated whether representations are affected by recent experiences with a concept. Concept names (e.g., APPLE) were presented twice in a property verification task with a different property on each occasion. The two properties were either from the same perceptual modality (e.g.,green, shiny) or from different modalities (e.g.,tart, shiny). All stimuli were words. There was a lag of several intervening trials between the first and second presentation. Verification times and error rates for the second presentation of the concept were higher if the properties were from different modalities than if they were from the same modality.     

Sensory load incurs conceptual processing costs

Theories of grounded cognition propose that modal simulations underlie cognitive representation of concepts [Barsalou, L. W. (1999). Perceptual symbol systems. Behavioral and Brain Sciences, 22(4), 577-660; Barsalou, L. W. (2008). Grounded cognition. Annual Review of Psychology, 59, 617-645]. Based on recent evidence of modality-specific resources in perception, we hypothesized that verifying properties of concepts encoded in different modalities are hindered more by perceptual short-term memory load to the same versus different sensory modality as that used to process the property. We manipulated load to visual and auditory modalities by having participants store one or three items in short-term memory during property verification. In the high (but not low) load condition, property verification took longer when the property (e.g., yellow) involved the same modality as that used by the memory load (e.g., pictures). Interestingly, similar interference effects were obtained on the conceptual verification and on the memory task. These findings provide direct support for the view that conceptual processing relies on simulation in modality-specific systems.     

The situated nature of concepts

For decades the importance of background situations has been documented across all areas of cognition. Nevertheless, theories of concepts generally ignore background situations, focusing largely on bottom-up, stimulus-based processing. Furthermore, empirical research on concepts typically ignores background situations, not incorporating them into experimental designs. A selective review of relevant literatures demonstrates that concepts are not abstracted out of situations but instead are situated. Background situations constrain conceptual processing in many tasks (e.g., recall, recognition, categorization, lexical decision, color naming, property verification, property generation) across many areas of cognition (e.g., episodic memory, conceptual processing, visual object recognition, language comprehension). A taxonomy of situations is proposed in which grain size, meaningfulness, and tangibility distinguish the cumulative situations that structure cognition hierarchically.     

Verifying different-modality properties for concepts produces switching costs

According to perceptual symbol systems, sensorimotor simulations underlie the representation of concepts. It follows that sensorimotor phenomena should arise in conceptual processing. Previous studies have shown that switching from one modality to another during perceptual processing incurs a processing cost. If perceptual simulation underlies conceptual processing, then verifying the properties of concepts should exhibit a switching cost as well. For example, verifying a property in the auditory modality (e.g., BLENDER-loud) should be slower after verifying a property in a different modality (e.g., CRANBERRIES-tart) than after verifying a property in the same modality (e.g., LEAVES-rustling). Only words were presented to subjects, and there were no instructions to use imagery. Nevertheless, switching modalities incurred a cost, analogous to the cost of switching modalities in perception. A second experiment showed that this effect was not due to associative priming between properties in the same modality. These results support the hypothesis that perceptual simulation underlies conceptual processing.     

Modality switching costs emerge in concept creation as well as retrieval

Theories of embodied cognition hold that the conceptual system uses perceptual simulations for the purposes of representation. A strong prediction is that perceptual phenomena should emerge in conceptual processing, and, in support, previous research has shown that switching modalities from one trial to the next incurs a processing cost during conceptual tasks. However, to date, such research has been limited by its reliance on the retrieval of familiar concepts. We therefore examined concept creation by asking participants to interpret modality-specific compound phrases (i.e., conceptual combinations). Results show that modality switching costs emerge during the creation of new conceptual entities: People are slower to simulate a novel concept (e.g., auditory jingling onion) when their attention has already been engaged by a different modality in simulating a familiar concept (e.g., visual shiny penny). Furthermore, these costs cannot be accounted for by linguistic factors alone. Rather, our findings support the embodied view that concept creation, as well as retrieval, requires situated perceptual simulation.     

Influence of prior events on cognitive judgments in amnesia.

Amnesic patients and control Ss read the names of famous and nonfamous persons. Subsequently, both groups were more likely to designate a name as famous if it had been encountered previously. The facilitatory effect of prior presentation was similar for amnesic patients and control Ss and similar for famous and nonfamous names. For amnesic patients, the effect occurred despite severely impaired recognition memory for the names. In a 2nd experiment, recombining the first and last names that had been presented together did not diminish the facilitatory effect of prior presentation, which indicates that the effect does not depend on forming an association between first and last names. The results show that nondeclarative (implicit) memory can support the acquisition of information that is specific (e.g., names of persons) and that has no preexisting representation (e.g., nonfamous names).     

Remembering the past and imagining the future: a role for nonvisual imagery in the everyday cognition of blind and sighted people

Images can be generated in all sensory modalities. Nevertheless, research on the everyday use of mental imagery, for example in autobiographical memory tasks, has suggested that it is only visual images that facilitate memory retrieval (e.g., Williams, Healy, & Ellis, 1999). If this is the case, individuals born without sight may be forced to rely more on verbal encoding (Goddard & Pring, 2001). This paper explores the presence and everyday role of nonvisual sensory imagery in 16 individuals with and 16 without sight. Using a cue word paradigm, contrary to previous research, Experiment 1 suggested that for both blind and sighted people, nonvisual imageries have a significant role to play in the generation of autobiographical memories. These results were reinforced by similar findings in Experiment 2, which used the same cue word method to explore the role of visual and nonvisual (auditory) imagery when generating future events. The results refute the claim that "useful" imagery in everyday tasks is exclusively visual.     

Representing properties locally

Theories of knowledge such as feature lists, semantic networks, and localist neural nets typically use a single global symbol to represent a property that occurs in multiple concepts. Thus, a global symbol represents mane across HORSE, PONY, and LION. Alternatively, perceptual theories of knowledge, as well as distributed representational systems, assume that properties take different local forms in different concepts. Thus, different local forms of mane exist for HORSE, PONY, and LION, each capturing the specific form that mane takes in its respective concept. Three experiments used the property verification task to assess whether properties are represented globally or locally (e.g., Does a PONY have mane?). If a single global form represents a property, then verifying it in any concept should increase its accessibility and speed its verification later in any other concept. Verifying mane for PONY should benefit as much from having verified mane for LION earlier as from verifying mane for HORSE. If properties are represented locally, however, verifying a property should only benefit from verifying a similar form earlier. Verifying mane for PONY should only benefit from verifying mane for HORSE, not from verifying mane for LION. Findings from three experiments strongly supported local property representation and ruled out the interpretation that object similarity was responsible (e.g., the greater overall similarity between HORSE and PONY than between LION and PONY). The findings further suggest that property representation and verification are complicated phenomena, grounded in sensory-motor simulations.     

Looking at anything that is green when hearing “frog”: How object surface colour and stored object colour knowledge influence language-mediated overt attention

Three eye-tracking experiments investigated the influence of stored colour knowledge, perceived surface colour, and conceptual category of visual objects on language-mediated overt attention. Participants heard spoken target words whose concepts are associated with a diagnostic colour (e.g., “spinach”; spinach is typically green) while their eye movements were monitored to (a) objects associated with a diagnostic colour but presented in black and white (e.g., a black-and-white line drawing of a frog), (b) objects associated with a diagnostic colour but presented in an appropriate but atypical colour (e.g., a colour photograph of a yellow frog), and (c) objects not associated with a diagnostic colour but presented in the diagnostic colour of the target concept (e.g., a green blouse; blouses are not typically green). We observed that colour-mediated shifts in overt attention are primarily due to the perceived surface attributes of the visual objects rather than stored knowledge about the typical colour of the object. In addition our data reveal that conceptual category information is the primary determinant of overt attention if both conceptual category and surface colour competitors are copresent in the visual environment.     

Time course of melody recognition: a gating paradigm study

Recognizing a well-known melody (e.g., one's national anthem) is not an all-or-none process. Instead, recognition develops progressively while the melody unfolds over time. To examine which factors govern the time course of this recognition process, the gating paradigm, initially designed to study auditory word recognition, was adapted to music. Musicians and nonmusicians were presented with segments of increasing duration of familiar and unfamiliar melodies (i.e., the first note, then the first two notes, then the first three notes, and so forth). Recognition was assessed after each segment either by requiring participants to provide a familiarity judgment (Experiment 1) or by asking them to sing the melody that they thought had been presented (Experiment 2). In general, the more familiar the melody, the fewer the notes required for recognition. Musicians judged music's familiarity within fewer notes than did nonmusicians, whereas the reverse situation (i.e., musicians were slower than nonmusicians) occurred when a sung response was requested. However, both musicians and nonmusicians appeared to segment melodies into the same perceptual units (i.e., motives) in order to access the correct representation in memory. These results are interpreted in light of the cohort model (Marslen-Wilson, 1987), as applied to the music domain.     

An asymmetric effect of relational integration on recognition memory

Two experiments assessed whether conceptual relations (e.g., contains: COOKIE JAR) facilitate the retrieval of concepts (e.g., COOKIE and JAR) from long-term memory. The CARIN model of nominal combination asserts that conceptual relations are represented with and selected from the modifier noun (COOKIE). Thus, the model predicts that relational integration will facilitate memory for modifiers but not for head nouns (JAR). In Experiment 1, concepts were studied in pairs and were then tested individually. Recognition accuracy was higher for modifiers than for head nouns. In Experiment 2, the studied concepts (e.g., COOKIE JAR) were tested in the context of a new pair that instantiated either the same relation (e.g., COOKIE plate) or a different relation (e.g., COOKIE crumb). Recognition was again better for modifiers than for head nouns, but only when the same conceptual relation was instantiated at both study and test. Thus, results indicate that conceptual relations (a) facilitate recognition memory, and (b) are associated more strongly with the modifier than with the head noun.     

Guided search through memory

In “hybrid” search, observers search a visual display for any of several targets held in memory. It is known that the contents of the memory set can guide visual search (e.g., if the memorized targets are all animals, visual attention can be guided away from signs). It is not known if the visual display can guide memory search (e.g., if the memory set is composed of signs and animals, can a visual display of signs restrict memory search to just the signs?). In three hybrid search experiments, participants memorized sets of items that belonged to either one or several categories. Participants were then presented with visual displays containing multiple items, also drawn from one or several categories. Participants were asked to determine if any of the items from their current memory set were present in the visual display. We replicate the finding that visual search can be guided by the contents of memory. We find weaker, novel evidence that memory search can be guided by the contents of the visual display.     

Phonological and visual working memory in mental addition

The goal of the present research was to examine the role of working memory in mental arithmetic. Adults (n = 96) solved multidigit arithmetic problems (e.g., 52 + 3; 3 + 52) alone and in combination with either a phonological memory load (i.e., nonwords, such as gup) or a visual memory load (i.e., random pattern of asterisks). The participants solved problems presented in a vertical format significantly faster than problems presented in a horizontal format. They also solved double digit first problems (e.g., 52 + 3) more quickly than the reverse (e.g., 3 + 52), but only when the problems were presented horizontally. Performance was worse in the phonological load condition than in the visual load condition for the participants who solved problems presented horizontally, whereas performance was worse in the visual load condition than in the phonological load condition when problems were presented vertically. The present research provides evidence that both phonological and visual aspects of working memory are involved in mental arithmetic but that the role of each working memory component will depend on such factors as presentation format.     

Recognition Practice Results in a Generalizable Skill in Older Adults: Decreased Intrusion Errors to Novel Objects Belonging to Practiced Categories

Accessing memories is often accompanied by both positive and negative consequences. For example, practice recognizing some visual images held in memory can improve memory for the practiced images and hurt memory for related images (i.e., recognition‐induced forgetting). However, visual stimuli have been shown to improve memory for older adults by decreasing false memories. This suggests that older adults may be immune to recognition‐induced forgetting and that recognition practice may decrease susceptibility to intrusion errors. We first tested the hypothesis that older adults are immune to recognition‐induced forgetting. We found older adults exhibit recognition‐induced forgetting. Next, we tested the hypothesis that recognition practice decreases older adult's rates of intrusion errors. We found lower intrusion errors for novel objects from practiced categories. This represents a generalizable learning effect; practice recognizing a target object (e.g., your pill bottle) improves the rejection of new lures (e.g., identifying the pill bottle that is not yours).Copyright © 2016 John Wiley & Sons, Ltd.     

Function follows form: activation of shape and function features during object identification

Most theories of semantic memory characterize knowledge of a given object as comprising a set of semantic features. But how does conceptual activation of these features proceed during object identification? We present the results of a pair of experiments that demonstrate that object recognition is a dynamically unfolding process in which function follows form. We used eye movements to explore whether activating one object's concept leads to the activation of others that share perceptual (shape) or abstract (function) features. Participants viewed 4-picture displays and clicked on the picture corresponding to a heard word. In critical trials, the conceptual representation of 1 of the objects in the display was similar in shape or function (i.e., its purpose) to the heard word. Importantly, this similarity was not apparent in the visual depictions (e.g., for the target Frisbee, the shape-related object was a triangular slice of pizza, a shape that a Frisbee cannot take); preferential fixations on the related object were therefore attributable to overlap of the conceptual representations on the relevant features. We observed relatedness effects for both shape and function, but shape effects occurred earlier than function effects. We discuss the implications of these findings for current accounts of the representation of semantic memory.     

Priming contour-deleted images: evidence for intermediate representations in visual object recognition

The speed and accuracy of perceptual recognition of a briefly presented picture of an object is facilitated by its prior presentation. Picture priming tasks were used to assess whether the facilitation is a function of the repetition of: (a) the object's image features (viz., vertices and edges), (b) the object model (e.g., that it is a grand piano), or (c) a representation intermediate between (a) and (b) consisting of convex or singly concave components of the object, roughly corresponding to the object's parts. Subjects viewed pictures with half their contour removed by deleting either (a) every other image feature from each part, or (b) half the components. On a second (primed) block of trials, subjects saw: (a) the identical image that they viewed on the first block, (b) the complement which had the missing contours, or (c) a same name-different exemplar of the object class (e.g., a grand piano when an upright piano had been shown on the first block). With deletion of features, speed and accuracy of naming identical and complementary images were equivalent, indicating that none of the priming could be attributed to the features actually present in the image. Performance with both types of image enjoyed an advantage over that with the different exemplars, establishing that the priming was visual rather than verbal or conceptual. With deletion of the components, performance with identical images was much better than that with their complements. The latter were equivalent to the different exemplars, indicating that all the visual priming of an image of an object is through the activation of a representation of its components in specified relations. In terms of a recent neural net implementation of object recognition (Hummel & Biederman, in press), the results suggest that the locus of object priming may be at changes in the weight matrix for a geon assembly layer, where units have self-organized to represent combinations of convex or singly concave components (or geons) and their attributes (e.g., aspect ratio, orientation, and relations with other geons such as TOP-OF). The results of these experiments provide evidence for the psychological reality of intermediate representations in real-time visual object recognition.     

Divided attention: Storing and classifying briefly presented objects

Although in many studies divided attention has been examined by having people perform the same task (e.g., report or search) with a large number of objects, in few studies have people had to perform two logically independent tasks involving the same brief display. In two experiments, subjects saw 200-msec arrays of characters. In dual-task blocks, theyclassified the color of some or all of the items (making an immediate response) andstored the shape of some of the items for a later recognition test. There was not much mutual interference between classifying and storing per se. However, the tasks were by no means independent: there was substantial interference whendifferent objects from the array had to be stored for one task and classified for the other. The results confirm that wholly unrelated visual tasks depend on the same input-attention system and suggest that attending to an object for any purpose may entail storing a representation of it in visual short-term memory.     

多视点学习条件下空间表征的朝向特异性

让被试在没有外界参照线索的条件下,从三个观察视点学习物体场景,首次以局部场景再认范式探讨了空间记忆中场景表征的朝向特异性。结果表明：（1）多视点学习条件下,场景空间表征依然是依赖于特定朝向的,不支持空间表征独立于朝向和空间表征依赖于多个观察朝向的观点。（2）场景空间表征的朝向依赖性不仅表现于方位判断任务,也同样表现于场景再认任务。（3）场景本身的内在结构对空间表征中参照系的选取有重要影响。     

Divided attention and prerecognition processing of spoken words and nonwords

Three recognition memory experiments examined phonemic similarity and false recognition under conditions of divided attention. The manipulation was presumed to have little effect on automatic, perceptual influences of memory. Prior research demonstrated that false recognition of a test word (e.g., discrepancy) was higher if the study list included a nonword derived from the future test word by changing a phoneme near the end of the item (e.g., discrepan/l/y) relative to an early phoneme change (e.g., /l/iscrepancy). The difference has been attributed to automatic, implicit activation of test words during prerecognition processing of related nonwords. Three experiments demonstrated that the late-change condition also contributed to higher false recognition rates with divided attention at encoding. Dividing attention disrupted recognition memory of studied words in Experiments 1 and 3. Results are discussed in terms of their relevance for an interpretation emphasizing the automatic, implicit activation of candidate words that occurs in the course of identifying spoken words and nonwords.     

Modeling recognition memory using the similarity structure of natural input

The natural input memory (NIM) model is a new model for recognition memory that operates on natural visual input. A biologically informed perceptual preprocessing method takes local samples (eye fixations) from a natural image and translates these into a feature-vector representation. During recognition, the model compares incoming preprocessed natural input to stored representations. By complementing the recognition memory process with a perceptual front end, the NIM model is able to make predictions about memorability based directly on individual natural stimuli. We demonstrate that the NIM model is able to simulate experimentally obtained similarity ratings and recognition memory for individual stimuli (i.e., face images).