Mental representation in visual/haptic crossmodal memory: evidence from interference effects

Two experiments used visual-, verbal-, and haptic-interference tasks during encoding (Experiment 1) and retrieval (Experiment 2) to examine mental representation of familiar and unfamiliar objects in visual/haptic crossmodal memory. Three competing theories are discussed, which variously suggest that these representations are: (a) visual; (b) dual-code—visual for unfamiliar objects but visual and verbal for familiar objects; or (c) amodal. The results suggest that representations of unfamiliar objects are primarily visual but that crossmodal memory for familiar objects may rely on a network of different representations. The pattern of verbal-interference effects suggests that verbal strategies facilitate encoding of unfamiliar objects regardless of modality, but only haptic recognition regardless of familiarity. The results raise further research questions about all three theoretical approaches.     

Mental representation in visual/haptic crossmodal memory: evidence from interference effects

Two experiments used visual-, verbal-, and haptic-interference tasks during encoding (Experiment 1) and retrieval (Experiment 2) to examine mental representation of familiar and unfamiliar objects in visual/haptic crossmodal memory. Three competing theories are discussed, which variously suggest that these representations are: (a) visual; (b) dual-code—visual for unfamiliar objects but visual and verbal for familiar objects; or (c) amodal. The results suggest that representations of unfamiliar objects are primarily visual but that crossmodal memory for familiar objects may rely on a network of different representations. The pattern of verbal-interference effects suggests that verbal strategies facilitate encoding of unfamiliar objects regardless of modality, but only haptic recognition regardless of familiarity. The results raise further research questions about all three theoretical approaches.     

Fluency does not express implicit knowledge of artificial grammars

It is commonly held that implicit knowledge expresses itself as fluency. A perceptual clarification task was used to examine the relationship between perceptual processing fluency, subjective familiarity, and grammaticality judgments in a task frequently used to produce implicit knowledge, artificial grammar learning (AGL). Four experiments examined the effects of naturally occurring differences and manipulated differences in perceptual fluency, where decisions were based on a brief exposure to test-strings (during the clarification task only) or normal exposure. When perceptual fluency was not manipulated, it was weakly related to familiarity and grammaticality judgments, but unrelated to grammatical status and hence not a source of accuracy. Counterbalanced grammatical and ungrammatical strings did not differ in perceptual fluency but differed substantially in subjective familiarity. When fluency was manipulated, faster clarifying strings were rated as more familiar and were more often endorsed as grammatical but only where exposure was brief. Results indicate that subjective familiarity derived from a source other than perceptual fluency, is the primary basis for accuracy in AGL. Perceptual fluency is found to be a dumb heuristic influencing responding only in the absence of actual implicit knowledge.     

The perceptual nature of audiovisual interactions for semantic knowledge in young and elderly adults

Audiovisual interactions for familiar objects are at the core of perception. The nature of these interactions depends on the amodal – sensory abstracted – or modal – sensory-dependent – approach of knowledge. According to these approaches, the interactions should be respectively semantic and indirect or perceptual and direct. This issue is therefore a central question to memory and perception, yet the nature of these interactions remains unexplored in young and elderly adults. We used a cross-modal priming paradigm combined with a visual masking procedure of half of the auditory primes. The data demonstrated similar results in the young and elderly adult groups. The mask interfered with the priming effect in the semantically congruent condition, whereas the mask facilitated the processing of the visual target in the semantically incongruent condition. These findings indicate that audiovisual interactions are perceptual, and support the grounded cognition theory.     

Representation-saving effects of prior knowledge in memory for simple technical prose

Prior knowledge of the content of a passage should reduce the effort required to encode the passage and facilitate its recall. This paper presents such effects of prior knowledge upon comprehension and memory of simple technical prose. The basic procedure was to collect ratings of the amount of prior knowledge for individual passage sentences and individual subjects, and then to determine whether these familiarity ratings predicted study time and recall. In order to validate the rating method, and also to obtain data on prior knowledge of individual facts, subjects also completed an objective test of prior knowledge of the passage facts. Three different encoding-task conditions were used: a self-paced study task, a forced-pace study task, and an incidental-learning task. A cued-recall test followed each condition. In the self-paced task, readers studied unfamiliar material longer than familiar material, but recalled at the same level regardless of familiarity. In contrast, familiarity did predict recall in the forced-pace and incidental tasks. This task specificity is explained in terms of the subjects’ encoding strategies. The basic effect of prior knowledge can be explained not only by the elaboration principle, but also by a representation-saving principle, which is presented in a simulation model that can account for the effects of familiarity on study time.     

The role of perceptually represented structure in analogical problem solving

Current models of analogical reasoning assume that representations of source examples and target problems occur in an amodal format--that is, a representation whose construction and processing are independent of activity in the perceptual and motor cortices of the brain. We examined the possible use of kinesthetic information--perceptual structures associated with the sensation of space and force--in the representation of source examples and target problems. Participants who recreated a source story while acting out the key elements were more likely to access the story when later working on the target problem than were participants who only verbally recreated the story or who verbally recreated it as well as sketched it. We argue that enactment made kinesthetic and spatial features more salient in participants' source story representations and that this aided performance. These results suggest that current models of analogical reasoning might be improved by including perceptual information as part of their representational schemes.     

Perceptual symbol systems

Prior to the twentieth century, theories of knowledge were inherently perceptual. Since then, developments in logic, statistics, and programming languages have inspired amodal theories that rest on principles fundamentally different from those underlying perception. In addition, perceptual approaches have become widely viewed as untenable because they are assumed to implement recording systems, not conceptual systems. A perceptual theory of knowledge is developed here in the context of current cognitive science and neuroscience. During perceptual experience, association areas in the brain capture bottom-up patterns of activation in sensory-motor areas. Later, in a top-down manner, association areas partially reactivate sensory-motor areas to implement perceptual symbols. The storage and reactivation of perceptual symbols operates at the level of perceptual components--not at the level of holistic perceptual experiences. Through the use of selective attention, schematic representations of perceptual components are extracted from experience and stored in memory (e.g., individual memories of green, purr, hot). As memories of the same component become organized around a common frame, they implement a simulator that produces limitless simulations of the component (e.g., simulations of purr). Not only do such simulators develop for aspects of sensory experience, they also develop for aspects of proprioception (e.g., lift, run) and introspection (e.g., compare, memory, happy, hungry). Once established, these simulators implement a basic conceptual system that represents types, supports categorization, and produces categorical inferences. These simulators further support productivity, propositions, and abstract concepts, thereby implementing a fully functional conceptual system. Productivity results from integrating simulators combinatorially and recursively to produce complex simulations. Propositions result from binding simulators to perceived individuals to represent type-token relations. Abstract concepts are grounded in complex simulations of combined physical and introspective events. Thus, a perceptual theory of knowledge can implement a fully functional conceptual system while avoiding problems associated with amodal symbol systems. Implications for cognition, neuroscience, evolution, development, and artificial intelligence are explored.     

Scripts or scraps: reconsidering the development of sequential understanding

A growing literature attests to temporally ordered recall of events by children under 2 years of age. Other data suggest a developmental sequence wherein the ability to reproduce unfamiliar and/or arbitrarily ordered events, and familiar events in other than canonical order develops well after the first ordered productions of events. Early ordering is thus argued to be dependent upon familiarity, rather than upon general temporal principles. This suggestion was investigated by using elicited imitation to assess 21-month-olds' recall of familiar-canonical, familiar-reversed, novel-causal, and novel-arbitrary event sequences. Subjects reproduced canonical and both types of novel sequences in modeled order. On reversed sequences they vacillated between reproducing the events as modeled and "correcting" them to canonical order. The results suggest that temporal organization is not imposed upon an existing unordered event representation, but rather, is an integral aspect of the representation from its initial construction. It is suggested that young children's difficulty with reversed sequences may be attributed to a reluctance to reorganize existing representations, rather than to the absence of applicable temporal principles.     

Effect of skill level on recall of visually presented patterns of musical notes

Expertise effects in music were studied in a new task: the construction of mental representations from separate fragments. Groups of expert musicians and non-musicians were asked to recall note patterns presented visually note by note. Skill-level, musical well-formedness of the note patterns and presentation mode were varied. The musicians recalled note patterns better than the non-musicians, even though the presentation was visual and successive. Furthermore, only musicians' performance was affected by musical well-formedness of the note patterns when visual gestalt properties, verbal rehearsability, and familiarity of the stimuli were controlled. Musicians were also able to use letter names referring to notes as efficiently as visual notes, which indicates that the better recall of musicians cannot be explained by perceptual visual chunking. These results and the effect of skill level on the distribution of recall errors indicate that the ability to chunk incoming information into meaningful units does not require that complete familiar patterns are accessible to encoding processes, yet previous knowledge stored in long-term memory affects representation construction in working memory. The present method offers a new reliable tool, and its implications to the research on construction of representations and musical imagery are discussed.     

Event perception: a mind-brain perspective

People perceive and conceive of activity in terms of discrete events. Here the authors propose a theory according to which the perception of boundaries between events arises from ongoing perceptual processing and regulates attention and memory. Perceptual systems continuously make predictions about what will happen next. When transient errors in predictions arise, an event boundary is perceived. According to the theory, the perception of events depends on both sensory cues and knowledge structures that represent previously learned information about event parts and inferences about actors' goals and plans. Neurological and neurophysiological data suggest that representations of events may be implemented by structures in the lateral prefrontal cortex and that perceptual prediction error is calculated and evaluated by a processing pathway, including the anterior cingulate cortex and subcortical neuromodulatory systems.     

Something old,something new: a developmental transition from familiarity to novelty preferences with hidden objects

Novelty seeking is viewed as adaptive, and novelty preferences in infancy predict cognitive performance into adulthood. Yet 7‐month‐olds prefer familiar stimuli to novel ones when searching for hidden objects, in contrast to their strong novelty preferences with visible objects ( Shinskey & Munakata, 2005 ). According to a graded representations perspective on object knowledge, infants gradually develop stronger object representations through experience, such that representations of familiar objects can be better maintained, supporting greater search than with novel objects. Object representations should strengthen with further development to allow older infants to shift from familiarity to novelty preferences with hidden objects. The current study tested this prediction by presenting 24 11‐month‐olds with novel and familiar objects that were sometimes visible and sometimes hidden. Unlike 7‐month‐olds, 11‐month‐olds showed novelty preferences with both visible and hidden objects. This developmental shift from familiarity to novelty preference with hidden objects parallels one that infants show months earlier with perceptible stimuli, but the two transitions may reflect different underlying mechanisms. The current findings suggest both change and continuity in the adaptive development of object representations and associated cognitive processes.     

The role of familiar units in perception of words and nonwords

This paper investigates the effects of familiarity with whole-word units and letter-cluster units in perceptual encoding of letter strings. Subjects viewed brief, masked presentations of words and pronounceable pseudowords differing in letter cluster frequency. Identification of both display types was compared to control single letters. Perceptual accuracy was indexed by probe forced-choice responses and full verbal reports of the displays. Evidence that familiarity of whole-word units facilitated encoding was mixed but, on balance, favorable. Evidence that familiarity of letter-cluster units facilitated encoding was completely absent. This negative finding is surprising in view of the fact that we did obtain a large advantage of letters in pseudo-words as well as words over single letters. The discussion section considers an alternative to the view that perceivers use detectors for familiar letter-cluster units in the process of forming representations of pronounceable, orthographically regular letter strings.     

Geometric and featural representations in semantic concepts

We explore the adequacy of two types of similarity representation in the context of semantic concepts. To this end, we evaluate different categorization models, assuming either a geometric or a featural representation, using categorization decisions involving familiar and unfamiliar foods and animals. The study aims to assess the optimal stimulus representation as a function of the familiarity of the stimuli. For the unfamiliar stimuli, the geometric categorization models provide the best account of the categorization data, whereas for the familiar stimuli, the featural categorization models provide the best account. This pattern of results suggests that people rely on perceptual information to assign an unfamiliar stimulus to a category but rely on more elaborate conceptual knowledge when assigning a familiar stimulus.     

The perirhinal cortex and long-term familiarity memory.

To analyse the functions of the perirhinal cortex, the activity of single neurons in the perirhinal cortex was recorded while macaques performed a delayed matching-to-sample task with up to three intervening stimuli. Some neurons had activity related to working memory, in that they responded more to the sample than to the match image within a trial, as shown previously. However, when a novel set of stimuli was introduced, the neuronal responses were on average only 47% of the magnitude of the responses to the set of very familiar stimuli. Moreover, it was shown in three monkeys that the responses of the perirhinal cortex neurons gradually increased over hundreds of presentations (mean = 400 over 7-13 days) of the new set of (initially novel) stimuli to become as large as those to the already familiar stimuli. Thus perirhinal cortex neurons represent the very long-term familiarity of visual stimuli. Part of the impairment in temporal lobe amnesia may be related to the difficulty of building representations of the degree of familiarity of stimuli. A neural network model of how the perirhinal cortex could implement long-term familiarity memory is proposed using Hebbian associative learning.     

Audiovisual speech facilitates voice learning

In this research, we investigated the effects of voice and face information on the perceptual learning of talkers and on long-term memory for spoken words. In the first phase, listeners were trained over several days to identify voices from words presented auditorily or audiovisually. The training data showed that visual information about speakers enhanced voice learning, revealing cross-modal connections in talker processing akin to those observed in speech processing. In the second phase, the listeners completed an auditory or audiovisual word recognition memory test in which equal numbers of words were spoken by familiar and unfamiliar talkers. The data showed that words presented by familiar talkers were more likely to be retrieved from episodic memory, regardless of modality. Together, these findings provide new information about the representational code underlying familiar talker recognition and the role of stimulus familiarity in episodic word recognition.     

Using neural response properties to draw the distinction between modal and amodal representations

ABSTRACT

Barsalou has recently argued against the strategy of identifying amodal neural representations by using their cross-modal responses (i.e., their responses to stimuli from different modalities). I agree that there are indeed modal structures that satisfy this “cross-modal response” criterion (CM), such as distributed and conjunctive modal representations. However, I argue that we can distinguish between modal and amodal structures by looking into differences in their cross-modal responses. A component of a distributed cell assembly can be considered unimodal because its responses to stimuli from a given modality are stable, whereas its responses to stimuli from any other modality are not (i.e., these are lost within a short time, plausibly as a result of cell assembly dynamics). In turn, conjunctive modal representations, such as superior colliculus cells in charge of sensory integration, are multimodal because they have a stable response to stimuli from different modalities. Finally, some prefrontal cells constitute amodal representations because they exhibit what has been called ‘adaptive coding’. This implies that their responses to stimuli from any given modality can be lost when the context and task conditions are modified. We cannot assign them a modality because they have no stable relation with any input type.

Abbreviatons: CM: cross-modal response criterion; CCR: conjuntive cross-modal representations; fMRI: functional magnetic resonance imaging; MVPA: multivariate pattern analysis; pre-SMA: pre-supplementary motor area; PFC: prefrontal cortex; SC: superior colliculus; GWS: global workspace     

Causal Learning: Association Versus Computation

Causal learning enables humans and other animals not only to predict important events or outcomes, but also to control their occurrence in the service of needs and desires. Computational theories assume that causal judgments are based on an estimate of the contingency between a causal cue and an outcome. However, human causal learning exhibits many of the characteristics of the associative learning processes thought to underlie animal conditioning. One problem for associative theory arises from the finding that judgments of the causal power of a cue can be revalued retrospectively after learning episodes when that cue is not present. However, if retrieved representations of cues can support learning, retrospective revaluation is anticipated by modified versions of standard associative theories.     

Perceptual simulation in property verification

If people represent concepts with perceptual simulations, two predictions follow in the property verification task (e.g., Isface a property of GORILLA?). First, perceptual variables such as property size should predict the performance of neutral subjects, because these variables determine the ease of processing properties in perceptual simulations (i.e., perceptual effort). Second, uninstructed neutral subjects should spontaneously construct simulations to verify properties and therefore perform similarly to imagery subjects asked explicitly to use images (i.e., instructional equivalence). As predicted, neutral subjects exhibited both perceptual effort and instructional equivalence, consistent with the assumption that they construct perceptual simulations spontaneously to verify properties. Notably, however, this pattern occurred only when highly associated false properties prevented the use of a word association strategy. In other conditions that used unassociated false properties, the associative strength between concept and property words became a diagnostic cue for true versus false responses, so that associative strength became a better predictor of verification than simulation. This pattern indicates that conceptual tasks engender mixtures of simulation and word association, and that researchers must deter word association strategies when the goal is to assess conceptual knowledge.