The supramodal brain: implications for auditory perception

ABSTRACT

The perceptual brain is designed around multisensory input. Areas once thought dedicated to a single sense are now known to work with multiple senses. It has been argued that the multisensory nature of the brain reflects a cortical architecture for which task, rather than sensory system, is the primary design principle. This supramodal thesis is supported by recent research on human echolocation and multisensory speech perception. In this review, we discuss the behavioural implications of a supramodal architecture, especially as they pertain to auditory perception. We suggest that the architecture implies a degree of perceptual parity between the senses and that cross-sensory integration occurs early and completely. We also argue that a supramodal architecture implies that perceptual experience can be shared across modalities and that this sharing should occur even without bimodal experience. We finish by briefly suggesting areas of future research.     

Experiments and models in sequential integration of information

Using the example of bimodal temporal order judgments, we investigate and explain the transition from static designs mapping single stimulus presentations to single responses to designs which require some kind of integration of the processed information. First, we estimate the parameters of several well-known channel-oriented models (Baron, 1969; Gibbon & Rutschmann, 1969) for temporal order judgments. Subsequently, we generalize these models to explain the results of two more complex experimental designs and derive and test their predictions. Special concern is given to the question whether or not the models parameters remain invariant across the experiments. The basic assumptions of the channel-oriented models are confirmed by our data, they represent a well-suited platform to explain more complex integrative processes. Within a sequential decision design, where subjects are allowed to observe a stimulus arbitrarily often before their terminal decision, the individual representations of the stimuli are additively integrated. Hence, even information, which did not yet lead to a terminal decision, will generally influence the later outcome. This representation of the stimuli is accompanied by a dynamic decision rule, which is clearly time-dependent. Several too simplizistic alternative models can be rejected if (and only if) one derives their predictions in some detail and tests them with sufficient statistical power.     

Misleading postevent information: Testing parameterized models of integration in memory

A classic series of experiments by Loftus, Miller and Burns (1978) showed that a person's recollection of an event can be changed by misleading postevent information. Several hypotheses accounting for this effect have been proposed. Loftus' hypothesis of destructive updating claims that the original memory is destroyed by the postevent information. The coexistence hypothesis asserts that the older memory survives but is rendered inaccessible through a mechanism of inhibition or suppression. The non-conflict hypothesis simply accounts for the effect by claiming that subjects can only be misled if they did not encode or if they forgot the original event. These three hypotheses were modelled with the help of all-or-none probabilistic event trees. An experiment was conducted in order to test the three models and to assess parameter values. The experiment followed the classic Loftus paradigm. We suggested to some subjects that they had seen a stopsign, whereas in fact they had seen a traffic light. The misleading postevent information resulted in poorer reproduction of traffic light. Later, all subjects were asked whether they could remember the color of the traffic light, even if they believed they had seen a stopsign. The results showed that subjects who received the misleading post-event information were at least as good at recalling the color of the traffic light as subjects who did not receive misleading information. The no-conflict model accounts well for the obtained results, although the two other, less parsimonious, models cannot be entirely rejected.     

Time course of item and associative information: implications for global memory models

The time course of availability of associative and item information was examined by using a response signal procedure. Associative information discriminates between a studied pair of words and a pair with words from two different studied pairs. Item information is sufficient to discriminate between a studied pair and a pair not studied. In two experiments, discriminations that require associative information are delayed relative to those based on item information. Two additional experiments discount alternative explanations in terms of the time to encode the test items or task strategies. Examination of the global memory models of Gillund and Shiffrin (1984), Hintzman (1988), and Murdock (1982) shows that the models treat item and associative information inseparably. Modifications to these models which can produce separate contributions for item and associative information do not predict any difference in their availability. Two possible mechanisms for the delayed availability of associative information are considered: the involvement of recall in recognition and the time required to form a compound cue.     

Evaluation and integration of visual and auditory information in speech perception

Three experiments were carried out to investigate the evaluation and integration of visual and auditory information in speech perception. In the first two experiments, subjects identified /ba/ or /da/ speech events consisting of high-quality synthetic syllables ranging from /ba/ to /da/ combined with a videotaped /ba/ or /da/ or neutral articulation. Although subjects were specifically instructed to report what they heard, visual articulation made a large contribution to identification. The tests of quantitative models provide evidence for the integration of continuous and independent, as opposed to discrete or nonindependent, sources of information. The reaction times for identification were primarily correlated with the perceived ambiguity of the speech event. In a third experiment, the speech events were identified with an unconstrained set of response alternatives. In addition to /ba/ and /da/ responses, the /bda/ and /tha/ responses were well described by a combination of continuous and independent features. This body of results provides strong evidence for a fuzzy logical model of perceptual recognition.     

Fitts's law holds for action perception

Fitts's law is one of the most well-established principles in psychology. It captures the relation between speed and accuracy in performed and imagined movements. The aim of this study was to determine whether this law also holds during the perception of other people's actions. Subjects were shown apparent motion displays of a person moving his arm between two identical targets. Target width, the separation between targets, and movement speed were varied. Subjects reported whether the person could move at the perceived speed without missing the targets. The movement times reported as being just possible were exactly those predicted by Fitts's law (r(2)= .96). A subsequent experiment demonstrated the same lawful relation for the perception of a robot arm (r(2)= .93). To our knowledge, this makes Fitts's law the first motor principle that holds in imagery and the perception of biological and non-biological agents.     

Individual differences in components of slow cortical potentials: implications for models of information processing

Individual differences in the effects of pharmacological and behavioural manipulations on slow cortical potentials (SCP), may reflect personality differences in type of informational control under attentional stress. Two experiments were conducted to examine SCP component differences in extraverts and introverts under different attentional tasks and with and without nicotine smoking ‘stressor’ conditions. In an initial experiment a decrease in late negativity for introverts and an increase in late negativity for extraverts suggested that smoking enhanced introverts' stimulus set and extraverts' motor set. In a second experiment, personality differences in SCP were examined again, but within a signal-detection paradigm, which allows separate assessment of the contribution of sensory sensitivity and response bias factors to performance. Smoking increased sensitivity in both personality groups, but response bias (caution) increased in introverts only. Extraverts showed an increase in central negativity during smoking whilst introverts showed a decrease in negativity and a decrease in positive wave components. The results are explained in terms of a motor model of attentional control whereby smoking regulates inhibitory controlled actions in introverts, but activates general motor processes for extraverts.     

Hemispheric contributions to the integration of visual and auditory information in speech perception

Differential hemispheric contributions to the perceptual phenomenon known as theMcGurk effect were examined in normal subjects, I callosotomy patient, and 4 patients with intractable epilepsy. Twenty-five right-handed subjects were more likely to demonstrate an influence of a mouthed ward on identification of a dubbed acoustic word when the speaker’s fag vase lateralized to the LVF as compared with the RVF. In contrast, display of printed response alternatives in the RVF elicited a greater percentage of McGurk responses than display in the LVF. Visual field differences were absent in a group of 15 left-handed subjects. These results suggest that in right-handers, the two hemispheres may make distinct contributions to the McGurk effect. The callosotomy patient demonstrated reliable McGurk effects, but at a lower rate than the normal subjects and the epileptic control subjects. These data support the view that both the right and left hemisphere can make significant contributions to the McGurk effect.     

A unifying interpretation of different models for the integration of information when evaluating gambles

Abstract.— Experimental results reported by Sjöberg, Anderson and Shanteau, and Tversky on the structure underlying judgments of the utility of gambles are discussed. Although the authors find different models applicable to their data, it is shown that they may all have the same theoretical basis. Differences in experimental design such as stimulus ranges and response measures used in the experiments may explain the divergent results reported earlier.     

The power law for the perception of rotation by airline pilots

The purpose of this study was to determine the power laws for the perception of rotation about the three major body axes. Eighteen airline pilots made magnitude estimates of 5-sec pulses of nine angular accelerations having a range of acceleration x time of 10–150 deg/sec. The results showed that (1) the power law with an exponent of 1.4 describes the subjective motion of these pilots for all three major body axes, (2) the power law also describes the perception of motion for individual pilots with a substantial range of exponents, (3) there were significant correlations among the exponents for the three body axes, and (4) the data suggest that the power law over the wide range used may be more complex than implied by a formula with a single exponent.     

Visual motion integration for perception and pursuit

To examine the relationship between visual motion processing for perception and pursuit, we measured the pursuit eye-movement and perceptual responses to the same complex-motion stimuli. We show that humans can both perceive and pursue the motion of line-figure objects, even when partial occlusion makes the resulting image motion vastly different from the underlying object motion. Our results show that both perception and pursuit can perform largely accurate motion integration, i.e. the selective combination of local motion signals across the visual field to derive global object motion. Furthermore, because we manipulated perceived motion while keeping image motion identical, the observed parallel changes in perception and pursuit show that the motion signals driving steady-state pursuit and perception are linked. These findings disprove current pursuit models whose control strategy is to minimize retinal image motion, and suggest a new framework for the interplay between visual cortex and cerebellum in visuomotor control.     

The phenomenology of spatial integration: data and models

A briefly presented visual stimulus followed by darkness seems to persist beyond its physical offset. We are concerned here with the relation between two characteristics of this visible persistence: first, its phenomenological resemblance to the stimulus that spawned it and second, its usefulness as a basis for integrating visual stimuli that are separated in time. We describe two experiments using a task in which two halves of a visual stimulus were presented successively and observers reported how complete the stimulus appeared to be. Stimuli appeared less complete with increases in both the duration of the interval intervening between presentation of the two halves and the duration of the initially presented stimulus half. This data pattern is similar to that obtained in tasks in which spatial integration of two temporally disparate stimuli is necessary for correct responding. On the basis of this similarity, we argue that phenomenological appearance and ability to integrate stimuli over time are two facets of the same perceptual events. We describe a formal model to account for these and other data.     

Need for closure and compensatory rule-based perception: The role of information consistency

When making comparisons, people tend to use routinized standards, rules, and knowledge structures. Compensatory rules (e.g., “if competent, then cold”, “if incompetent, then warm”) allow for the quick and easy evaluation of groups when they are compared. We claim that the application of these rules is especially attractive for people who are motivated to seek quick and firm answers (people high in the need for closure—NFC). However, we assume that when people are confronted with expectancy-inconsistent information, higher levels of NFC lead to a lower reliance on these rules. This is because the inconsistency may serve as a signal that the rules no longer provide guidance on how to act. We demonstrated these effects in three studies set in different group contexts, where we manipulated expectancy-consistent and expectancy-inconsistent information. These findings allow for a more comprehensive view of the dynamic and diverse effects of NFC.     

The cognate facilitation effect: implications for models of lexical access

Do nonselected lexical nodes activate their phonological information? Catalan-Spanish bilinguals were asked to name (a) pictures whose names are cognates in the 2 languages (words that are phonologically similar in the 2 languages) and (b) pictures whose names are noncognates in the 2 languages. If nonselected lexical nodes are phonologically encoded, naming latencies should be shorter for cognate words, and because the cognate status of words is only meaningful for bilingual speakers, this difference should disappear when testing monolingual speakers. The results of Experiment 1 fully supported these predictions. In Experiment 2, the difference between cognate and noncognate words was larger when naming in the nondominant language than when naming in the dominant language. The results of the 2 experiments are interpreted as providing support to cascaded activation models of lexical access.     

Hemispheric specialization for temporal information: Implications for the perception of voicing cues during speech perception

Developmental research reporting electrophysiological correlates of voice onset time (VOT) during speech perception is reviewed. By two months of age a right hemisphere mechanism appears which differentiates voiced from voiceless stop consonants. This mechanism was found at 4 years of age and again with adults.A new study is described which represents an attempt to determine a more specific basis for VOT perception. Auditory evoked responses (AER) were recorded over the left and right hemispheres while 16 adults attended to repetitive series of two-tone stimuli. Portions of the AERs were found to vary systematically over the two hemispheres in a manner similar to that previously reported for VOT stimuli. These findings are discussed in terms of a temporal detection mechanism which is involved in speech perception.     

Forgetting curves: implications for connectionist models

Forgetting in long-term memory, as measured in a recall or a recognition test, is faster for items encoded more recently than for items encoded earlier. Data on forgetting curves fit a power function well. In contrast, many connectionist models predict either exponential decay or completely flat forgetting curves. This paper suggests a connectionist model to account for power-function forgetting curves by using bounded weights and by generating the learning rates from a monotonically decreasing function. The bounded weights introduce exponential forgetting in each weight and a power-function forgetting results when weights with different learning rates are averaged. It is argued that these assumptions are biologically reasonable. Therefore power-function forgetting curves are a property that may be expected from biological networks. The model has an analytic solution, which is a good approximation of a power function displaced one lag in time. This function fits better than any of the 105 suggested two-parameter forgetting-curve functions when tested on the most precise recognition memory data set collected by. Unlike the power-function normally used, the suggested function is defined at lag zero. Several functions for generating learning rates with a finite integral yield power-function forgetting curves; however, the type of function influences the rate of forgetting. It is shown that power-function forgetting curves cannot be accounted for by variability in performance between subjects because it requires a distribution of performance that is not found in empirical data. An extension of the model accounts for intersecting forgetting curves found in massed and spaced repetitions. The model can also be extended to account for a faster forgetting rate in item recognition (IR) compared to associative recognition in short but not long retention intervals.     

The integration of motivational information: A conjoint measurement analysis

Within Anderson's (1974a, b, c, 1978b) information integration theory, the integration of motivational information was investigated by means of conjoint measurement techniques. Eighteen university students were asked to judge hypothetical co-students characterized by three features (intelligence, motivation and the extent to which they study) according to their chances to pass. Both rank order data and ratings were obtained. The orderings of most subjects could be represented very well by an additive model. A polynomial regression procedure was applied to determine the shape of the response function for the ratings. As this function was quite linear for all subjects, the ratings could be said to form an interval scale.     

Improved episodic integration through enactment: implications for aging

Enactment may improve memory for verb phrases by facilitating episodic integration of object-action components into a unitized whole. It is unclear, however, whether the influence of enactment on episodic integration is related to or independent of the strength of the preexisting semantic relationship between components. To address this issue, the authors examined the influence of enactment on memory for lists of semantically related object-action phrases ("Put money in the wallet") and semantically unrelated phrases created by repairing these objects and actions to make phrases that were unusual but still were possible to perform ("String a thread through the wallet," "Put money in the napkin"). As such, phrases in the related and unrelated lists were matched for familiarity of the individual components and differed only in the associative strength of the object-action relationship. Although verbatim recall of unrelated lists was poorer under standard verbal encoding conditions, enactment succeeded in bringing performance to the level of related lists, indicating that enactment's influence on episodic integration was independent of the semantic relatedness of the object and action components. Analysis of partial recall errors (accurate recall of only one component) suggested that enactment benefited recall in the unrelated lists by improving memory for the action and reducing fragmentation of the association, providing further support for the unitization view. This pattern of results was replicated in normal older adults, a population that exhibits particular difficulty with episodic memory for unrelated associations. The cognitive mechanisms by which enactment may improve episodic integration in both younger and older adults are discussed.