Processing linguistic and perceptual dimensions of speech: interactions in speeded classification.

Performance on selective-attention and divided-attention tasks shows strong and consistent interactions when participants rapidly classify auditory stimuli whose linguistic and perceptual dimensions (the words low vs. high, low and high pitch, low and high position in space) share common labels. Compared with baseline performance, response times were greater when one or two irrelevant dimensions varied (Garner interference) and when combinations of attributes were incongruent rather than congruent (congruence effects). Performance depended only on the congruence relationships between the relevant dimension and each of the irrelevant dimensions and not on the congruence relationships between the irrelevant dimensions themselves. In selective attention, an additive multidimensional model accounts well for the patterns of both Garner interference and congruence effects.     

Perceptual and linguistic interactions in speeded classification: tests of the semantic coding hypothesis

We tested the semantic coding hypothesis, which states that cross-modal interactions observed in speeded classification tasks arise after perceptual information is recoded into an abstract format common to perceptual and linguistic systems. Using a speeded classification task, we first confirmed the presence of congruence interactions between auditory pitch and visual lightness and observed Garner-type interference with nonlinguistic (perceptual) stimuli (low-frequency and high-frequency tones, black and white squares). Subsequently, we found that modifying the visual stimuli by (a) making them lexical (related words) or (b) reducing their compactness or figural 'goodness' altered congruence effects and Garner interference. The results are consistent with the semantic coding hypothesis, but only in part, and suggest the need for additional assumptions regarding the role of perceptual organization in cross-modal dimensional interactions.     

Contextual influences on interactive processing: Effects of discriminability,quantity, and uncertainty

Three contextual factors—(1) the discriminability of stimuli in pitch, (2) the number of stimuli differing in pitch, and (3)the uncertainty regarding which stimuli or tasks would appear—were manipulated as subjects performed speeded loudness classifications in each of six experiments. The magnitude of Garner interference and effects of congruity were used to gauge the degree of interactive processing. Enhancing pitch discriminability caused monotonic increases in interference and congruity. Stimulus-task uncertainty mediated the changes in Garner interference wrought by increased discriminability. Uncertainty also caused a surprising shift in congruity from strongly positive to strongly negative as uncertainty grew. Increasing stimulus quantity lowered interference, but had inconsistent effects on congruity. Regression analyses suggested that, collectively, these three contextual variables underlie most failures of selective attention in speeded classification.     

Processes underlying dimensional interactions: Correspondences between linguistic and nonlinguistic dimensions

In six experiments, we examined speeded classification when one dimension was linguistic and the other was nonlinguistic. In five of these, attributes on the dimensions corresponded meaningfully, having in common the concepts "high" and "low." For example, in Experiment 1, the visually presented words HI and LO were paired with high- or low-pitched tones; in Experiment 2, the dimensions were visual words and vertical position, in Experiment 3, they were spoken words and position, and in Experiments 4 and 5, spoken words and pitch. For each dimension in each pair, subjects suffered Garner interference when dimensions were varied orthogonally. Garner interference remained constant across 15 blocks of trials (Experiment 5). Subjects also showed significant congruity effects in all experiments, with attributes from congruent stimuli (e.g., HI/high pitch) classified faster than attributes from incongruent stimuli (e.g., HI/low pitch). These results differ from those obtained previously with noncorresponding pairs of linguistic-nonlinguistic dimensions. The results also differ from those obtained with traditional Stroop dimensions (colors and color words; Experiment 6), which showed minimal Garner interference and diminishing congruity effects across blocks of trials. We conclude that the interactions found here represent cross-talk between channels within a semantic level of processing. We contrast our view with current models of dimensional interaction.     

Visually induced gains in pitch discrimination: Linking audio-visual processing with auditory abilities

Perception is fundamentally a multisensory experience. The principle of inverse effectiveness (PoIE) states how the multisensory gain is maximal when responses to the unisensory constituents of the stimuli are weak. It is one of the basic principles underlying multisensory processing of spatiotemporally corresponding crossmodal stimuli that are well established at behavioral as well as neural levels. It is not yet clear, however, how modality-specific stimulus features influence discrimination of subtle changes in a crossmodally corresponding feature belonging to another modality. Here, we tested the hypothesis that reliance on visual cues to pitch discrimination follow the PoIE at the interindividual level (i.e., varies with varying levels of auditory-only pitch discrimination abilities). Using an oddball pitch discrimination task, we measured the effect of varying visually perceived vertical position in participants exhibiting a wide range of pitch discrimination abilities (i.e., musicians and nonmusicians). Visual cues significantly enhanced pitch discrimination as measured by the sensitivity index d’, and more so in the crossmodally congruent than incongruent condition. The magnitude of gain caused by compatible visual cues was associated with individual pitch discrimination thresholds, as predicted by the PoIE. This was not the case for the magnitude of the congruence effect, which was unrelated to individual pitch discrimination thresholds, indicating that the pitch-height association is robust to variations in auditory skills. Our findings shed light on individual differences in multisensory processing by suggesting that relevant multisensory information that crucially aids some perceivers’ performance may be of less importance to others, depending on their unisensory abilities.     

Are crossmodal correspondences relative or absolute? Sequential effects on speeded classification

Crossmodal correspondences have often been demonstrated using congruency effects between pairs of stimuli in different sensory modalities that vary along separate dimensions. To date, however, it is still unclear the extent to which these correspondences are relative versus absolute in nature: that is, whether they result from pre-defined values that rigidly link the two dimensions or rather result from flexible values related to the previous occurrence of the crossmodal stimuli. Here, we investigated this issue in a speeded classification task featuring the correspondence between auditory pitch and visual size (e.g., congruent correspondence between high pitch/small disc and low pitch/large disc). Participants classified the size of the visual stimuli (large vs. small) while hearing concurrent high- or low-pitched task-irrelevant sounds. On some trials, visual stimuli were paired instead with “intermediate” pitch, that could be interpreted differently according to the auditory stimulus on the preceding trial (i.e., as “lower” following the presentation of a high pitch tone, but as “higher” following the presentation of a low pitch tone). Performance on sequence-congruent trials (e.g., when a small disc paired with the intermediate-pitched tone was preceded by a low pitch tone) was compared to sequence-incongruent trials (e.g., when a small disc paired with the intermediate-pitch tone was by a high-pitched tone). The results revealed faster classification responses on sequence-congruent than on sequence-incongruent trials. This demonstrates that the effect of the pitch/size correspondence is relative in nature, and subjected to trial-by-trial interpretation of the stimulus pair.     

权力概念的垂直方位空间隐喻对权力判断的影响

通过两个反应时实验,探讨权力的垂直方位表征对权力判断的影响。实验1发现,当权力词汇出现的位置(电脑屏幕上方或下方)与权力的空间隐喻一致时(高权力词在低权力词上方),相比于隐喻相反时(高权力词在低权力词下方),被试的反应时更短;实验2重复了这一结果,并进一步发现,隐喻一致促进被试的反应,隐喻相反抑制被试的反应。这表明权力的空间隐喻具有心理现实性,会影响权力信息的认知加工。     

Simon congruency effects based on stimulus and response numerosity

Three choice reaction time experiments documented a Simon-type congruence effect involving the numbers of stimuli and responses. In Experiment 1 the stimulus consisted of one or two high- or low-pitch tones, and participants were required to respond with one or two taps of a response key depending on stimulus pitch. Responses were faster when the number of tones matched the required number of response taps than when these numbers mismatched. Experiment 2 showed that a weaker version of this effect can also be obtained using visual stimuli, and Experiment 3 showed that the analogous effect can even be obtained, albeit very weakly, with bimodal stimuli. There was also evidence that the number of stimuli affected the rate of emitting the two-tap response. These results indicate that stimulus numerosity is processed automatically to the level at which it can affect the selection and possibly the execution of a varying number of responses.     

Garner and congruence effects in the speeded classification of bimodal signals

The role of attention in speeded Garner classification of concurrently presented auditory and visual signals was examined in 4 experiments. Within-trial interference (i.e., congruence effects) occurred regardless of the attentional demands of the task. Between-trials interference (i.e., Garner interference) occurred only under conditions of divided attention when making judgments about auditory signals. Of importance, the data show congruence effects in the absence of Garner interference. Such a pattern has been rarely reported in studies of the classification of purely visual stimuli and contradicts theoretical accounts asserting that the effects share a common locus. The data question the notion that Garner classification reveals fundamental insights about the nature of the perceptual processing of bimodal stimuli.     

Simon congruency effects based on stimulus and response numerosity

Three choice reaction time experiments documented a Simon-type congruence effect involving the numbers of stimuli and responses. In Experiment 1 the stimulus consisted of one or two high- or low-pitch tones, and participants were required to respond with one or two taps of a response key depending on stimulus pitch. Responses were faster when the number of tones matched the required number of response taps than when these numbers mismatched. Experiment 2 showed that a weaker version of this effect can also be obtained using visual stimuli, and Experiment 3 showed that the analogous effect can even be obtained, albeit very weakly, with bimodal stimuli. There was also evidence that the number of stimuli affected the rate of emitting the two-tap response. These results indicate that stimulus numerosity is processed automatically to the level at which it can affect the selection and possibly the execution of a varying number of responses.     

Dimensional interactions in vibrotactile information processing

The effect of irrelevant dimensional variation on the processing of vibrotactile stimuli was measured. Six observers performed a speeded classification task with stimuli varying along the dimensions of pitch and loudness. Choice reaction times were obtained for stimuli differing on one dimension alone, on two correlated dimensions, or on two orthogonally varied dimensions. Compared to one-dimension performance, reaction times were faster in the correlated condition and slower in the orthogonal condition. In general, these findings agreed with similar experiments in other modalities, with the exception that the effects in this study tended to be stronger for cases in which loudness, rather than pitch, was the relevant dimension. The results are explained in terms of the integrality of pitch and loudness and of the relative discriminability of dimensions.     

Circle size and diameter tilt: A new look at integrality and separability

In six experiments using the speeded classification paradigm, we provide evidence that the ostensibly “separable” dimensions of size and orientation can produce patterns of either separability or asymmetric configurality, depending on the spatial arrangement of the stimuli. In all experiments, subjects classified large or small circles contairting a single line in one of two possible orientations. When the line touched the circle’s perimeter, thereby defining the diameter of the circle (Experiments 1–4), asymmetric configurality obtained: Variations in size interfered with classification by orientation, but variations in orientation did not interfere with classification by size, and redundancy gain was weak or absent. When the lines fell completely within (i.e., did not touch) the circles (Experiments 5 and 6), the results were consistent with separability: There was neither redundancy gain nor interference. Taken together, the results add to the growing body of evidence that classification of specific dimensional pairs as separable or integral may be less feasible than identifying the more general conditions that increase or decrease the psychological salience of dimensional structures and facilitate or interfere with selection of optimal processing strategies.     

Interference in memory for tonal pitch: Implications for a working-memory model

The degree of interference caused by different kinds of stimuli on memory for tonal pitch was studied. Musically trained and untrained subjects heard a sequence of two tones separated by an interval of 5 sec. The tones were either identical in pitch or differed by a semitone. Subjects had to decide whether the tones were identical or not. The interval was filled with tonal, verbal, or visual material under attended and unattended conditions. The results revealed clear group differences. Musically trained subjects' retention of the first test tone was only affected by the interposition of other tones. In contrast, the performance of musically untrained subjects was also affected by verbal and visual items. The findings are discussed in the framework of Baddeley's (1986) working-memory model.     

Dimensional interaction between color and pitch

In three experiments I investigated the nature of cross-modal dimensional interaction by testing speeded classification of the synesthetically corresponding dimensions of color (white-black) and pitch (high-low). Experiment 1 showed significant Garner interference when these dimensions were varied orthogonally--redundancy gain for positively correlated dimensions and redundancy loss for negatively correlated dimensions. Attributes from synesthetically congruent stimuli were classified faster than attributes from incongruent stimuli (a congruity effect). Experiment 2 tested a perceptual explanation of this interaction (i.e., that color and pitch are configural dimensions) by using Pomerantz's (1986) diagnostic (comparison of selective and divided attention performance). The configurality hypothesis received little support. Experiment 3 examined the effect of optional processes on color and pitch classification. The results suggest that partly strategic and partly mandatory components may constitute overall performance. Three alternative explanations of the color-pitch interaction--perceptual, semantic, and response based--are evaluated in the context of the present results.     

Implicit Association Test (IAT) Studies Investigating Pitch-Shape Audiovisual Cross-modal Associations Across Language Groups

Previous studies have shown that Chinese speakers and non-Chinese speakers exhibit different patterns of cross-modal congruence for the lexical tones of Mandarin Chinese, depending on which features of the pitch they attend to. But is this pattern of language-specific listening a conscious cultural strategy or an automatic processing effect? If automatic, does it also apply when the same pitch contours no longer sound like speech? Implicit Association Tests (IATs) provide an indirect measure of cross-modal association. In a series of IAT studies, conducted with participants with three kinds of language backgrounds (Chinese-dominant bilinguals, Chinese balanced bilinguals, and English speakers with no Chinese experience) we find language-specific congruence effects for Mandarin lexical tones but not for matched sine-wave stimuli. That is, for linguistic stimuli, non-Chinese speakers show advantages for pitch-height congruence (high-pointy, low-curvy); no congruence effects were found for Chinese speakers. For non-linguistic stimuli, all participant groups showed advantages for pitch-height congruence. The present findings suggest that non-lexical tone congruence (high-pointy, low-curvy) is a basic congruence pattern, and the acquisition of a language with lexical tone can alter this perception.     

Higher order sequential effects in psychophysical judgments

We found that the depth of sequential effects depends on the judgment task. An experiment with squares indicated that stimulus-response pairs up to two trials back were included in the judgment process when subjects were required to make category judgments of size, whereas only the immediately preceding event was incorporated when subjects were making magnitude estimations. In the case of category judgment, interactions between the current stimulus and prior stimuli as well as configural effects indicated that events one and two trials back meet an equivalent function in the judgment process and that these events may jointly operate in one trial. These findings can be explained by a class of models that assume that the position of preceding stimuli relative to the current stimulus is decisive in the judgment process. The multiple-standards model is a representative of this class according to which there are two types of standards: (1) the endpoints of the range as long-term standards and (2) traces of preceding stimuli as short-term standards.     

Higher order sequential effects in psychophysical judgments

We found that the depth of sequential effects depends on the judgment task. An experiment with squares indicated that stimulus-response pairs up to two trials back were included in the judgment process when subjects were required to make category judgments of size, whereas only the immediately preceding event was incorporated when subjects were making magnitude estimations. In the case of category judgment, interactions between the current stimulus and prior stimuli as well as configural effects indicated that events one and two trials back meet an equivalent function in the judgment process and that these events may jointly operate in one trial. These findings can be explained by a class of models that assume that the position of preceding stimuli relative to the current stimulus is decisive in the judgment process. The multiple-standards model is a representative of this class according to which there are two types of standards: (1) the endpoints of the range as long-term standards and (2) traces of preceding stimuli as short-term standards.     

Does intermodal equivalence exist between heteromodal stimulus dimensions or between stimulus values on those dimensions?

In a same--different judgement task with successively presented signals, subjects matched dots in different vertical positions and tones of different frequencies intramodally and intermodally. The first and second stimuli of trials in each of the four modality conditions were drawn from a set consisting of two, three or five alternatives. In all intermodal set size conditions, the dimensions of pitch and vertical position were related by the same equivalence rule. While intramodal performance improvement depended only on the total number of practice trials at matching on the relevant dimensions, intermodal performance improvement appeared to be related to the number of trials practice with each heteromodal stimulus pairing in a particular set. After performance had approached asymptotic level neither intramodal nor intermodal matching reaction time depended on set size. Mean “same” reaction time was less than mean “different” reaction time, and this difference was greater for intermodal matching than for intramodal matching. The results indicated that intermodal equivalence exists between discrete stimulus values on heteromodal dimensions rather than between the dimensons themselves.     

Perception of two-component electrocutaneous stimuli

Electrocutaneous stimuli varying in strength (6, 8, 10 V) and frequency (9, 27, 81 Hzj were presented in pairs. Subjects were asked to give a numerical score to represent their perception of overall difference in a pair. To assess their ability to use numbers and their performance on unfamiliar stimuli, no guidance on the nature of the stimuli or the range or scale of the scores was given. Dimensional analysis indicated that perception of differences in one component was influenced by the absolute level of the other component, if this was constant in the pair, or by the difference. Considerable variation among subjects in the nature and extent of this interference effect was observed. Individual differences scaling (INDSCAL) enabled a satisfactory consensus model to be obtained. An empirical relationship expressing the observed twocomponent difference scores in terms of one-component scores was derived. Increasing voltage level diminished perception of frequency difference. Higher frequency levels augmented perception of voltage difference. This relationship is a modification of the combinatorial model proposed by Krantz and Tversky (1975). Individual differences in the combinatorial process are allowed for by application of the subject weightings from the INDSCAL solution.