Breaking the rules in perceptual information integration

We develop a broad theoretical framework for modelling difficult perceptual information integration tasks under different decision rules. The framework allows us to compare coactive architectures, which combine information before it enters the decision process, with parallel architectures, where logical rules combine independent decisions made about each perceptual source. For both architectures we test the novel hypothesis that participants break the decision rules on some trials, making a response based on only one stimulus even though task instructions require them to consider both. Our models take account of not only the decisions made but also the distribution of the time that it takes to make them, providing an account of speed-accuracy tradeoffs and response biases occurring when one response is required more often than another. We also test a second novel hypothesis, that the nature of the decision rule changes the evidence on which choices are based. We apply the models to data from a perceptual integration task with near threshold stimuli under two different decision rules. The coactive architecture was clearly rejected in favor of logical-rules. The logical-rule models were shown to provide an accurate account of all aspects of the data, but only when they allow for response bias and the possibility for subjects to break those rules. We discuss how our framework can be applied more broadly, and its relationship to Townsend and Nozawa’s (1995) Systems-Factorial Technology.     

Is neocortex essentially multisensory?

Although sensory perception and neurobiology are traditionally investigated one modality at a time, real world behaviour and perception are driven by the integration of information from multiple sensory sources. Mounting evidence suggests that the neural underpinnings of multisensory integration extend into early sensory processing. This article examines the notion that neocortical operations are essentially multisensory. We first review what is known about multisensory processing in higher-order association cortices and then discuss recent anatomical and physiological findings in presumptive unimodal sensory areas. The pervasiveness of multisensory influences on all levels of cortical processing compels us to reconsider thinking about neural processing in unisensory terms. Indeed, the multisensory nature of most, possibly all, of the neocortex forces us to abandon the notion that the senses ever operate independently during real-world cognition.     

多感觉整合范式中潜在的跨通道转换效应

大脑可以对来自不同感觉通道的信息进行处理与整合。与单一感觉通道相比, 个体对同时呈现在不同感觉通道的目标信号的响应会更快。对于这种现象的一种主要理论解释是共同激活模型, 该模型认为来自不同通道的刺激在特定的脑区汇聚整合, 比如顶叶内沟、颞上沟和前额叶皮层区域。整合后的信号强度更大, 可以更快地触发反应, 但是信号的整合发生在认知加工的哪一阶段目前尚未有明确结论。当个体对出现在不同感觉通道之间的任务转换进行加工时, 产生与感觉通道相关的任务转换的损失小于跨感觉通道转换损失与任务转换损失的总和, 这为与感觉通道相关的转换代价来源于任务设置的惯性和干扰提供了证据。而在单通道和多通道之间发生转换时, 跨通道转换代价会减小甚至消失, 这是由于同时发生的多感觉整合抵消了一部分损失, 这种现象支持了共同激活模型理论。然而, 多感觉信号整合对任务转换的神经加工过程产生怎样的影响并不清楚, 在未来的研究中可以把多感觉整合范式同经典的任务转换范式结合改进, 进而确定跨通道转换的加工机制和多感觉信号整合的发生阶段。     

Crossmodal integration in the identification of consonant segments

Although speechreading can be facilitated by auditory or tactile supplements, the process that integrates cues across modalities is not well understood. This paper describes two “optimal processing” models for the types of integration that can be used in speechreading consonant segments and compares their predictions with those of the Fuzzy Logical Model of Perception (FLMP, Massaro, 1987). In “pre-labelling” integration, continuous sensory data is combined across modalities before response labels are assigned. In “post-labelling” integration, the responses that would be made under unimodal conditions are combined, and a joint response is derived from the pair. To describe pre-labelling integration, confusion matrices are characterized by a multidimensional decision model that allows performance to be described by a subject's sensitivity and bias in using continuous-valued cues. The cue space is characterized by the locations of stimulus and response centres. The distance between a pair of stimulus centres determines how well two stimuli can be distinguished in a given experiment. In the multimodal case, the cue space is assumed to be the product space of the cue spaces corresponding to the stimulation modes. Measurements of multimodal accuracy in five modern studies of consonant identification are more consistent with the predictions of the pre-labelling integration model than the FLMP or the post-labelling model.     

Attending to multiple sources of information: I. The integration of information in decision making

Information relevant to a decision is often available from several sources, such as different sensory modalities or different spatial locations. Processing or utilization of such information is often selective in that some sources seem to have more influence on a decision or response than others. Many different theoretical mechanisms have been proposed to account for such selective (i.e., attentional) aspects of human information processing. This paper considers theoretical mechanisms which operate to integrate separate sources of information in order to make a binary decision. In particular, these mechanisms are studied in simple visual and auditory detection tasks as well as a Bayesian decision problem. Data from these experiments reject the idea that one integrates the relevant information from several sources prior to making a decision. Rather, one seems to form separate decisions about the information from each source and then integrate these decisions in order to select a response.     

Cross-modal integration of multimodal courtship signals in a wolf spider

Cross-modal integration, i.e., cognitive binding of information transmitted in more than one signal mode, is important in animal communication, especially in complex, noisy environments in which signals of many individuals may overlap. Males of the brush-legged wolf spider Schizocosa ocreata (Hentz) use multimodal communication (visual and vibratory signals) in courtship. Because females may be courted by multiple males at the same time, they must evaluate co-occurring male signals originating from separate locations. Moreover, due to environmental complexity, individual components of male signals may be occluded, altering detection of sensory modes by females. We used digital multimodal playback to investigate the effect of spatial and temporal disparity of visual and vibratory components of male courtship signals on female mate choice. Females were presented with male courtship signals with components that varied in spatial location or temporal synchrony. Females responded to spatially disparate signal components separated by ≥90° as though they were separate sources, but responded to disparate signals separated by ≤45° as though they originated from a single source. Responses were seen as evidence for cross-modal integration. Temporal disparity (asynchrony) in signal modes also affected female receptivity. Females responded more to male signals when visual and vibratory modes were in synchrony than either out-of-synch or interleaved/alternated. These findings are consistent with those seen in both humans and other vertebrates and provide insight into how animals overcome communication challenges inherent in a complex environment.     

“先扬后抑”vs.“先抑后扬”:反馈顺序对决策信心建构的影响

本研究采用2×3被试内设计,通过操纵不同性质信息的反馈比例和反馈顺序,重点考察了反馈顺序对决策信心动态建构的影响。结果表明:(1)在不同反馈比例条件下,当个体接收到的正性反馈多于负性反馈时,个体的决策信心上升,反之则下降;当个体接收到的正性反馈等于负性反馈时,个体的决策信心下降,表现出"负性偏向";(2)"先扬后抑"与"先抑后扬"的信息反馈顺序对决策信心的动态建构产生了不同影响,表现出类似于"近因效应"的现象,"惊讶假设(surprise hypothesis)"可为此提供解释。本研究表明,在决策信心的动态建构过程中,信息的反馈顺序发挥着重要作用。     

视听跨通道信息的整合与冲突控制

多通道信息交互是指来自某个感觉通道的信息与另一感觉通道的信息相互作用、相互影响的一系列加工过程。主要包括两个方面：一是不同感觉通道的输入如何整合;二是跨通道信息的冲突控制。本文综述了视听跨通道信息整合与冲突控制的行为心理机制和神经机制,探讨了注意对视听信息整合与冲突控制的影响。未来需探究视听跨通道信息加工的脑网络机制,考察特殊群体的跨通道整合和冲突控制以帮助揭示其认知和社会功能障碍的机制。     

The epistemic force of perceptual experience

What is the metaphysical nature of perceptual experience? What evidence does experience provide us with? These questions are typically addressed in isolation. In order to make progress in answering both questions, perceptual experience needs to be studied in an integrated manner. I develop a unified account of the phenomenological and epistemological role of perceptual experience, by arguing that sensory states provide perceptual evidence due to their metaphysical structure. More specifically, I argue that sensory states are individuated by the perceptual capacities employed and that there is an asymmetric dependence between their employment in perception and their employment in hallucination and illusion. Due to this asymmetric dependence, sensory states provide us with evidence.     

Estimating the growth of internal evidence guiding perceptual decisions

Perceptual decision-making is thought to involve a gradual accrual of noisy evidence. Temporal integration of the evidence reduces the relative contribution of dynamic internal noise to the decision variable, thereby boosting its signal-to-noise ratio. We aimed to estimate the internal evidence guiding perceptual decisions over time, using a novel combination of external noise and the response signal methods. Observers performed orientation discrimination of patterns presented in external noise. We varied the contrast of the patterns and the delay at which observers were forced to signal their decision. Each test stimulus (patterns and noise sample) was presented twice. Across two experiments we varied the availability of the visual stimulus for processing. Observer model analyses of discrimination accuracy and response consistency to two passes of the same stimulus, suggested that there was very little growth in the internal evidence. The improvement in accuracy over time characterised by the speed-accuracy trade-off function predominantly reflected a decreasing proportion of non-visual decisions, or pure guesses. There was no advantage to having the visual patterns visible for longer than 80 ms, indicating that only the visual information in a short window after display onset was used to drive the decisions. The remarkable constancy of the internal evidence over time suggests that temporal integration of the sensory information was very limited. Alternatively, more extended integration of the evidence from memory could have taken place, provided that the dominant source of internal noise limiting performance occurs between-trials, which cannot be reduced by prolonged evidence integration.     

The function of primate multimodal communication

Language is commonly narrowed down to speech, but human face-to-face communication is in fact an intrinsically multimodal phenomenon. Despite growing evidence that the communication of non-human primates, our main model for the evolution of language, is also inherently multimodal, most studies on primate communication have focused on either gestures or vocalizations in isolation. Accordingly, the biological function of multimodal signalling remains poorly understood. In this paper, we aim to merge the perspectives of comparative psychology and behavioural ecology on multimodal communication, and review existing studies in great apes for evidence of multimodal signal function based on content-based, efficacy-based and inter-signal interaction hypotheses. We suggest that cross-species comparisons of great ape interactions in both captive and wild settings will allow us to test the conditions in which these hypotheses apply. We expect such studies to provide novel insights into the function of speech-accompanying signals and cues, such as gestures, facial expressions, and eye gaze.     

Absolutely relative or relatively absolute: violations of value invariance in human decision making

Making decisions based on relative rather than absolute information processing is tied to choice optimality via the accumulation of evidence differences and to canonical neural processing via accumulation of evidence ratios. These theoretical frameworks predict invariance of decision latencies to absolute intensities that maintain differences and ratios, respectively. While information about the absolute values of the choice alternatives is not necessary for choosing the best alternative, it may nevertheless hold valuable information about the context of the decision. To test the sensitivity of human decision making to absolute values, we manipulated the intensities of brightness stimuli pairs while preserving either their differences or their ratios. Although asked to choose the brighter alternative relative to the other, participants responded faster to higher absolute values. Thus, our results provide empirical evidence for human sensitivity to task irrelevant absolute values indicating a hard-wired mechanism that precedes executive control. Computational investigations of several modelling architectures reveal two alternative accounts for this phenomenon, which combine absolute and relative processing. One account involves accumulation of differences with activation dependent processing noise and the other emerges from accumulation of absolute values subject to the temporal dynamics of lateral inhibition. The potential adaptive role of such choice mechanisms is discussed.     

创伤后应激障碍个体对威胁刺激的注意偏向：眼动研究的证据

对威胁刺激的注意偏向是创伤后应激障碍(PTSD)个体的一种表现。近年来, 研究者利用眼动技术探讨了注意偏向的加工特点及其内在机制。尽管针对是否存在注意定向加速、注意解除困难或注意回避的研究结果并不一致, 但都试图为验证如警戒-回避模型、注意保持模型等提供眼动证据。未来研究在记录PTSD个体注视模式时, 不仅需要从多个维度构建、整合PTSD个体的眼动模型, 也要结合多种技术从多模态角度共同考察PTSD个体的威胁信息注意偏向机制。     

How race affects evidence accumulation during the decision to shoot

The biasing role of stereotypes is a central theme in social cognition research. For example, to understand the role of race in police officers’ decisions to shoot, participants have been shown images of Black and White males and instructed to shoot only if the target is holding a gun. Findings show that Black targets are shot more frequently and more quickly than Whites. The decision to shoot has typically been modeled and understood as a signal detection process in which a sample of information is compared against a criterion, with the criterion set for Black targets being lower. We take a different approach, modeling the decision to shoot as a dynamic process in which evidence is accumulated over time until a threshold is reached. The model accounts for both the choice and response time data for both correct and incorrect decisions using a single set of parameters. Across four studies, this dynamic perspective revealed that the target’s race did not create an initial bias to shoot Black targets. Instead, race impacted the rate of evidence accumulation with evidence accumulating faster to shoot for Black targets. Some participants also tended to be more cautious with Black targets, setting higher decision thresholds. Besides providing a more cohesive and richer account of the decision to shoot or not, the dynamic model suggests interventions that may address the use of race information in decisions to shoot and a means to measure their effectiveness.     

A comparison of bounded diffusion models for choice in time controlled tasks

The Wiener diffusion model (WDM) for 2-alternative tasks assumes that sensory information is integrated over time. Recent neurophysiological studies have found neural correlates of this integration process in certain neuronal populations. This paper analyses the properties of the WDM with two different boundary conditions in decision making tasks in which the time of response is indicated by a cue. A dual reflecting boundary mechanism is proposed and its performance is compared with a well-established absorbing boundary in the cases of the WDM, the WDM with extensions, and the WDM with prior probability. The two types of boundary influence the dynamics of the model and introduce differential weighting of evidence. Comparisons with Ornstein-Uhlenbeck models are also done, and it is shown that the WDM with both types of boundary achieves similar performance and produces similar fits to existing behavioural data. Further studies are proposed to distinguish which boundary mechanism is more consistent with experimental data.     

视听整合加工及其神经机制

通常人们接收到来自不同感觉通道的信息时, 首先在大脑中各个分离的区域单独进行加工处理, 而后在多感官区进行整合。前人关于言语感知中视听整合加工的神经成像研究认为, 视觉和听觉信息能够相互影响; 两者进行整合的关键区域是人脑左后侧的颞上沟, 其整合效应受时间和空间因素的限制。未来的研究应致力于建立更加合理的实验范式和数据分析方法来探讨整合加工的脑区机制, 把多感官整合研究进一步延伸到更加复杂的领域。     

Cells,circuits, and choices: Social influences on perceptual decision making

Making decisions is an integral part of everyday life. Social psychologists have demonstrated in many studies that humans’ decisions are frequently and strongly influenced by the opinions of others—even in simple perceptual decisions, where, for example, participants have to judge what an image looks like. However, because the effect of other people’s opinions on decision making has remained largely unaddressed by the neuroimaging and neurophysiology literature, we are only beginning to understand how social influence is integrated into the decision-making process. We put forward the thesis that by probing the neurophysiology of social influence with perceptual decisionmaking tasks similar to those used in the seminal work of Asch (1952, 1956), this gap could be remedied. Perceptual paradigms are already widely used to probe neuronal mechanisms of decision making in nonhuman primates. There is also increasing evidence about how nonhuman primates’ behavior is influenced by observing conspecifics. The high spatial and temporal resolution of neurophysiological recordings in awake monkeys could provide insight into where and how social influence modulates decision making, and thus should enable us to develop detailed functional models of the neural mechanisms that support the integration of social influence into the decision-making process.     

A new framework for modeling decisions about changing information: The Piecewise Linear Ballistic Accumulator model

In the real world, decision making processes must be able to integrate non-stationary information that changes systematically while the decision is in progress. Although theories of decision making have traditionally been applied to paradigms with stationary information, non-stationary stimuli are now of increasing theoretical interest. We use a random-dot motion paradigm along with cognitive modeling to investigate how the decision process is updated when a stimulus changes. Participants viewed a cloud of moving dots, where the motion switched directions midway through some trials, and were asked to determine the direction of motion. Behavioral results revealed a strong delay effect: after presentation of the initial motion direction there is a substantial time delay before the changed motion information is integrated into the decision process. To further investigate the underlying changes in the decision process, we developed a Piecewise Linear Ballistic Accumulator model (PLBA). The PLBA is efficient to simulate, enabling it to be fit to participant choice and response-time distribution data in a hierarchal modeling framework using a non-parametric approximate Bayesian algorithm. Consistent with behavioral results, PLBA fits confirmed the presence of a long delay between presentation and integration of new stimulus information, but did not support increased response caution in reaction to the change. We also found the decision process was not veridical, as symmetric stimulus change had an asymmetric effect on the rate of evidence accumulation. Thus, the perceptual decision process was slow to react to, and underestimated, new contrary motion information.     

早期视听整合加工——来自MMN的证据

视听整合是指当呈现的视觉和听觉信号在时间、空间上大致接近时,视觉和听觉系统倾向于整合的加工过程。失匹配负波(Mismatch Negativity,MMN)作为反映大脑早期加工的成分,表征偏差的信息输入与感觉记忆痕迹之间的神经失匹配。以MMN作为探测指标的视听整合加工研究,主要包括MMN在阅读理解中字母和语音、韵律信息、麦格克效应等方面的视听整合加工,以及分析跨通道视听整合相互竞争、相互补充的关系。未来研究需聚焦于其他通道的跨通道整合加工,同时应拓展MMN的诱发范式。