Decision noise: An explanation for observed violations of signal detection theory

In signal detection theory (SDT), responses are governed by perceptual noise and a flexible decision criterion. Recent criticisms of SDT (see, e.g., Balakrishnan, 1999) have identified violations of its assumptions, and researchers have suggested that SDT fundamentally misrepresents perceptual and decision processes. We hypothesize that, instead, these violations of SDT stem from decision noise: the inability to use deterministic response criteria. In order to investigate this hypothesis, we present a simple extension of SDT—the decision noise model—with which we demonstrate that shifts in a decision criterion can be masked by decision noise. In addition, we propose a new statistic that can help identify whether the violations of SDT stem from perceptual or from decision processes. The results of a stimulus classification experiment—together with model fits to past experiments—show that decision noise substantially affects performance. These findings suggest that decision noise is important across a wide range of tasks and needs to be better understood in order to accurately measure perceptual processes.     

Judgments of Frequency and Recency in a Distributed Memory Model

TODAM is a theory of distributed associative memory based on the convolution-correlation formalism of A. Borsellino and T. Poggio (1973, Kybernetik, 122, 113-122), and TODAM2 is a revised version which includes context, auto-associations for binding, a dual basis for item information, and mediators for associative information. It can explain some complex interactions (differential forgetting and differential attention) between item and associative information. In this paper we derive the basic expressions for the memory-probe dot product for the 2x2 cases (item and pair study crossed with item and pair probe), and then we apply these expressions to judgments of frequency (JOF) and judgments of recency (JOR). We report an experiment which tests both JOF and JOR for single items with post-cuing to control for encoding strategies and suggest an attenuation factor for repetition to improve the fits. With attenuation, TODAM2 can fit the JOR data, but the JOF fits, while not too bad, consistently predict too much dependence between item and associative information. Copyright 2001 Academic Press.     

On the statistical and theoretical basis of signal detection theory and extensions: Unequal variance, random coefficient, and mixture models

Basic results for conditional means and variances, as well as distributional results, are used to clarify the similarities and differences between various extensions of signal detection theory (SDT). It is shown that a previously presented motivation for the unequal variance SDT model (varying strength) actually leads to a related, yet distinct, model. The distinction has implications for other extensions of SDT, such as models with criteria that vary over trials. It is shown that a mixture extension of SDT is also consistent with unequal variances, but provides a different interpretation of the results; mixture SDT also offers a way to unify results found across several types of studies.     

Disentangling the effects of word frequency and contextual diversity on serial recall performance

Research shows that contextual diversity (CD; the number of different contexts in which a word appears within a corpus) constitutes a better predictor of reading performance than word frequency (WF), that it mediates the access to lexical representations, and that controlling for contextual CD abolishes the effect of WF in lexical decision tasks. Despite the theoretical relevance of these findings for the study of serial memory, it is not known how CD might affect serial recall performance. We report the first independent manipulation of CD and WF in a serial recall task. Experiment 1 revealed better performance for low CD and for high WF words independently. Both effects affected omissions and item errors, but contrary to past research, word frequency also affected order errors. These results were confirmed in two more experiments comparing pure and alternating lists of low and high CD (Experiment 2) or WF (Experiment 3). The effect of CD was immune to this manipulation, while that of WF was abolished in alternating lists. Altogether the findings suggest a more difficult episodic retrieval of item information for words of high CD, and a role for both item and order information in the WF effect.     

Decision criteria do not shift: Commentary on Mueller and Weidemann (2008)

The effects of base rates and payoffs on the shapes of rating receiver operating characteristic curves are inconsistent with the basic assumptions of signal detection theory (SDT), in particular the notion of a shifting decision criterion. Mueller and Weidemann (2008) propose that these unexpected phenomena are not due to problems with the decision- criterion construct but are instead due to two compounded effects: instability of the decision criterion across trials, and even greater instability in the flanking criteria that determine which confidence rating will be reported. There are several problems with the authors’ decision-noise hypothesis. First, even if their hypothesis about decision noise were taken for granted, the key feature of the ratings data that rejects the SDT model would remain a mystery. Second, the same violations of SDT that are exhibited in the ratings paradigm are also exhibited in the yes-no detection task when response time is substituted for confidence as a basis for analysis. Finally, the decision-noise hypothesis predicts that sensitivity will increase when one source of this variation-the response on a previous trial-is controlled for. This prediction was consistently violated in both the yes-no and ratings conditions of Mueller and Weidemann’s experiment. In an Addendum, we respond to Weidemann and Mueller’s (2008) reply to this Comment.     

A signal-detection analysis of fast-and-frugal trees

Models of decision making are distinguished by those that aim for an optimal solution in a world that is precisely specified by a set of assumptions (a so-called "small world") and those that aim for a simple but satisfactory solution in an uncertain world where the assumptions of optimization models may not be met (a so-called "large world"). Few connections have been drawn between these 2 families of models. In this study, the authors show how psychological concepts originating in the classic signal-detection theory (SDT), a small-world approach to decision making, can be used to understand the workings of a class of simple models known as fast-and-frugal trees (FFTs). Results indicate that (a) the setting of the subjective decision criterion in SDT corresponds directly to the choice of exit structure in an FFT; (b) the sensitivity of an FFT (measured in d') is reflected by the order of cues searched and the properties of cues in an FFT, including the mean and variance of cues' individual d's, the intercue correlation, and the number of cues; and (c) compared with the ideal and the optimal sequential sampling models in SDT and a majority model with an information search component, FFTs are extremely frugal (i.e., do not search for much cue information), highly robust, and well adapted to the payoff structure of a task. These findings demonstrate the potential of theory integration in understanding the common underlying psychological structures of apparently disparate theories of cognition.     

Judgments of frequency and recency: how they relate to reports of subjective awareness.

In 2 experiments, the author explored the relations of remember versus familiar ratings to judgments of frequency (JOFs) and to judgments of recency (JORs). In both cases, remembered items were associated with more accurate memory judgments. In general, familiar items were judged to have occurred less frequently and less recently than remembered items. However, JOFs and JORs associated with familiar items were more accurate than chance. Implications for theories of remember versus familiar ratings, JOFs, and JORs are considered. Some basic findings that constrain these theories are that (a) remember versus familiar ratings were less sensitive than JOF to presentation frequency and less sensitive than JOR to recency and (b) although remember versus familiar ratings are strongly related to both JOF and JOR, as measured by gamma, the relations are far from perfect.     

Signal detection theory analyses of semantic priming in word recognition

Word recognition, semantic priming, and cognitive impenetrability research have used signal detection theory (SDT) measures to separate perceptual and postperceptual processes. In the D. Norris (1986) checking model and model simulation (D. Norris, 1995), priming alters only postperceptual word decision criteria: Stimulus-related priming reduces uncertainty, increasing sensitivity; stimulus-unrelated priming increases false alarms more than hits, reducing sensitivity. This work is cited as strong evidence that criterion changes can alter perceptual sensitivity and that SDT is inappropriate for investigating complex cognitive processes. The authors' current SDT ideal observer analysis of the model demonstrates that related priming does not directly alter sensitivity and that unrelated priming increases only false-alarm rate, reducing sensitivity. This analysis provides new perspectives on SDT concepts of complex decision processing.     

Context matching and judgments of recency

An experiment was done to test a context-matching explanation of memory for recency under steady-state conditions. Subjects went through a list of 550 names, in which individual names were repeated at lags of 5–30 other items. The names were shown in two different styles or contexts. An old versus new recognition decision was made on each name, and eachold response was followed by a numerical judgment of recency (JOR). When first- and second-presentation contexts were the same, recognition hit rates were higher, and mean JORs were shorter (more recent), than when the two contexts were different. The JOR result is as predicted by the context-matching hypothesis.     

Invariance of a two-factor model of Social Dominance Orientation across gender

This research examined the measurement and structural invariance of the 16-item Social Dominance Orientation (SDO) scale across gender. The Social Dominance Theory (SDT) invariance hypothesis proposes that men are higher than women in SDO. Tests of this hypothesis have compared summary scores with t-tests, but there is no psychometric work of measurement invariance across gender, which makes interpretation of t-test comparisons problematic. Some research suggests the scale captures two distinct factors: support for group-based dominance (GBD) and general opposition to equality (OEQ). However, gender differences in GBD and OEQ have mixed findings. Results replicated the two-factor structure. Multiple-group CFA indicated that loadings, intercepts, factor variances, and factor covariance are equivalent across gender. Males demonstrated higher latent means of GBD and OEQ, supporting the SDT invariance hypothesis.     

ROC curves and confidence judgements in recognition memory

Most models of recognition memory rely on a strength/familiarity-based signal detection account that assumes that the processes giving rise to a confidence judgment are the same as those giving rise to an old-new decision. Confidence is assumed to be scaled directly from the perceived familiarity of a probe. This assumption was tested in 2 experiments that examine the shape of confidence-based z receiver operating characteristic (zROC) curves under different levels of response bias induced by changing stimulus probabilities (Experiment 1) and payoffs (Experiment 2). Changes in the shape of the zROC curves with bias indicate that confidence is not scaled directly from perceived familiarity or likelihood. A model of information accumulation in recognition memory is proposed that can account for the observed effects.     

Testing theories of recognition memory by predicting performance across paradigms

Signal-detection theory (SDT) accounts of recognition judgments depend on the assumption that recognition decisions result from a single familiarity-based process. However, fits of a hybrid SDT model, called dual-process theory (DPT), have provided evidence for the existence of a second, recollection-based process. In 2 experiments, the authors tested predictions of DPT and SDT by comparing the invariance of parameter estimates between yes/no (Y/N) and 2-alternative forced-choice (2AFC) testing paradigms. Both experiments showed DPT recollection estimates in Y/N to be poorly correlated with recollection estimates in 2AFC. In Experiment 2, SDT predictions explained more variance than DPT predictions. The authors evaluate and discuss the extent to which each model possesses theoretical validity versus computational flexibility in curve fitting.     

Describing and testing a moderator of the moral judgment and action relationship

The relationship between moral judgment development and action for Ss who differ in the degree to which justice-based moral judgments influence moral decision making was investigated. First, a method for quantifying Ss' use of justice-based moral judgments in moral decision making (the utilizer variable) is described. Second, 5 moral judgment and action studies are reanalyzed to assess the moderating effect of the utilizer variable information. Results indicate that the relationship between moral judgment scores and action intensifies as utilization increases. Moreover, the utilizer effect appears uniform across different age and educational levels and behavioral domains. These findings support the validity of the utilizer variable and are consistent with recent multiprocess models of moral action that suggest that moral judgements are a necessary, but insufficient, factor in the production of moral action.     

口语产生中词频效应、音节频率效应和语音促进效应的认知年老化

采用图画-词汇干扰实验范式,考察了汉语口语产生过程中词频与音节频率效应、语音促进效应的认知年老化现象及其认知机制。结果发现,老年组的图画命名时间长于青年组,且老年组的词频效应和音节频率效应高于青年组,而语音相关效应低于青年组。老年组中词频和音节频率存在两因素交互作用,青年组则没有,这表明随着年龄增加,老年人的心理词典中不仅语义水平和音韵水平之间的联结强度变弱,而且信息之间的作用模式发生了变化。研究结果为激活不充分假设或传输不足假设提供了证据。     

Working memory moderates the effect of the integrative process of implicit and explicit autonomous motivation on academic achievement

In previous research, autonomous motivation (AM) has been found to be associated with school achievement, but the relation has been largely heterogeneous across studies. AM has typically been assessed with explicit measures such as self‐report questionnaires. Recent self‐determination theory (SDT) research has suggested that converging implicit and explicit measures can be taken to characterize the integrative process in SDT. Drawing from dual‐process theories, we contended that explicit AM is likely to promote school achievement when it is part of an integrated cognitive system that combines easily accessible mental representations (i.e., implicit AM) and efficient executive functioning. A sample of 272 university students completed a questionnaire and a lexical decision task to assess their explicit and implicit AM, respectively, and they also completed working memory capacity measures. Grades were obtained at the end of the semester to examine the short‐term prospective effect of implicit and explicit AM, working memory, and their interaction. Results of moderation analyses have provided support for a synergistic interaction in which the association between explicit AM and academic achievement was positive and significant only for individuals with high level of implicit AM. Moreover, working memory was moderating the synergistic effect of explicit and implicit AM. Explicit AM was positively associated with academic achievement for students with average‐to‐high levels of working memory capacity, but only if their motivation operated synergistically with high implicit AM. The integrative process thus seems to hold better proprieties for achievement than the sole effect of explicit AM. Implications for SDT are outlined.     

On a signal detection approach to m-alternative forced choice with bias,with maximum likelihood and Bayesian approaches to estimation

The standard signal detection theory (SDT) approach to $m$-alternative forced choice uses the proportion correct as the outcome variable and assumes that there is no response bias. The assumption of no bias is not made for theoretical reasons, but rather because it simplifies the model and estimation of its parameters. The SDT model for $m$AFC with bias is presented, with the cases of two, three, and four alternatives considered in detail. Two approaches to fitting the model are noted: maximum likelihood estimation with Gaussian quadrature and Bayesian estimation with Markov chain Monte Carlo. Both approaches are examined in simulations. SAS and OpenBUGS programs to fit the models are provided, and an application to real-world data is presented.     

On the measurement of criterion noise in signal detection theory: the case of recognition memory

Traditional approaches within the framework of signal detection theory (SDT; Green & Swets, 1966), especially in the field of recognition memory, assume that the positioning of response criteria is not a noisy process. Recent work (Benjamin, Diaz, & Wee, 2009; Mueller & Weidemann, 2008) has challenged this assumption, arguing not only for the existence of criterion noise but also for its large magnitude and substantive contribution to individuals' performance. A review of these recent approaches for the measurement of criterion noise in SDT identifies several shortcomings and confoundings. A reanalysis of Benjamin et al.'s (2009) data sets as well as the results from a new experimental method indicate that the different forms of criterion noise proposed in the recognition memory literature are of very low magnitudes, and they do not provide a significant improvement over the account already given by traditional SDT without criterion noise.     

Domain-specific and domain-general processes underlying metacognitive judgments

Metacognition and self-awareness are commonly assumed to operate as global capacities. However, there have been few attempts to test this assumption across multiple cognitive domains and metacognitive evaluations. Here, we assessed the covariance between “online” metacognitive processes, as measured by decision confidence judgments in the domains of perception and memory, and error awareness in the domain of attention to action. Previous research investigating metacognition across task domains have not matched stimulus characteristics across tasks raising the possibility that any differences in metacognitive accuracy may be influenced by local task properties. The current experiment measured metacognition in perceptual, memorial and attention tasks that were closely matched for stimulus characteristics. We found that metacognitive accuracy across the three tasks was dissociated suggesting that domain specific networks support an individual’s capacity for accurate metacognition. This finding was independent of objective performance, which was controlled using a staircase procedure. However, response times for metacognitive judgments and error awareness were associated suggesting that shared mechanisms determining how these meta-level evaluations unfold in time may underlie these different types of decision. In addition, the relationship between these laboratory measures of metacognition and reports of everyday functioning from participants and their significant others (informants) was investigated. We found that informant reports, but not self reports, predicted metacognitive accuracy on the perceptual task and participants who underreported cognitive difficulties relative to their informants also showed poorer metacognitive accuracy on the perceptual task. These results are discussed in the context of models of metacognitive regulation and neuropsychological evidence for dissociable metacognitive systems. The potential for the refinement of metacognitive assessment in clinical populations is also discussed.     

Some task demands induce collapsing bounds: Evidence from a behavioral analysis

Traditional models of choice-response time assume that sensory evidence accumulates for choice alternatives until a threshold amount of evidence has been obtained. Although some researchers have characterized the threshold as varying randomly from trial to trial, these investigations have all assumed that the threshold remains fixed across time within a trial. Despite decades of successful applications of these models to a variety of experimental manipulations, the time-invariance assumption has recently been called into question, and a time-variant alternative implementing collapsing decision thresholds has been proposed instead. Here, we investigated the fidelity of the collapsing threshold assumption by assessing relative model fit to data from a highly constrained experimental design that coupled a within-subject mixture of two classic response time paradigms—interrogation and free response—within a random dot motion (RDM) task. Overall, we identified strong evidence in favor of collapsing decision thresholds, suggesting that subjects may adopt a dynamic decision policy due to task characteristics, specifically to account for the mixture of response time paradigms and motion strengths across trials in the mixed response signal task. We conclude that time-variant mechanisms may serve as a viable explanation for the strategy used by human subjects in our task.     

A Psychometric Evaluation of the Thought Control Ability Questionnaire (TCAQ) and the Prediction of Cognitive Control

Study 1 evaluated the psychometric properties of the English version of the Thought Control Ability Questionnaire (TCAQ; Luciano et al. 2005), an index of perceived control over intrusive cognitions. Confirmatory factor analysis in a sample of 720 University students revealed a clear uni-dimensional structure (after removal of items 5, 7, 8, 14, and 25) with high internal consistency (α = .87, 95% CI = [.86, .88]) and test-retest reliability after a six month interval (r = .68). Correlational analyses supported an inverse relationship with measures of depression, anxiety, maladaptive cognitive control strategies, and obsessive–compulsive symptomatology. Study 2 tested the ability of the TCAQ to predict successful cognitive control during an experimental suppression protocol. Results demonstrated that weak thought control ability was predictive of the frequency and associated levels of distress of a target thought while under instruction to suppress. Additionally, weak perceived thought control ability was predictive of increased efforts to suppress the target material. Collectively, results suggest that thought control ability is a measurable individual difference variable and that the TCAQ is a reliable index of perceived cognitive control.