Decision noise may mask criterion shifts: Reply to Balakrishnan and MacDonald (2008)

J. D. Balakrishnan and J. A. MacDonald (2008) argue that RTbased measures of signal detection processes provide evidence against signal detection theory’s notion of a flexible decision criterion. They argue that this evidence is immune to the alternative explanation proposed by S. T. Mueller and C. T. Weidemann (2008), that decision noise may mask criterion shifts. We show that noise in response times can produce the same effects as are produced by noise in confidence ratings. Given these results, the evidence is not sufficient to categorically reject the notion of a flexible response policy implemented through shifts in a decision criterion. nt]mis|This research was supported in part by a postdoctoral fellowship to C.T.W. from the German Academic Exchange Service (DAAD).     

On the measurement of criterion noise in signal detection theory: the case of recognition memory

Traditional approaches within the framework of signal detection theory (SDT; Green & Swets, 1966), especially in the field of recognition memory, assume that the positioning of response criteria is not a noisy process. Recent work (Benjamin, Diaz, & Wee, 2009; Mueller & Weidemann, 2008) has challenged this assumption, arguing not only for the existence of criterion noise but also for its large magnitude and substantive contribution to individuals' performance. A review of these recent approaches for the measurement of criterion noise in SDT identifies several shortcomings and confoundings. A reanalysis of Benjamin et al.'s (2009) data sets as well as the results from a new experimental method indicate that the different forms of criterion noise proposed in the recognition memory literature are of very low magnitudes, and they do not provide a significant improvement over the account already given by traditional SDT without criterion noise.     

Decision noise: An explanation for observed violations of signal detection theory

In signal detection theory (SDT), responses are governed by perceptual noise and a flexible decision criterion. Recent criticisms of SDT (see, e.g., Balakrishnan, 1999) have identified violations of its assumptions, and researchers have suggested that SDT fundamentally misrepresents perceptual and decision processes. We hypothesize that, instead, these violations of SDT stem from decision noise: the inability to use deterministic response criteria. In order to investigate this hypothesis, we present a simple extension of SDT—the decision noise model—with which we demonstrate that shifts in a decision criterion can be masked by decision noise. In addition, we propose a new statistic that can help identify whether the violations of SDT stem from perceptual or from decision processes. The results of a stimulus classification experiment—together with model fits to past experiments—show that decision noise substantially affects performance. These findings suggest that decision noise is important across a wide range of tasks and needs to be better understood in order to accurately measure perceptual processes.     

Binary ROCs in perception and recognition memory are curved

In recognition memory, a classic finding is that receiver operating characteristics (ROCs) are curvilinear. This has been taken to support the fundamental assumptions of signal detection theory (SDT) over discrete-state models such as the double high-threshold model (2HTM), which predicts linear ROCs. Recently, however, Br?der and Schütz (2009) challenged this argument by noting that most of the data on which support for SDT is based have involved confidence ratings. The authors argued that certain types of rating scale usage may result in curved ROCs even if the generating process is thresholded in nature. From this point of view, only ROCs constructed via experimental bias manipulations are useful for discriminating between the models. Br?der and Schütz conducted a meta-analysis and new experiments that compared SDT and the 2HTM using binary (yes-no) ROCs and found that many of these functions were linear, supporting 2HTM over SDT. We examine all the data reported by Br?der and Schütz, noting important limitations in their methodology, analyses, and conclusions. We report a new meta-analysis and 2 new experiments to examine the issue more closely while avoiding the limitations of Br?der and Schütz's study. These new data indicate that binary ROCs are curved in recognition, consistent with previous findings in perception and reasoning. Our results support classic arguments in favor of SDT and indicate that curvature in ratings ROCs is not task specific. We recommend the ratings procedure and suggest that analyses based on threshold models be treated with caution.     

Is signal detection theory fundamentally flawed? A response to Balakrishnan (1998a, 1998b, 1999)

For nearly 50 years, signal detection theory (SDT; Green & Swets, 1966; Macmillan & Creelman, 1991) has been of central importance in the development of psychophysics and other areas of psychology. The theory has recently been challenged by Balakrishnan (1998b), who argues that, within SDT, an alternative index is “better justified” than d’ and who claims to show (1998a, 1999) that SDT is fundamentally flawed and should be rejected. His evidence is based on new nonparametric measures that he has introduced and applied to experimental data. He believes his results show that basic assumptions of SDT are not supported—in particular, that payoff and probability manipulations do not affect the position of the decision criterion. In view of the importance of SDT in psychology, these claims deserve careful examination. They are critically reviewed here. It appears that it is Balakrishnan’s arguments that fail, and not SDT.     

Response latency,confidence, and ROCs in auditory signal detection

Response latencies were obtained from 10 Ss in auditory signal-detection experiments. The response latencies were inversely related to certainty that a signal was (or was not) presented. The S’s decision criterion was found to have an influence on response latency, which was consistent with the hypothesis that stimuli close to the current criterion elicit longer response latencies than stimuli more distant from the criterion. Comparisons among receiver operating characteristics derived from binary decisions, from the latencies of binary decisions, and from confidence ratings show that response latencies and binary decisions together yield more information about the stimulus than does the binary decision alone. However, the increment in information gained from the measurement of response latencies is in general (though not for every S) smaller than that gained by shifting from yes-no responses to a confidence-rating procedure.     

Positive and negative remember judgments and ROCs in the plurals paradigm: Evidence for alternative decision strategies

Using old-new ratings and remember-know judgments we explored the plurals paradigm, in which studied words must be distinguished from plurality-changed lures. The paradigm allowed us to investigate negative remembering—that is, the remembering of a plural-altered study item; capacity for this judgment was found to be poorer than or equivalent to the conventional positive remembering. A response-bias manipulation affected positive but not negative remembering. The ratings were used to construct ROC curves and test the prediction of the most common dual-process theory of recognition memory (Yonelinas, 2001) that the amount of recollection can be independently estimated from ROC curves and from remember judgments. By fitting the individual data with pure signal detection theory (SDT) models and dual-process models that combined SDT and high-threshold components (HTSDT), we identified two types of subjects. For those who were better described by HTSDT, the predicted convergence of remember-know and ROC measures was observed. For those who were better described by SDT, the ROC intercept could not predict the remember rate. The data are consistent with the idea that all subjects rely on the same representation but base their decisions on different partitions of a decision space.     

The effect of signal probability on the slope of the receiver operating characteristic given by the rating procedure

Signal detection theory (SDT) requires that the slope of the receiver operating characteristic (ROC) is independent of the probability of signal and noise. But it has been shown that when the rating procedure is applied to detection, the slope of the receiver operating characteristic may increase as a function of the probability of the signal (Schulman & Greenberg, 1970). This presents a serious problem for signal detection theory. This problem is examined in relation to a recent theory of criterion setting (Treisman & Williams, 1984), and an explanation for the effect of signal probability on ROC slope is derived which is compatible with SDT. Further data on the relation between signal probability and ROC slope are reported.     

Detection and identification of geometric forms using peripheral and central viewing

Two tasks. using identical geometric forms. compared nasal peripheral. temporal peripheral. and central viewing. In the first task. using a yes-no confidence rating scale procedure, four Ss judged the presence or absence of a known geometric form (detection). In the second task. the same Ss judged which of several know forms was presented (identification). The major result is that nasal and temporal peripheral viewing are similar and that both peripheral regions differ from central viewing. Also, detection and identification are relatively independent processes; the ease of detection of a form does not predict the ease of identification of that form.     

Conscious and unconscious discriminations between true and false memories

When subjects give higher confidence or memory ratings to a test word in a recognition test, do they simply raise their criterion without making better discrimination, or do they raise both criterion and true discrimination between the studied words (SW) and the lures? Given that previous studies found subjects’ false alarm responses to lures slower than to SW, and recognition latency inversely correlated with the confidence rating, can the latency difference between the lures and SW be accounted for by confidence or memory ratings? The present results showed that when subjects gave higher confidence or memory ratings, both their bias and sensitivity were raised, indicating that they could consciously distinguish the lures from the SW. However, a latency difference between true and false recognitions persisted after confidence and memory ratings were held constant, suggesting an unconscious source of discrimination between the two types of memory.     

The revelation effect for autobiographical memory: A mixture-model analysis

Participants provided information about their childhood by rating their confidence about whether they had experienced various events (e.g., “broke a window playing ball”). On some trials, participants unscrambled a key word from the event phrase (e.g., wdinwo—window) or an unrelated word (e.g., gnutge—nugget) before seeing the event and giving their confidence ratings. The act of unscrambling led participants to increase their confidence that the event occurred in their childhood, but only when the confidence rating immediately followed the act of unscrambling. This increase in confidence mirrors the “revelation effect” observed in word recognition experiments. In the present article, we analyzed our data using a new signal detection mixture distribution model that does not require the researcher to know the veracity of memory judgments a priori. Our analysis reveals that unscrambling a key word or an unrelated word affects response bias and discriminability in autobiographical memory tests in ways that are very similar to those that have been previously found for word recognition tasks.     

Hedonic arousal, time intervals, and excitation transfer

Zillmann’s (1978, 1983) excitation transfer theory was tested in relation to arousal processes opposite in hedonic valence. Sixty female subjects, randomly assigned to six experimental conditions, were initially exposed either to arousal stimulation positive in hedonic valence (sexual humor), or to arousal stimulation negative in hedonic valence (insulting remarks), or to a neutral unarousing task. In each condition, half of the subjects rated mildly arousing positively valenced male models’ photographs on several scales either immediately or with a 4 minute delay following prior arousal. In support of Zillmann’s theory, subjects in the positive hedonic arousal condition gave significantly lower photograph ratings on the arousal-related scales at the immediate, than at the delayed, rating time. However, contrary to Zillmann’s theory, the pattern of these ratings was reversed in the negative hedonic arousal condition, with the significantly higher ratings at the immediate, rather than delayed, rating time. Also, there were significant differences in these ratings among all three conditions both at the immediate and at the delayed rating time, but, as predicted, no significant differences were obtained for these ratings between the rating times in the control unarousing condition. The differences in excitation transfer between arousal processes positive and negative in hedonic valence may be due to their differences on a temporal dimension with regard to the speed of accumulation and the speed of dissipation of each arousal type.     

The detectability of the odor of butanol

Three experiments were conducted to measure the sensitivity of two Ss to the odor of butanol. In the first two experiments, the method of double random, yes-no staircases was used. A practice effect of over a log₁₀ unit in millimoles decrease in apparent threshold was observed in both Ss. Consistent shifts in the response criterion were induced when Ss were paid to meet an arbitrarily determined physical criterion. In Experiment S, the confidence rating procedure was used. Results at eight different signal intensities are of the form predicted by signal detection theory. d’ is shown to be related to signal strength by a power function with a slope of about .30 which suggests that the olfactory transducer compresses sensory input produced by weak concentrations of butanol.     

Searching for the sequential line-up advantage: A distinctiveness explanation

Signal Detection models as well as the Two-High-Threshold model (2HTM) have been used successfully as measurement models in recognition tasks to disentangle memory performance and response biases. A popular method in recognition memory is to elicit confidence judgements about the presumed old/new status of an item, allowing for the easy construction of ROCs. Since the 2HTM assumes fewer latent memory states than response options are available in confidence ratings, the 2HTM has to be extended by a mapping function which models individual rating scale usage. Unpublished data from 2 experiments in Bröder and Schütz (2009) validate the core memory parameters of the model, and 3 new experiments show that the response mapping parameters are selectively affected by manipulations intended to affect rating scale use, and this is independent of overall old/new bias. Comparisons with SDT show that both models behave similarly, a case that highlights the notion that both modelling approaches can be valuable (and complementary) elements in a researcher's toolbox.     

Type 2 tasks in the theory of signal detectability: Discrimination between correct and incorrect decisions

It has been known for over 40 years that there are two fundamentally different kinds of detection tasks in the theory of signal detectability. The Type 1 task is to distinguish between events defined independently of the observer; the Type 2 task is to distinguish between one’s own correct and incorrect decisions about those Type 1 events. For the Type 1 task, the behavior of the detector can be summarized by the traditional receiver operating characteristic (ROC) curve. This curve can be compared with a theoretical ROC curve, which can be generated from overlapping probability functions conditional on the Type 1 events on an appropriate decision axis. We show how to derive the probability functions underlying Type 2 decisions from those for the Type 1 task. ROC curves and the usual measures of performance are readily obtained from those Type 2 functions, and some relationships among various Type 1 and Type 2 performance measures are presented. We discuss the relationshiPbetween Type 1 and Type 2 confidence ratings and caution against the practice of presenting transformed Type 2 ratings as empirical Type 1 ratings.     

The calibration and resolution of confidence in perceptual judgments

Carleton University, Ottawa, Ontariot Canada Confidence rating based calibration and resolution indices were obtained in two experiments requiring perceptual comparisons and in a third with visual gap detection. Four important results were obtained. First, as in the general knowledge domain, subjects were underconfident when judgments were easy and overconfident when they were difficult. Second, paralleling the clear dependence of calibration on decisional difficulty, resolution decreased with increases in decision difficulty arising either from decreases in discriminability or from increasing demands for speed at the expense of accuracy. Third, providing trial-by-trial response feedback on difficult tasks improved resolution but had no effect on calibration. Fourth, subjects can accurately reportsubjective errors (i.e., trials in which they have indicated that they made an error) with their confidence ratings. It is also shown that the properties of decision time, conditionalized on confidence category, impose a rigorous set of constraints on theories of confidence calibration.     

A computational technique for improving estimates of discriminability and bias across multiple dimensions of choice

Brown and White (2009) proposed measures of discriminability and bias that accommodate additional dimensions of choice—and hence, bias—in conditional discriminations such as matching-to-sample and the yes-no signal detection task. Their proposed measures increase the statistical independence of discriminability and bias estimates, thus improving their accuracy. Because Brown and White’s (2009) equations partition response data more than do standard equations, however, their measures have a slightly lower ceiling. Consequently, measurements can be less accurate when there are few trials and discriminability and bias are extreme. We introduce a computational estimation technique that overcomes this limitation. It estimates Brown and White’s (2009) discriminability and bias measurements from an array of related measures that have a higher ceiling. Simulations show that resulting estimates of discriminability and bias are either comparable to or more accurate than measurements calculated from traditional equations or Brown and White’s (2009) direct measures, even with few trials. A worked example of our technique may be downloaded from brm.psychonomic-journals.org/ content/supplemental.     

SDT analysis of binary decisions and sensory intensity ratings to noxious thermal stimuli

During three sessions, each of 24 Ss responded to noxious thermal stimuli, using the following judgments: binary decision, S responded “high” or “low”; sensory intensity rating, S rated his sensory experience along a thermal intensity continuum; and concurrent report, S’s binary decision was followed by an intensity rating. The binary-decision d’ was significantly higher than the rating d′, suggesting that Ss could not maintain multiple thermal criteria in a consistent fashion. The criteria for pain obtained with single and concurrent intensity rating judgments did not differ. These results suggest that the most efficacious and valid method for the study of experimental pain is to obtain concurrent responses, and to use binary decisions to compute d’ and sensory intensity ratings to locate S’s criterion for reporting pain.     

Disambiguating conscious and unconscious influences: do exclusion paradigms demonstrate unconscious perception?

Does exclusion failure--responding with previously presented words despite instructions to avoid doing so--demonstrate unconscious influences? This article examines exclusion-based evidence for unconscious perception. I propose an alternative signal detection theory (SDT) framework that can account for exclusion failure and ostensibly convergent qualitative differences without positing additional unconscious perceptual mechanisms. In the proposed SDT model, exclusion failure is a criterion artifact, similar to classic SDT-based critiques of subjective threshold approaches. However, it is suggested that exclusion approaches do demonstrate that response strategies are applied only to above-criterion stimuli and thereby illustrate important qualitative differences between two conscious processes: phenomenal awareness itself and higher-order (i.e., metacognitive) decision processes.     

决策信心在信息化从众中的作用

研究采用JAS范式,通过操纵群体信息的性质（支持和反对）以及一致性程度,考察了决策信心在信息化从众过程中的作用。结果表明：（1）个体仅在接收到反对信息时会发生决策的偏转,表现出从众;（2）群体参照信息能够显著影响个体的信心：支持性的群体信息使个体的信心显著增高,而反对性的群体信息使信心显著降低,且表现出一种“负性偏向”（即个体对来自群体的负性信息更加敏感）;（3）在反对条件下,个体信心降低的程度能够很好地预测其决策偏转的概率,即信息化从众行为。这说明,决策信心可能在信息化从众中起着核心的中介作用——反对性的群体信息使得决策信心下降,而决策信心的下降导致了决策的偏转,从而表现出从众行为。