Applying signal-detection theory to the study of observer accuracy and bias in behavioral assessment

We evaluated the feasibility and utility of a laboratory model for examining observer accuracy within the framework of signal-detection theory (SDT). Sixty-one individuals collected data on aggression while viewing videotaped segments of simulated teacher-child interactions. The purpose of Experiment 1 was to determine if brief feedback and contingencies for scoring accurately would bias responding reliably. Experiment 2 focused on one variable (specificity of the operational definition) that we hypothesized might decrease the likelihood of bias. The effects of social consequences and information about expected behavior change were examined in Experiment 3. Results indicated that feedback and contingencies reliably biased responding and that the clarity of the definition only moderately affected this outcome.     

Bias in self-evaluation: Signal probability effects

Two experiments examined apparent signal probability effects in simple verbal self-reports. After each trial of a delayed matching-to-sample task, young adults pressed either a “yes” or a “no” button to answer a computer-presented query about whether the most recent choice met a point contingency requiring both speed and accuracy. A successful matching-to-sample choice served as the “signal” in a signal-detection analysis of self-reports. Difficulty of matching to sample, and thus signal probability, was manipulated via the number of nonmatching sample and comparison stimuli. In Experiment 1, subjects exhibited a bias (log b) for reporting matching-to-sample success when success was frequent, and no bias or a bias for reporting failure when success was infrequent. Contingencies involving equal conditional probabilities of point consequences for “I succeeded” and “I failed” reports had no systematic effect on this pattern. Experiment 2 found signal probability effects to be evident regardless of whether referent-response difficulty was manipulated in different conditions or within sessions. These findings indicate that apparent signal probability effects in self-report bias that were observed in previous studies probably were not an artifact of contingencies intended to improve self-report accuracy or of the means of manipulating signal probability. The findings support an analogy between simple self-reports and psychophysical judgments and bolster the conclusion of Critchfield (1993) that signal probability effects can influence simple self-reports much as they do reports about external stimuli in psychophysical experiments.     

Traditional difference-score analyses of reasoning are flawed

Studies of the belief bias effect in syllogistic reasoning have relied on three traditional difference score measures: the logic index, belief index, and interaction index. Dube, Rotello, and Heit (2010, 2011) argued that the interaction index incorrectly assumes a linear receiver operating characteristic (ROC). Here, all three measures are addressed. Simulations indicated that traditional analyses of reasoning experiments are likely to lead to incorrect conclusions. Two new experiments examined the role of instructional manipulations on the belief bias effect. The form of the ROCs violated assumptions of traditional measures. In comparison, signal detection theory (SDT) model-based analyses were a better match for the form of the ROCs, and implied that belief bias and instructional manipulations are predominantly response bias effects. Finally, reanalyses of previous studies of conditional reasoning also showed non-linear ROCs, violating assumptions of traditional analyses. Overall, reasoning research using traditional measures is at risk of drawing incorrect conclusions.     

The effect of signal probability on the slope of the receiver operating characteristic given by the rating procedure

Signal detection theory (SDT) requires that the slope of the receiver operating characteristic (ROC) is independent of the probability of signal and noise. But it has been shown that when the rating procedure is applied to detection, the slope of the receiver operating characteristic may increase as a function of the probability of the signal (Schulman & Greenberg, 1970). This presents a serious problem for signal detection theory. This problem is examined in relation to a recent theory of criterion setting (Treisman & Williams, 1984), and an explanation for the effect of signal probability on ROC slope is derived which is compatible with SDT. Further data on the relation between signal probability and ROC slope are reported.     

Memorial Monitoring and Control: How Confidence and Social and Financial Consequences Affect Eyewitnesses' Reporting of Fine‐Grain Information

Research suggests that accessible coarse‐grain (i.e. general) information is sometimes under‐reported in memory accounts. In two experiments, socially motivating conditions were manipulated to determine whether, and under what contexts, coarse‐grain information is avoided because eyewitnesses are motivated to be specific and willing to risk accuracy. In Experiment 1, response privacy (private, public) and audience (high authority, low authority) were manipulated. Response privacy and penalty for inaccurate reporting (penalty, no penalty) were manipulated in Experiment 2. Across both experiments, eyewitnesses' confidence estimation (i.e. monitoring) was effective, suggesting that coarse‐grain information is under‐reported through poor decision making (i.e. control). Eyewitnesses avoided coarse‐grain information because they sometimes displayed a bias towards reporting fine‐grain information. This bias was more apparent, and coarse‐grain information avoiding more likely, when the perceived consequences for reporting were minimal (i.e. when in private in Experiment 1 and when no penalty for inaccuracy was imposed in Experiment 2). Copyright © 2016 John Wiley & Sons, Ltd.     

The detection model of recognition using know and remember judgments

The signal detection model forknow andremember recognition judgments was tested in two experiments. In Experiment 1, two predictions of the model were tested: (1) that measures of memory sensitivity,A′, are equivalent in value when based on either the recognition (know or remember) criterion or on the remember criterion; and (2) that there is a positive correlation between recognition bias and the proportion of know judgments that are hits, but no correlation between recognition bias and proportion of remember hits (Donaldson, 1996). Both predictions were supported by the data. In Experiment 2, the context of test items was manipulated to make it more or less similar to learning context. The detection model requires that memory sensitivity be the same for both recognition and remember judgments, regardless of test context. Alternatively, if remember judgments reflect only the retrieval of episodic information from memory, the two measures of memory sensitivity should become more disparate in value as learning and test context are made more similar. Memory sensitivity was generally the same in value for recognition and remember criteria but different across context conditions, thus supporting the detection model. The nature of the memory continuum used in detection theory is also discussed.     

Eye movements and a dynamic stimulus situation

Two experiments were conducted to investigate performance with and without voluntary eye movements in a dynamic stimulus situation. Experiment I used a combined tracking and prediction task. Level of training, complexity of the signal, and visual region sampled were the variables of interest. Experiment II manipulated the same variables in only the prediction task. Thus, the amount of attention allotted to the prediction task was varied between experiments. The d' measure indicated that under peripheral vision instructions accuracy on the prediction task was the same as under foveal vision instructions provided that: (1) the level of task complexity was low, (2) the subjects were well trained, and (3) only the prediction task was performed, or in the dual task situation only visual regions near the fovea were sampled. All other combinations of the variables resulted in a lower performance scores under peripheral vision instructions. Results are interpreted within the framework of current theories of the functional visual field.     

Task predictability and remembered duration

The effects of structural predictability on remembered duration judgments were examined within the context of the performance of a series of highly familiar tasks. Across a set of three experiments, task predictability was manipulated by the presence or absence of advance expectancies of what tasks were to be performed (Experiment 1), an (in)variant ordering of task performance (Experiment 2), and the placement of interruptions at between- versus within-task locations (Experiment 3). In each case, a higher degree of predictability led to more accurate and reliable duration estimates that were relatively free of bias, while uncertainty decreased accuracy through an overestimation bias. These results not only render insight into the mediational mechanisms responsible for temporal judgments, but also sug-gest some practical applications for everyday behavior.     

The Impact of Asymmetric Partial Feedback on Response‐Bias

For many decisions we make, we are more likely to receive feedback about some outcomes than about others, and thus we must operate under conditions of asymmetric partial feedback. In this paper, we examine the effects of asymmetric partial feedback on decision making across three experiments. Experiment 1 examines the effects of four partial feedback schedules on decision making in a single‐attribute auditory detection task. Experiment 2 examines these same schedules in the context of a simulated experiment in which virtual participants perform a multi‐attribute choice task. Experiment 3 then examines effects of one partial feedback schedule representative of a real‐world decision‐making domain (personnel selection) using an ecologically informed multi‐attribute choice task. All three experiments reveal that asymmetric partial feedback primarily impacts on response‐bias, a parameter of signal detection theory. These findings can be explained in terms of decision makers' attempts to maximize accuracy and the fact that shifts in response‐bias will, under partial feedback, produce misleading impressions of greater accuracy. Copyright © 2013 John Wiley & Sons, Ltd.     

The whereabouts of visual attention: Involuntary attentional bias toward the default gaze direction

This study proposed and verified a new hypothesis on the relationship between gaze direction and visual attention: attentional bias by default gaze direction based on eye-head coordination. We conducted a target identification task in which visual stimuli appeared briefly to the left and right of a fixation cross. In Experiment 1, the direction of the participant’s head (aligned with the body) was manipulated to the left, front, or right relative to a central fixation point. In Experiment 2, head direction was manipulated to the left, front, or right relative to the body direction. This manipulation was based on results showing that bias of eye position distribution was highly correlated with head direction. In both experiments, accuracy was greater when the target appeared at a position where the eyes would potentially be directed. Consequently, eye–head coordination influences visual attention. That is, attention can be automatically biased toward the location where the eyes tend to be directed.     

On Judgments of Approximately Equal

Although precision is often important in quantitative judgment, sometimes, it is valuable to recognize that two quantities are roughly the same. Fuzzy‐trace theory suggests that approximately equal judgments rely on gist representations (i.e., meaningful fuzzy categories of quantity). We conducted three experiments to investigate approximately equal judgments with number pairs presented in different formats, both with and without semantic content (breast cancer statistics). In each study, the ratio of the smaller divided by larger number predicted approximately equal judgments. Experiment 1 also examined how knowledge of breast cancer, presentation format (frequencies vs. percentages), and differences in gist comprehension of breast cancer information influence fuzzy equality judgments. As predicted by the fuzzy‐trace theory concept of denominator neglect, approximately equal judgments were more sensitive, as measured by signal detection theory (SDT) analyses, when presented as percentages. In both experiments with breast cancer statistics, people were more likely to judge number pairs approximately equal when they were embedded in sentences about breast cancer, and breast cancer knowledge predicted increased perception of equality, when appropriately consistent with reliable sources, and increased judgment sensitivity. In Experiment 2, a simple intervention focusing on gist meaning increased source‐consistent approximately equal judgments, increased SDT judgment sensitivity, and decreased SDT response bias. In Experiment 3, using number pairs devoid of semantic context spanning four orders of magnitude, we further examined ratio similarity. Overall, more knowledgeable judges and those who better understood the gist of meaningful numbers were more likely to judge literally different numbers as “approximately equal” rather than make precise discriminations that were meaningless. Copyright © 2017 John Wiley & Sons, Ltd.     

Selective attention fails to alter the dichotic listening lag effect: evidence that the lag effect is preattentional

Berlin et al. (1973) reported that either stimulus from a dichotic pair of consonant-vowel syllables is processed preferentially when its presentation is delayed by 30-60 ms. In the first of three experiments with 60 normal right-handed adults, we replicated the Berlin et al. "lag effect," but only for asynchronies between 60 and 90 ms. In Experiment 2 subjects focused attention selectively on one ear. The results indicated that focused attention and stimulus asynchrony have additive effects: Performance improved at the attended ear irrespective of stimulus asynchrony, but the lag effect remained unchanged relative to the divided-attention condition. Experiment 3 entailed a signal detection task that allowed separate analysis of detection and localization accuracy. As in previous studies, selective attention to one ear increased the accuracy of localization but not detection at the attended ear. Both dependent measures indicated a lag effect that remained invariant as attention was manipulated. These findings imply that the lag effect is attributable to a preattentional stage of auditory processing.     

Signal detection theory with finite mixture distributions: theoretical developments with applications to recognition memory

An extension of signal detection theory (SDT) that incorporates mixtures of the underlying distributions is presented. The mixtures can be motivated by the idea that a presentation of a signal shifts the location of an underlying distribution only if the observer is attending to the signal; otherwise, the distribution is not shifted or is only partially shifted. Thus, trials with a signal presentation consist of a mixture of 2 (or more) latent classes of trials. Mixture SDT provides a general theoretical framework that offers a new perspective on a number of findings. For example, mixture SDT offers an alternative to the unequal variance signal detection model; it can also account for nonlinear normal receiver operating characteristic curves, as found in recent research.     

Motivation sharpens exogenous spatial attention

Although both attention and motivation affect behavior, how these 2 systems interact is currently unknown. To address this question, 2 experiments were conducted in which participants performed a spatially cued forced-choice localization task under varying levels of motivation. Participants were asked to indicate the location of a peripherally cued target while ignoring a distracter. Motivation was manipulated by varying magnitude and valence (reward and punishment) of an incentive linked to task performance. Attention was manipulated via a peripheral cue, which correctly predicted the presence of a target stimulus on 70% of the trials. Taken together, our findings revealed that the signal detection measure, reflecting perceptual sensitivity, increased as a function of incentive value during both valid and invalid trials. In addition, trend analyses revealed a linear increase in detection sensitivity as a function of incentive magnitude for both reward and punishment conditions. Our results suggest that elevated motivation leads to improved efficiency in orienting and reorienting of exogenous spatial attention and that one mechanism by which attention and motivation interact involves the sharpening of attention during motivationally salient conditions.     

Schema bias in source monitoring varies with encoding conditions: Support for a probability-matching account

Two experiments examined reliance on schematic knowledge in source monitoring. Based on a probability-matching account of source guessing, a schema bias will only emerge if participants do not have a representation of the source-item contingency in the study list, or if the perceived contingency is consistent with schematic expectations. Thus, the account predicts that encoding conditions that affect contingency detection also affect schema bias. In Experiment 1, the schema bias commonly found when schematic information about the sources is not provided before encoding was diminished by an intentional source-memory instruction. In Experiment 2, the depth of processing of schema-consistent and schema-inconsistent source-item pairings was manipulated. Participants consequently overestimated the occurrence of the pairing type they processed in a deep manner, and their source guessing reflected this biased contingency perception. Results support the probability-matching account of source guessing. (PsycINFO Database Record (c) 2012 APA, all rights reserved).     

Self-enhancement and theory-based correction processes

Two studies examined bias correction by manipulating a perceived chronic judgmental bias (i.e., overestimator/underestimator) using a modified dot estimation task. In Experiment 1, participants corrected for this perceived estimation bias by making adjustments away from the arbitrary feedback about their personal bias tendencies. In Experiment 2, the perceived desirability of the same estimation bias was manipulated. Results indicated that self-enhancement concerns impacted perceivers’ motivation to correct, at a cost to accuracy. These studies expand our current understanding of theory-based correction by including self-enhancement motives as causes of correction, demonstrating that such corrections can decrease rather than increase judgment accuracy, and illustrating the usefulness of a new perceived bias manipulation in theory testing.     

Neural and behavioral effects of regulating emotional responses to errors during an implicit racial bias task

Affect regulation plays a key role in several theories of racial bias reduction. Here, we tested whether engaging in emotion regulation strategies while performing an implicit racial bias task (Weapons Identification Task; WIT) would alter neural and behavioral manifestations of bias. Participants either suppressed or reappraised in a positive light the distress associated with making errors during the WIT, while an electroencephalogram (EEG) was recorded. We hypothesized that engaging in emotion regulation strategies would reduce the distress associated with making errors indicative of bias, resulting in smaller error-related negativity (ERN) amplitude during errors and increased expression of racial bias. Results of within-subjects comparisons (Experiment 1) generally supported these predictions. However, when emotion regulation strategies were manipulated between subjects (Experiment 2) there was no effect of suppression or reappraisal on bias expression. Across both experiments, engaging in emotion regulation led to larger ERNs for errors occurring on Black- relative to White-primed trials. In addition, a number of significant order effects were observed, indicating important differences in the effects of engaging in emotion regulation strategies when those strategies are attempted in participants’ first versus second block of trials. No such order effects were evident when a second trial block was completed with no emotion regulation instructions. Findings are discussed in terms of the need for greater specificity in experimental tests of emotion regulation on error processing and cognitive performance.     

Belief-based and analytic processing in transitive inference depends on premise integration difficulty

Three experiments investigated belief-based versus analytic processing in transitive inference. Belief-based and analytic processing were inferred from conclusion acceptance rates for problems with conclusions that were either valid or invalid and believable or unbelievable. Premise integration difficulty was manipulated by varying premise integration time (Experiment 1), premise presentation order (Experiment 2), and the markedness of the relational terms in the premises (Experiment 3). In all the conditions, reasoning accuracy and rated confidence were lower on conflict problems, where belief-based and analytic processes yielded different responses. Participants relied more on analytic processing and less on belief-based processing in conditions in which premise integration was easier. Fluid intelligence and premise integration ability predicted analytical reasoning on conflict problems after reasoning on the no-conflict problems was controlled for. The findings were related to three dual-process models of belief bias. They provide the first evidence of belief bias in transitive inference.     

Action outcome probability influences the size of the head-fake effect in basketball

Both kinematic and contextual information (e.g., action outcome probability) play a significant role in action anticipation. However, few researchers have examined the reciprocal influence of the two types of information and fewer still have investigated this issue for deceptive actions in sports. In the present study, we investigate the impact of action outcome probability on the processing of deceptive kinematic cues for the head fake in basketball. We manipulated the probability of the action outcome to either pass the ball to the left or to the right side (i.e., 75%, 50%, 25%) and examined how this contextual information affected the influence of head orientation on pass direction judgments. Outcome probability information was either provided explicitly (Experiment 1) or implicitly (Experiment 2). Both experiments indicated an increased head-fake effect with increasing outcome probability. Moreover, the bias to respond in line with the player’s head direction increased linearly with outcome probability. Also, discriminability between deceptive and genuine actions was poorer for high outcome probability (75%) associated with head orientation than for the 25% and 50% values. Last, a stronger response bias toward the higher probability side for deceptive trials than for genuine trials was only significant when the outcome probability information was processed implicitly in Experiment 2. The results of this study fit well with recent literature on contextual information in action prediction and are discussed in light of confirmation bias and signal detection theory.     

Positive and negative remember judgments and ROCs in the plurals paradigm: Evidence for alternative decision strategies

Using old-new ratings and remember-know judgments we explored the plurals paradigm, in which studied words must be distinguished from plurality-changed lures. The paradigm allowed us to investigate negative remembering—that is, the remembering of a plural-altered study item; capacity for this judgment was found to be poorer than or equivalent to the conventional positive remembering. A response-bias manipulation affected positive but not negative remembering. The ratings were used to construct ROC curves and test the prediction of the most common dual-process theory of recognition memory (Yonelinas, 2001) that the amount of recollection can be independently estimated from ROC curves and from remember judgments. By fitting the individual data with pure signal detection theory (SDT) models and dual-process models that combined SDT and high-threshold components (HTSDT), we identified two types of subjects. For those who were better described by HTSDT, the predicted convergence of remember-know and ROC measures was observed. For those who were better described by SDT, the ROC intercept could not predict the remember rate. The data are consistent with the idea that all subjects rely on the same representation but base their decisions on different partitions of a decision space.