Assessing the belief bias effect with ROCs: reply to Dube, Rotello, and Heit (2010)

Dube, Rotello, and Heit (2010) argued (a) that the so-called receiver operating characteristic is nonlinear for data on belief bias in syllogistic reasoning; (b) that their data are inconsistent with Klauer, Musch, and Naumer's (see record 2000-02818-008) model of belief bias; (c) that their data are inconsistent with any of the existing accounts of belief bias and only consistent with a theory provided by signal detection theory; and (d) that in fact, belief bias is a response bias effect. In this reply, we present reanalyses of Dube et al.'s data and of old data suggesting (a) that the receiver operating characteristic is linear for binary "valid" versus "invalid" responses, as employed by the bulk of research in this field; (b) that Klauer et al.'s model describes the old data significantly better than does Dube et al.'s model and that it describes Dube et al.'s data somewhat better than does Dube et al.'s model; (c) that Dube et al.'s data are consistent with the account of belief bias by misinterpreted necessity, whereas Dube et al.'s signal detection model does not fit their data; and (d) that belief bias is more than a response bias effect.     

Traditional difference-score analyses of reasoning are flawed

Studies of the belief bias effect in syllogistic reasoning have relied on three traditional difference score measures: the logic index, belief index, and interaction index. Dube, Rotello, and Heit (2010, 2011) argued that the interaction index incorrectly assumes a linear receiver operating characteristic (ROC). Here, all three measures are addressed. Simulations indicated that traditional analyses of reasoning experiments are likely to lead to incorrect conclusions. Two new experiments examined the role of instructional manipulations on the belief bias effect. The form of the ROCs violated assumptions of traditional measures. In comparison, signal detection theory (SDT) model-based analyses were a better match for the form of the ROCs, and implied that belief bias and instructional manipulations are predominantly response bias effects. Finally, reanalyses of previous studies of conditional reasoning also showed non-linear ROCs, violating assumptions of traditional analyses. Overall, reasoning research using traditional measures is at risk of drawing incorrect conclusions.     

Accounting for belief bias in a mental model framework: comment on Klauer, Musch, and Naumer (2000)

K. C. Klauer, J. Musch, and B. Naumer (2000; see record 2000-02818-008) presented a general multinomial model of belief bias effects in syllogistic reasoning. They claimed to map a particular mental model account of belief bias (J. V. Oakhill, P. N. Johnson-Laird, & A. Garnham, 1989; see record 1989-38845-001)) onto this model and to show empirically that it is incorrect. The authors argue that this mental model account does not map onto the multinomial model and that it can account for the data presented by Klauer et al. (Experiments 1-4). The authors further argue that additional data Klauer et al. presented in support of a new model of their own (Experiments 5-8) are explained by this mental model account. The mental model account is, therefore, refuted neither by Klauer et al.'s theoretical analysis nor by any of the results they presented. Furthermore, the account can accommodate more recent findings on belief bias in a more satisfactory way than can alternative models that have been proposed.     

Binary ROCs in perception and recognition memory are curved

In recognition memory, a classic finding is that receiver operating characteristics (ROCs) are curvilinear. This has been taken to support the fundamental assumptions of signal detection theory (SDT) over discrete-state models such as the double high-threshold model (2HTM), which predicts linear ROCs. Recently, however, Br?der and Schütz (2009) challenged this argument by noting that most of the data on which support for SDT is based have involved confidence ratings. The authors argued that certain types of rating scale usage may result in curved ROCs even if the generating process is thresholded in nature. From this point of view, only ROCs constructed via experimental bias manipulations are useful for discriminating between the models. Br?der and Schütz conducted a meta-analysis and new experiments that compared SDT and the 2HTM using binary (yes-no) ROCs and found that many of these functions were linear, supporting 2HTM over SDT. We examine all the data reported by Br?der and Schütz, noting important limitations in their methodology, analyses, and conclusions. We report a new meta-analysis and 2 new experiments to examine the issue more closely while avoiding the limitations of Br?der and Schütz's study. These new data indicate that binary ROCs are curved in recognition, consistent with previous findings in perception and reasoning. Our results support classic arguments in favor of SDT and indicate that curvature in ratings ROCs is not task specific. We recommend the ratings procedure and suggest that analyses based on threshold models be treated with caution.     

Characterizing belief bias in syllogistic reasoning: A hierarchical Bayesian meta-analysis of ROC data

The belief-bias effect is one of the most-studied biases in reasoning. A recent study of the phenomenon using the signal detection theory (SDT) model called into question all theoretical accounts of belief bias by demonstrating that belief-based differences in the ability to discriminate between valid and invalid syllogisms may be an artifact stemming from the use of inappropriate linear measurement models such as analysis of variance (Dube et al., Psychological Review, 117(3), 831–863, 2010). The discrepancy between Dube et al.’s, Psychological Review, 117(3), 831–863 (2010) results and the previous three decades of work, together with former’s methodological criticisms suggests the need to revisit earlier results, this time collecting confidence-rating responses. Using a hierarchical Bayesian meta-analysis, we reanalyzed a corpus of 22 confidence-rating studies (N =?993). The results indicated that extensive replications using confidence-rating data are unnecessary as the observed receiver operating characteristic functions are not systematically asymmetric. These results were subsequently corroborated by a novel experimental design based on SDT’s generalized area theorem. Although the meta-analysis confirms that believability does not influence discriminability unconditionally, it also confirmed previous results that factors such as individual differences mediate the effect. The main point is that data from previous and future studies can be safely analyzed using appropriate hierarchical methods that do not require confidence ratings. More generally, our results set a new standard for analyzing data and evaluating theories in reasoning. Important methodological and theoretical considerations for future work on belief bias and related domains are discussed.     

Support for a continuous (single-process) model of recognition memory and source memory

Does memory retrieval occur in a continuous or an all-or-none manner? The shape of the receiver operating characteristic (ROC) has been used to answer this question, with curvilinear and linear memory ROCs indicating continuous and all-or-none retrieval processes, respectively. Signal detection models (e.g., the unequal variance model) correspond to a continuous retrieval process, whereas threshold models (including the multinomial model and the recollection component of the dual-process model) correspond to an all-or-none process. In studies of source memory, Slotnick et al. (2000) and others have observed curvilinear ROCs (supporting the unequal variance model), whereas Yonelinas (1999) observed linear ROCs (supporting the dual-process model). We resolve these seemingly inconsistent results, showing that source memory ROCs are naturally curvilinear but can appear linear when nondiagnostic source information is included in the analysis. Furthermore, the unequal variance model accounted for both recognition memory and source memory ROCs, supporting a continuous process of memory retrieval.     

Recollection is a continuous process: Evidence from plurality memory receiver operating characteristics

Is recollection a continuous/graded process or a threshold/all-or-none process? Receiver operating characteristic (ROC) analysis can answer this question as the continuous model and the threshold model predict curved and linear recollection ROCs, respectively. As memory for plurality, an item's previous singular or plural form, is assumed to rely on recollection, the nature of recollection can be investigated by evaluating plurality memory ROCs. The present study consisted of four experiments. During encoding, words (singular or plural) or objects (single/singular or duplicate/plural) were presented. During retrieval, old items with the same plurality or different plurality were presented. For each item, participants made a confidence rating ranging from “very sure old”, which was correct for same plurality items, to “very sure new”, which was correct for different plurality items. Each plurality memory ROC was the proportion of same versus different plurality items classified as “old” (i.e., hits versus false alarms). Chi-squared analysis revealed that all of the plurality memory ROCs were adequately fit by the continuous unequal variance model, whereas none of the ROCs were adequately fit by the two-high threshold model. These plurality memory ROC results indicate recollection is a continuous process, which complements previous source memory and associative memory ROC findings.     

A misleading review of response bias: comment on McGrath, Mitchell, Kim, and Hough (2010)

In the May 2010 issue of Psychological Bulletin, R. E. McGrath, M. Mitchell, B. H. Kim, and L. Hough published an article entitled "Evidence for Response Bias as a Source of Error Variance in Applied Assessment" (pp. 450-470). They argued that response bias indicators used in a variety of settings typically have insufficient data to support such use in everyday clinical practice. Furthermore, they claimed that despite 100 years of research into the use of response bias indicators, "a sufficient justification for [their] use… in applied settings remains elusive" (p. 450). We disagree with McGrath et al.'s conclusions. In fact, we assert that the relevant and voluminous literature that has addressed the issues of response bias substantiates validity of these indicators. In addition, we believe that response bias measures should be used in clinical and research settings on a regular basis. Finally, the empirical evidence for the use of response bias measures is strongest in clinical neuropsychology. We argue that McGrath et al.'s erroneous perspective on response bias measures is a result of 3 errors in their research methodology: (a) inclusion criteria for relevant studies that are too narrow; (b) errors in interpreting results of the empirical research they did include; (c) evidence of a confirmatory bias in selectively citing the literature, as evidence of moderation appears to have been overlooked. Finally, their acknowledging experts in the field who might have highlighted these errors prior to publication may have prevented critiques during the review process.     

Debiasing by instruction: The case of belief bias

Abstract

The study is concerned with the question of whether robust biases in reasoning can be reduced or eliminated by verbal instruction in principles of reasoning. Three experiments are reported in which the effect of instructions upon the belief bias effect in syllogistic reasoning is investigated. Belief bias is most clearly marked by a tendency for subjects to accept invalid conclusions which are a priori believable. Experiment 1 attempted to replicate and extend an experiment reported by Newstead, Pollard, Evans and Allen (1992). In contrast with their experiment, it was found that belief bias was maintained despite the use of augmented instructions which emphasised the principle of logical necessity. Experiment 2 provided an exact replication of the augmented instructions condition of Newstead et al., including the presence of problems with belief-neutral conclusions. Once again, significant effects of conclusion believability were found. A third experiment examined the use of elaborated instructions which lacked specific reference to the notion of logical necessity. The use of these instructions significantly reduced the effects of belief on the reasoning observed.

Taking the current findings together with the experiment of Newstead et al., the overall conclusion is that elaborated instructions can reduce the belief bias effect in syllogistic reasoning, but not eliminate it. This conclusion is discussed with reference to (1) the practical implications for improving thinking and reasoning via verbal instruction and (2) the nature of the belief bias phenomenon.     

Using forced choice to test belief bias in syllogistic reasoning

In deductive reasoning, believable conclusions are more likely to be accepted regardless of their validity. Although many theories argue that this belief bias reflects a change in the quality of reasoning, distinguishing qualitative changes from simple response biases can be difficult (Dube, Rotello, & Heit, 2010). We introduced a novel procedure that controls for response bias. In Experiments 1 and 2, the task required judging which of two simultaneously presented syllogisms was valid. Surprisingly, there was no evidence for belief bias with this forced choice procedure. In Experiment 3, the procedure was modified so that only one set of premises was viewable at a time. An effect of beliefs emerged: unbelievable conclusions were judged more accurately, supporting the claim that beliefs affect the quality of reasoning. Experiments 4 and 5 replicated and extended this finding, showing that the effect was mediated by individual differences in cognitive ability and analytic cognitive style. Although the positive findings of Experiments 3–5 are most relevant to the debate about the mechanisms underlying belief bias, the null findings of Experiments 1 and 2 offer insight into how the presentation of an argument influences the manner in which people reason.     

From humans to rats and back again: bridging the divide between human and animal studies of recognition memory with receiver operating characteristics

Receiver operating characteristics (ROCs) have been used extensively to study the processes underlying human recognition memory, and this method has recently been applied in studies of rats. However, the extent to which the results from human and animal studies converge is neither entirely clear, nor is it known how the different methods used to obtain ROCs in different species impact the results. A recent study used a response bias ROC manipulation with rats and demonstrated that speeding memory responses reduced the contribution of recollection, not familiarity. The current study confirms this finding in humans using a comparable response bias method. Moreover, a comparison of the response bias methods commonly used in animal studies and the confidence rating method typically employed in human studies produced similar ROC functions. The present results suggest that the analysis of recognition memory ROCs provides a fruitful method to bridge the human and animal memory literatures.     

Effects of belief and logic on syllogistic reasoning: Eye-movement evidence for selective processing models

Studies of syllogistic reasoning have demonstrated a nonlogical tendency for people to endorse more believable conclusions than unbelievable ones. This belief bias effect is more dominant on invalid syllogisms than valid ones, giving rise to a logic by belief interaction. We report an experiment in which participants' eye movements were recorded in order to provide insights into the nature and time course of the reasoning processes associated with manipulations of conclusion validity and believability. Our main dependent measure was people's inspection times for syllogistic premises, and we tested predictions deriving from three contemporary mental-models accounts of the logic by belief interaction. Results supported recent "selective processing" theories of belief bias (e.g., Evans, 2000; Klauer, Musch, & Naumer, 2000), which assume that the believability of a conclusion biases model construction processes, rather than biasing the search for falsifying models (e.g., Oakhill & Johnson-Laird, 1985) or a response stage of reasoning arising from subjective uncertainty (e.g., Quayle & Ball, 2000). We conclude by suggesting that the eye-movement analyses in reasoning research may provide a useful adjunct to other process-tracing techniques such as verbal protocol analysis.     

The contribution of recollection and familiarity to recognition and source-memory judgments: a formal dual-process model and an analysis of receiver operating characteristics

A formal dual-process model that assumes that memory judgments can be based on a threshold recollection process and a signal-detection-based familiarity process is proposed to account for both recognition and source-memory performance. The model was tested in 4 experiments by examining recognition and source-memory receiver operating characteristics (ROCs). In agreement with the predictions of the model, recognition and source memory dissociated in certain conditions. Recognition ROCs were curvilinear in probability space and relatively linear in z-space, as expected if recollection and familiarity contributed to performance. In contrast, source ROCs typically were linear and exhibited a pronounced U shape in z-space, as expected if performance primarily relied on recollection. However, in conditions in which familiarity was clearly indicative of an item's source, the source ROC became curvilinear, suggesting that participants could use familiarity as a basis for source judgments. Several alternative models, including the unequal-variance signal-detection model, were found to be inconsistent with the ROC data.     

Optimistic update bias holds firm: Three tests of robustness following Shah et al.

A diverse body of research has demonstrated that people update their beliefs to a greater extent when receiving good news compared to bad news. Recently, a paper by Shah et al. claimed that this asymmetry does not exist. Here we carefully examine the experiments and simulations described in Shah et al. and follow their analytic approach on our data sets. After correcting for confounds we identify in the experiments of Shah et al., an optimistic update bias for positive life events is revealed. Contrary to claims made by Shah et al., we observe that participants update their beliefs in a more Bayesian manner after receiving good news than bad. Finally, we show that the parameters Shah et al. pre-selected for simulations are at odds with participants’ data, making these simulations irrelevant to the question asked. Together this report makes a strong case for a true optimistic asymmetry in belief updating.     

ROCs in Eyewitness Identification: Instructions versus Confidence Ratings

From the perspective of signal detection theory, different lineup instructions may induce different levels of response bias. If so, then collecting correct and false identification rates across different instructional conditions will trace out the receiver operating characteristic (ROC)—the same ROC that, theoretically, could also be traced out from a single instruction condition in which each eyewitness decision is accompanied by a confidence rating. We tested whether the two approaches do in fact yield the same ROC. Participants were assigned to a confidence rating condition or to an instructional biasing condition (liberal, neutral, unbiased, or conservative). After watching a video of a mock crime, participants were presented with instructions followed by a six‐person simultaneous photo lineup. The ROCs from both methods were similar, but they were not exactly the same. These findings have potentially important policy implications for how the legal system should go about controlling eyewitness response bias.Copyright © 2017 John Wiley & Sons, Ltd.     

Cultural Differences in Belief Bias Associated with Deductive Reasoning?

Norenzayan, Smith, Jun Kim, and Nisbett (2002) investigated cultural differences in the use of intuitive versus formal reasoning in 4 experiments. Our comment concerns the 4th experiment where Norenzayan et al. reported that, although there were no cultural differences in accuracy on abstract logical arguments, Koreans made more errors than U.S. undergraduates in judging the logical validity of concrete arguments. Norenzayan et al. concluded that Koreans are less likely than European Americans to decontextualize an argument's content from its logical structure, as Koreans were more likely to consider the believability of the conclusion when assessing an argument's validity (a belief bias). Notably, Korean participants were more conservative (less likely to say an argument is valid) than European American participants when assessing arguments. An analysis of the average of the hit and correct rejection rates in each of the conditions (abstract, concrete–believable, concrete–nonbelievable) revealed that, contrary to conclusions of Norenzayan et al., European Americans were no better than Koreans at determining the validity of concrete deductive arguments with conclusions varying in believability.     

The influence of averaging and noisy decision strategies on the recognition memory ROC

Many single- and dual-process models of recognition memory predict that the ratings and remember-know receiver operating characteristics (ROCs) are the same, but Rotello, Macmillan, and Reeder (2004) reported that the slopes of the remember-know and ratingsz-transformed ROCs (zROCs) are different. The authors show that averaging introduces nonlinearities to the form of thezROC and that ratings and remember-knowzROCs are indistinguishable when constructed in a conventional manner. The authors show, further, that some nonoptimal decision strategies have a distinctive, nonlinear effect on the form of the single-process continuous-statezROC. The conclusion is that many factors having nothing to do with the nature of recognition memory can affect the shape ofzROCs, and that therefore, the shape of thezROC does not, alone, characterize different memory models.     

The validity and appropriateness of methods, analyses, and conclusions in Rind et al. (1998): A rebuttal of victimological critique from Ondersma et al. (2001) and Dallam et al. (2001).

The authors respond to 2 victimological critiques of their 1998 meta-analysis on child sexual abuse (CSA). S. J. Dallam et al. (2001) claimed that B. Rind, P. Tromovitch, and R. Bauserman (1998) committed numerous methodological and statistical errors, and often miscoded and misinterpreted data. The authors show all these claims to be invalid. To the contrary, they demonstrate frequent bias in Dallam et al.'s criticisms. S. J. Ondersma et al. (2001) claimed that Rind et al.'s study is part of a backlash against psychotherapists, that its suggestions regarding CSA definitions were extrascientific, and that the moral standard is needed to understand CSA scientifically. The authors show their suggestions to have been scientific and argue that it is Ondersma et al.'s issue-framing and moral standard that are extrascientific. This reply supports the original methods, analyses, recommendations, and conclusions of Rind et al.     

Necessity, possibility and belief: A study of syllogistic reasoning

The present study extended the investigation of the belief bias effect in syllogistic reasoning in two ways: (1) The effect was studied under instructions to decide whether conclusions were possible, as well as necessary, given the premises; and (2) the effect was studied for types of syllogism where people rarely endorse the conclusions as well as those (valid and fallacious) where endorsements are common. Three experiments are reported, which show first that there is a marked tendency to reject unbelievable conclusions relative to abstract or neutral controls on all kinds of syllogism and under both types of instruction. There was also significant evidence of positive belief bias (increased acceptance of believable conclusions) and of interactions between belief bias effects and logical form. The results are discussed with particular respect to accounts of belief bias offered by theorists in the mental-model tradition.     

Signal-Detection, Threshold, and Dual-Process Models of Recognition Memory: ROCs and Conscious Recollection

Threshold- and signal-detection-based models have dominated theorizing about recognition memory. Building upon these theoretical frameworks, we have argued for a dual-process model in which conscious recollection (a threshold process) and familiarity (a signal-detection process) contribute to memory performance. In the current paper we assessed several memory models by examining the effects of levels of processing and the number of presentations on recognition memory receiver operating characteristics (ROCs). In general, when the ROCs were plotted in probability space they exhibited an inverted U shape; however, when they were plotted inzspace they exhibited a U shape. An examination of the ROCs showed that the dual-process model could account for the observed ROCs, but that models based solely on either threshold or signal-detection processes failed to provide a sufficient account of the data. Furthermore, an examination of subjects' introspective reports using the remember/know procedure showed that subjects were aware of recollection and familiarity and were able to consistently report on their occurrence. The remember/know data were used to accurately predict the shapes of the ROCs, and estimates of recollection and familiarity derived from the ROC data mirrored the subjective reports of these processes.