Response error and processing biases in confidence judgment

Previous studies showed that random error can explain overconfidence effects typically observed in the literature. One of these studies concluded that, after accounting for random error effects in the data, there is little support for cognitive‐processing biases in confidence elicitation. In this paper, we investigate more closely the random error explanation for overconfidence. We generated data from four models of confidence and then estimated the magnitude of random error in the data. Our results show that, in addition to the true magnitude of random error specified in the simulations, the error estimates are influenced by important cognitive‐processing biases in the confidence elicitation process. We found that random error in the response process can account for the degree of overconfidence found in calibration studies, even when that overconfidence is actually caused by other factors. Thus, the error models say little about whether cognitive biases are present in the confidence elicitation process. Copyright © 2008 John Wiley & Sons, Ltd.     

The asymptotic posterior normality of the latent trait in an IRT model

It has long been part of the item response theory (IRT) folklore that under the usual empirical Bayes unidimensional IRT modeling approach, the posterior distribution of examinee ability given test response is approximately normal for a long test. Under very general and nonrestrictive nonparametric assumptions, we make this claim rigorous for a broad class of latent models.This research was partially supported by Office of Naval Research Cognitive and Neural Sciences Grant N0014-J-90-1940, 442-1548, National Science Foundation Mathematics Grant NSF-DMS-91-01436, and the National Center for Supercomputing Applications. We wish to thank Kumar Joag-dev and Zhiliang Ying for enlightening suggestions concerning the proof of the basic result.The authors wish to thank Kumar Joag-Dev, Brian Junker, Bert Green, Paul Holland, Robert Mislevy, and especially Zhiliang Ying for their useful comments and discussions.     

Themes in Therapist Development During the First Three Months of Clinical Experience

The results of a qualitative research study designed to better understand the developmental processes active in beginning therapists during the first three months of clinical contact is reported. Thirteen beginning therapists were asked to complete a monthly log describing experiences impacting their clinical work and themselves as therapists during their first three months of client contact. Data analysis revealed that the primary developmental theme active during this period of time is the development of therapist confidence. Two additional themes, the development of an internal gauge on which to evaluate current experiences and the development of boundaries around the self as a professional, were also found to be active during this time. These additional themes support the development of therapist confidence. Implications for training and research are identified.     

Application of the bootstrap methods in factor analysis

A Monte Carlo experiment is conducted to investigate the performance of the bootstrap methods in normal theory maximum likelihood factor analysis both when the distributional assumption is satisfied and unsatisfied. The parameters and their functions of interest include unrotated loadings, analytically rotated loadings, and unique variances. The results reveal that (a) bootstrap bias estimation performs sometimes poorly for factor loadings and nonstandardized unique variances; (b) bootstrap variance estimation performs well even when the distributional assumption is violated; (c) bootstrap confidence intervals based on the Studentized statistics are recommended; (d) if structural hypothesis about the population covariance matrix is taken into account then the bootstrap distribution of the normal theory likelihood ratio test statistic is close to the corresponding sampling distribution with slightly heavier right tail.This study was carried out in part under the ISM cooperative research program (91-ISM · CRP-85, 92-ISM · CRP-102). The authors would like to thank the editor and three reviewers for their helpful comments and suggestions which improved the quality of this paper considerably.     

Calibration of sensory and cognitive judgments: A single model for both

In a recent issue of this journal, Winman and Juslin (34 , 135–148, 1993) present a model of the calibration of subjective probability judgments for sensory discrimination tasks. They claim that the model predicts a pervasive underconfidence bias observed in such tasks, and present evidence from a training experiment that they interpret as supporting the notion that different models are needed to describe judgment of confidence in sensory and in cognitive tasks. The model is actually part of the more comprehensive decision variable partition model of subjective probability calibration that was originally proposed in Ferrell and McGoey (Organizational Behavior and Human Performance, 26 , 32–53, 1980). The characteristics of the model are described and it is demonstrated that the model does not predict underconfidence, that it is fully compatible with the overconfidence frequently found in calibration studies with cognitive tasks, and that it well represents experimental results from such studies. It is concluded that only a single model is needed for both types of task.     

Choosing what to choose from: Preference for inclusion over exclusion when constructing consideration sets from large choice sets

Decision making is a two‐stage process, consisting of, first, consideration set construction and then final choice. Decision makers can form a consideration set from a choice set using one of two strategies: including the options they wish to further consider or excluding those they do not wish to further consider. The authors propose that decision makers have a relative preference for an inclusion (vs. exclusion) strategy when choosing from large choice sets and that this preference is driven primarily by a lay belief that inclusion requires less effort than exclusion, particularly in large choice sets. Study 1 demonstrates that decision makers prefer using an inclusion (vs. exclusion) strategy when faced with large choice sets. Study 2 replicates the effect of choice set size on preference for consideration set construction strategy and demonstrates that the belief that exclusion is more effortful mediates the relative preference for inclusion in large choice sets. Studies 3 and 4 further support the importance of perceived effort, demonstrating a greater preference for inclusion in large choice sets when decision makers are primed to think about effort (vs. accuracy; Study 3) and when the choice set is perceived as requiring more effort because of more information being presented about each alternative (vs. more alternatives in the choice set; Study 4). Finally, Study 5 manipulates consideration set construction strategy, showing that using inclusion (vs. exclusion) in large choice sets leads to smaller consideration sets, greater confidence in the decision process, and a higher quality consideration set.     

Confidence and Uncertainty in Judgmental Forecasting: Differential Effects of Scenario Presentation

Multiple scenario analysis is used by many organizations for long-term planning, in part because the development of more than one scenario of the future is thought to prevent overconfidence in any one specific forecast. The present study attempts to clarify some of the questions concerning the cognitive effects of scenario presentation by distinguishing between confidence and subjective uncertainty. It examined the impact of different types of scenario information on confidence ratings, credible intervals, and features of forecast series. Subjects (N=186) read either one, two, or no scenario(s) describing potential changes in the level of a stimulus variable, then made forecasts of that variable over the next fifty years. Reading any type of scenario information increased confidence in forecasts; there was no reduction in confidence due to presentation of multiple conflicting scenarios, compared to presentation of only one scenario. Differences in number and in type of scenarios presented did, however, affect subjective uncertainty.     

On the evaluation of one-sided evidence

We examine predictions and judgments of confidence based on one-sided evidence. Some subjects saw arguments for only one side of a legal dispute while other subjects (called ‘jurors’) saw arguments for both sides. Subjects predicted the number of jurors who favored the plaintiff in each case. Subjects who saw only one side made predictions that were biased in favor of that side. Furthermore, they were more confident but generally less accurate than subjects who saw both sides. The results indicate that people do not compensate sufficiently for missing information even when it is painfully obvious that the information available to them is incomplete. A simple manipulation that required subjects to evaluate the relative strength of the opponent's side greatly reduced the tendency to underweigh missing evidence.