Response error and processing biases in confidence judgment

Previous studies showed that random error can explain overconfidence effects typically observed in the literature. One of these studies concluded that, after accounting for random error effects in the data, there is little support for cognitive‐processing biases in confidence elicitation. In this paper, we investigate more closely the random error explanation for overconfidence. We generated data from four models of confidence and then estimated the magnitude of random error in the data. Our results show that, in addition to the true magnitude of random error specified in the simulations, the error estimates are influenced by important cognitive‐processing biases in the confidence elicitation process. We found that random error in the response process can account for the degree of overconfidence found in calibration studies, even when that overconfidence is actually caused by other factors. Thus, the error models say little about whether cognitive biases are present in the confidence elicitation process. Copyright © 2008 John Wiley & Sons, Ltd.     

Robust interactive decision-analysis (RID): Behavioral results and implications

We have proposed a novel interactive procedure for performing decision analysis, called Robust Interactive Decision Analysis (RID), which permits a decision maker (DM) to voluntarily and interactively express strong (viz, sure) binary preferences for actions, partial decision functions, and full decision functions, and only imprecise probability and utility function assessments. These serve as INPUTS TO operators to prune the state probability space and decision space until an optimal choice strategy is obtained. The viability of the RID approach depends on a DM's ability to provide such information consistently and meaningfully. On a limited scale we experimentally investigate the behavioral implications of the RID method in order to ascertain its potential operational feasibility and viability. More specifically, we examine whether a DM can (1) express strong preferences between pairs of vectors of unconditional and conditional payoffs or utilities consistently; (2) provide imprecise (ordinal and interval) state probabilities that are individually as well as mutually consistent with the state probabilities imputed from the expressed strong preferences. The results show that a DM can provide strong individually and mutually consistent preference and ordinal probability information. Moreover, most individuals also appear to be able to provide interval probabilities that are individually and mutually consistent with their strong preference inputs. However, the several violations observed, our small sample size, and the limited scope of our investigation suggest that further experimentation is needed to determine whether and/or how such inputs should be elicited. Overall, the results indicate that the RID method is behaviorally viable.     

The asymptotic posterior normality of the latent trait in an IRT model

It has long been part of the item response theory (IRT) folklore that under the usual empirical Bayes unidimensional IRT modeling approach, the posterior distribution of examinee ability given test response is approximately normal for a long test. Under very general and nonrestrictive nonparametric assumptions, we make this claim rigorous for a broad class of latent models.This research was partially supported by Office of Naval Research Cognitive and Neural Sciences Grant N0014-J-90-1940, 442-1548, National Science Foundation Mathematics Grant NSF-DMS-91-01436, and the National Center for Supercomputing Applications. We wish to thank Kumar Joag-dev and Zhiliang Ying for enlightening suggestions concerning the proof of the basic result.The authors wish to thank Kumar Joag-Dev, Brian Junker, Bert Green, Paul Holland, Robert Mislevy, and especially Zhiliang Ying for their useful comments and discussions.     

Themes in Therapist Development During the First Three Months of Clinical Experience

The results of a qualitative research study designed to better understand the developmental processes active in beginning therapists during the first three months of clinical contact is reported. Thirteen beginning therapists were asked to complete a monthly log describing experiences impacting their clinical work and themselves as therapists during their first three months of client contact. Data analysis revealed that the primary developmental theme active during this period of time is the development of therapist confidence. Two additional themes, the development of an internal gauge on which to evaluate current experiences and the development of boundaries around the self as a professional, were also found to be active during this time. These additional themes support the development of therapist confidence. Implications for training and research are identified.     

Application of the bootstrap methods in factor analysis

A Monte Carlo experiment is conducted to investigate the performance of the bootstrap methods in normal theory maximum likelihood factor analysis both when the distributional assumption is satisfied and unsatisfied. The parameters and their functions of interest include unrotated loadings, analytically rotated loadings, and unique variances. The results reveal that (a) bootstrap bias estimation performs sometimes poorly for factor loadings and nonstandardized unique variances; (b) bootstrap variance estimation performs well even when the distributional assumption is violated; (c) bootstrap confidence intervals based on the Studentized statistics are recommended; (d) if structural hypothesis about the population covariance matrix is taken into account then the bootstrap distribution of the normal theory likelihood ratio test statistic is close to the corresponding sampling distribution with slightly heavier right tail.This study was carried out in part under the ISM cooperative research program (91-ISM · CRP-85, 92-ISM · CRP-102). The authors would like to thank the editor and three reviewers for their helpful comments and suggestions which improved the quality of this paper considerably.     

Calibration of sensory and cognitive judgments: A single model for both

In a recent issue of this journal, Winman and Juslin (34 , 135–148, 1993) present a model of the calibration of subjective probability judgments for sensory discrimination tasks. They claim that the model predicts a pervasive underconfidence bias observed in such tasks, and present evidence from a training experiment that they interpret as supporting the notion that different models are needed to describe judgment of confidence in sensory and in cognitive tasks. The model is actually part of the more comprehensive decision variable partition model of subjective probability calibration that was originally proposed in Ferrell and McGoey (Organizational Behavior and Human Performance, 26 , 32–53, 1980). The characteristics of the model are described and it is demonstrated that the model does not predict underconfidence, that it is fully compatible with the overconfidence frequently found in calibration studies with cognitive tasks, and that it well represents experimental results from such studies. It is concluded that only a single model is needed for both types of task.     

Psychological determinants of decision attitude

Decision attitude — an analog of risk attitude — is the propensity to make (or avoid making) a decision: in decision aversion, a person finds it more desirable to receive through fiat the better of two options than to have a choice between them; in decision seeking, the choice is more desirable, even though it can lead to nothing better than the best option. Both decision aversion and decision seeking were found in hypothetical scenarios. Experimental manipulations and subjects' justifications point to anticipated regret, fear of blame for poor outcomes, and desire for equitable distributions as sources of decision aversion. One source of decision seeking (for self) and decision aversion (when deciding for others) appears to be the desire for the self-determination of the affected parties. We consider the implications of our results for personal choice and public policy decisions.     

Decisions,actions, and the appropriateness of confidence in knowledge

This research examines the appropriateness of confidence (i.e., subjective probability judgments) in knowledge associated with decisions and actions of social importance. One hundred and thirty seven participants completed a 50 item questionnaire assessing their knowledge of the two leading candidates in the 1988 presidential election in the U.S.A. Ninety one of the respondents completed the questionnaire one week prior to the election, whereas the other 46 completed the questionnaire on election day shortly after voting. After each item in the questionnaire, all respondents indicated whether or not the item content represented a reason why they voted (or intended to vote) for or against the candidate to whom the item referred. Within-person results indicated that, in comparison to items that were not cited as reasons for voting intentions or voting behavior, items endorsed as reasons were characterized by better accuracy and resolution, but worse overconfidence. Between groups, decision makers were significantly more accurate and better calibrated than those who had not made a decision between the candidates. Implications of inappropriate confidence on decision making effectiveness are discussed.     

Do medical decision analyses' largest gains grow from the smallest trees?

Some decision analysts recommend reserving their models for careful, complete study of complex problems. Others believe simple problem analyses provide the greatest gains. To clarify these issues, I tested: (1) whether simple published analyses compared to complex ones, produced results as non-intuitive; and (2) whether non-intuitiveness represented analytic gain (or equivalently, intuitive loss). Very limited data also examined (3) the gain from adding model intricacy for a given problem vs. selecting more complex problems for analysis. To assess two non-intuitiveness measures, 75 subjects (33 physicians and 42) non-physicians stated their intuitive preferences in 40 (22 published and 18 unpublished) medical dilemmas. For both physicians and non-physicians, simpler models (those with lees than 30 terminal branches in the decision tree) had about one third the non-intuitiveness of more complex ones. Three tests also supported the premise that analyses outperform intuition, therefore that on average the formers' non-intuitiveness reflects less analytic gain. In addition, for different models of the same problem, tree complexity did not correlate with gain. Thus, simpler trees may not generally gain less because they inadequately describe problems. Instead, simpler analyses may represent simpler problems, with more similar intuitive and formal solutions. If so, these findings may help us avoid some costly but unnecessary simple analyses.     

Decision Support System and Multi‐Criteria Decision Aid: A State of the Art and Perspectives

The aim of this paper is to highlight the role of the Decision Support System within the field of multi‐criteria decision aid (MCDA). The MCDA tools have been incorporated into systems to create Multi‐Criteria Decision Support Systems (MCDSSs). In our literature review, we noticed that more than 100 papers have been written over a 20‐year period in which MCDSS was used as a decision‐making tool. The present paper describes some real applications of MCDSS in different fields, harmoniously combined with decision‐making methods such as analytic hierarchy process, Utility Additive, and Goal Programming. The present study proposes an integrative MCDSS evaluation through guidance on the tools most useful for a specific user with a particular decision problem. Copyright © 2014 John Wiley & Sons, Ltd.