Reliability of test scores in nonparametric item response theory

Three methods for estimating reliability are studied within the context of nonparametric item response theory. Two were proposed originally by Mokken (1971) and a third is developed in this paper. Using a Monte Carlo strategy, these three estimation methods are compared with four classical lower bounds to reliability. Finally, recommendations are given concerning the use of these estimation methods.The authors are grateful for constructive comments from the reviewers and from Charles Lewis.     

Testing whether independent treatment groups have equal medians

The paper suggests new methods for comparing the medians corresponding to independent treatment groups. The procedures are based on the Harrell-Davis estimator in conjunction with a slight modification and extension of the bootstrap calibration technique suggested by Loh. Alternatives to the Harrell-Davis estimator are briefly discussed. For the special case of two treatment groups, the proposed procedure always had more power than the Fligner-Rust solution, as well as the procedure examined by Wilcox and Charlin. Included is an illustration, using real data, that comparing medians, rather than means, can yield a substantially different conclusion as to whether two distributions differ in terms of some measure of central location.     

Nonparametric IRT: Testing the bi-isotonicity of isotonic probabilistic models (ISOP)

Nonparametric tests for testing the validity of polytomous ISOP-models (unidimensional ordinal probabilistic polytomous IRT-models) are presented. Since the ISOP-model is a very general nonparametric unidimensional rating scale model the test statistics apply to a great multitude of latent trait models. A test for the comonotonicity of item sets of two or more items is suggested. Procedures for testing the comonotonicity of two item sets and for item selection are developed. The tests are based on Goodman-Kruskal's gamma index of ordinal association and are generalizations thereof. It is an essential advantage of polytomous ISOP-models within probabilistic IRT-models that the tests of validity of the model can be performed before and without the model being fitted to the data. The new test statistics have the further advantage that no prior order of items or subjects needs to be known.     

True Overconfidence in Interval Estimates: Evidence Based on a New Measure of Miscalibration

Overconfidence is often regarded as one of the most prevalent judgment biases. Several studies show that overconfidence can lead to suboptimal decisions of investors, managers, or politicians. Recent research, however, questions whether overconfidence should be regarded as a bias and shows that standard “overconfidence” findings can easily be explained by different degrees of knowledge of agents plus a random error in predictions. We contribute to the current literature and ongoing research by extensively analyzing interval estimates for knowledge questions, for real financial time series, and for artificially generated charts. We thereby suggest a new method to measure overconfidence in interval estimates, which is based on the implied probability mass behind a stated prediction interval. We document overconfidence patterns, which are difficult to reconcile with rationality of agents and which cannot be explained by differences in knowledge as differences in knowledge do not exist in our task. Furthermore, we show that overconfidence measures are reliable in the sense that there exist stable individual differences in the degree of overconfidence in interval estimates, thereby testing an important assumption of behavioral economics and behavioral finance models: stable individual differences in the degree of overconfidence across people. We do this in a “field experiment,” for different levels of expertise of subjects (students on the one hand and professional traders and investment bankers on the other hand), over time, by using different miscalibration metrics, and for tasks that avoid common weaknesses such as a non‐representative selection of trick questions. Copyright © 2012 John Wiley & Sons, Ltd.     

Considering too few alternatives: The mental model theory of extensional reasoning

When solving a simple probabilistic problem, people tend to build an incomplete mental representation. We observe this pattern in responses to probabilistic problems over a set of premises using the conjunction, disjunction, and conditional propositional connectives. The mental model theory of extensional reasoning explains this bias towards underestimating the number of possibilities: In reckoning with different interpretations of the premises (logical rules, mental model theoretical, and, specific to conditional premises, conjunction and biconditional interpretation) the mental model theory accounts for the majority of observations. Different interpretations of a premise result in a build-up of mental models that are often incomplete. These mental models are processed using either an extensional strategy relying on proportions amongst models, or a conflict monitoring strategy. The consequence of considering too few possibilities is an erroneous probability estimate akin to that faced by decision makers who fail to generate and consider all alternatives, a characteristic of bounded rationality. We compare our results to the results published by Johnson-Laird, Legrenzi, Girotto, Legrenzi, and Caverni [Johnson-Laird, P., Legrenzi, P., Girotto, V., Legrenzi, M., &; Caverni, J. (1999 Johnson-Laird, P., Legrenzi, P., Girotto, V., Legrenzi, M., &; Caverni, J. (1999). Naive probability: A mental model theory of extensional reasoning. Psychological Review, 106, 62–88. doi:10.1037/0033-295X.106.1.62[Crossref], [PubMed], [Web of Science ®] , [Google Scholar]). Naive probability: A mental model theory of extensional reasoning. Psychological Review, 106 Johnson-Laird, P., Legrenzi, P., Girotto, V., Legrenzi, M., &; Caverni, J. (1999). Naive probability: A mental model theory of extensional reasoning. Psychological Review, 106, 62–88. doi:10.1037/0033-295X.106.1.62[Crossref], [PubMed], [Web of Science ®] , [Google Scholar], 62 Johnson-Laird, P., Legrenzi, P., Girotto, V., Legrenzi, M., &; Caverni, J. (1999). Naive probability: A mental model theory of extensional reasoning. Psychological Review, 106, 62–88. doi:10.1037/0033-295X.106.1.62[Crossref], [PubMed], [Web of Science ®] , [Google Scholar]–88 Johnson-Laird, P., Legrenzi, P., Girotto, V., Legrenzi, M., &; Caverni, J. (1999). Naive probability: A mental model theory of extensional reasoning. Psychological Review, 106, 62–88. doi:10.1037/0033-295X.106.1.62[Crossref], [PubMed], [Web of Science ®] , [Google Scholar]. doi:10 Johnson-Laird, P., Legrenzi, P., Girotto, V., Legrenzi, M., &; Caverni, J. (1999). Naive probability: A mental model theory of extensional reasoning. Psychological Review, 106, 62–88. doi:10.1037/0033-295X.106.1.62[Crossref], [PubMed], [Web of Science ®] , [Google Scholar].1037 Johnson-Laird, P., Legrenzi, P., Girotto, V., Legrenzi, M., &; Caverni, J. (1999). Naive probability: A mental model theory of extensional reasoning. Psychological Review, 106, 62–88. doi:10.1037/0033-295X.106.1.62[Crossref], [PubMed], [Web of Science ®] , [Google Scholar]/0033 Johnson-Laird, P., Legrenzi, P., Girotto, V., Legrenzi, M., &; Caverni, J. (1999). Naive probability: A mental model theory of extensional reasoning. Psychological Review, 106, 62–88. doi:10.1037/0033-295X.106.1.62[Crossref], [PubMed], [Web of Science ®] , [Google Scholar]-295X Johnson-Laird, P., Legrenzi, P., Girotto, V., Legrenzi, M., &; Caverni, J. (1999). Naive probability: A mental model theory of extensional reasoning. Psychological Review, 106, 62–88. doi:10.1037/0033-295X.106.1.62[Crossref], [PubMed], [Web of Science ®] , [Google Scholar].106 Johnson-Laird, P., Legrenzi, P., Girotto, V., Legrenzi, M., &; Caverni, J. (1999). Naive probability: A mental model theory of extensional reasoning. Psychological Review, 106, 62–88. doi:10.1037/0033-295X.106.1.62[Crossref], [PubMed], [Web of Science ®] , [Google Scholar].1 Johnson-Laird, P., Legrenzi, P., Girotto, V., Legrenzi, M., &; Caverni, J. (1999). Naive probability: A mental model theory of extensional reasoning. Psychological Review, 106, 62–88. doi:10.1037/0033-295X.106.1.62[Crossref], [PubMed], [Web of Science ®] , [Google Scholar].62 Johnson-Laird, P., Legrenzi, P., Girotto, V., Legrenzi, M., &; Caverni, J. (1999). Naive probability: A mental model theory of extensional reasoning. Psychological Review, 106, 62–88. doi:10.1037/0033-295X.106.1.62[Crossref], [PubMed], [Web of Science ®] , [Google Scholar]], and we observe lower performance levels than those in the original article.     

Evidence for a young adult face bias in accuracy and consensus of age estimates

Adults' face processing may be specialized for the dimensions of young adult faces. For example, young and older adults exhibit increased accuracy in normality judgments and greater agreement in attractiveness ratings for young versus older adult faces. The present study was designed to examine whether there is a similar young adult face bias in facial age estimates. In Experiment 1, we created a face age continuum by morphing an averaged young adult face with an averaged older adult face in 5% increments, for a total of 21 faces ranging from 0 to 100% old. Young and older adults estimated facial age for three stimulus age categories [young (morphs 0–30%), middle‐aged (morphs 35–65%), and older adult (morphs 70–100%)]. Both age groups showed the least differentiation in age estimates for young adult faces, despite showing greater consensus across participants in estimates for young faces. In Experiment 2, young and older adults made age estimates for individual young and older adult identities. Both age groups were more accurate and showed greater consensus in age estimates for young faces. Collectively, these results provide evidence for a bias in processing young adult faces beyond that which is often observed in recognition and normality/attractiveness judgment tasks.     

Scarce Foods are Perceived as Having More Calories

People vary dramatically in their calorie estimates of food depending on the information available to them. Prior research has focused on information that is normatively relevant to the number of calories a food contains (e.g., fat content, serving size). The current research examines whether information that is normatively irrelevant to the number of calories a food contains—such as its availability—might also shape people's calorie estimates. Three studies found that a food perceived as limited in availability leads people to estimate the food to contain more calories. Serial mediation analyses revealed that this effect occurs because scarce food is seen as more valuable and expensive, which subsequently induces feelings of resource deprivation. This sense of resource deprivation, in turn, leads to motivated perception, whereby higher calorie estimates are the result of people wanting to acquire more resources. The findings provide insight into the psychology of scarcity and underscore the importance of understanding how contextual factors shape people's calorie estimates and the psychological mechanisms that drive them.     

用不同外部表征方式集中呈现信息条件下的因果力判断

在分别用文字陈述、表格和图形三种外部表征方式集中呈现因果信息的条件下,用直接估计因果力大小的实验范式考察单一因果关系因果力估计的特点,检验概率对比模型,效力PC理论和pCI规则。让287名大学生被试估计不同化学药物影响动物基因变异的能力。结果发现,对单一因果关系因果力估计具有以下4个特点：⑴不对称性：在预防原因条件下的因果力估计较多符合效力PC理论,而在产生原因条件下的因果力估计一般符合概率对比模型;⑵文字陈述、表格和图形三种信息外部表征方式,不影响产生原因条件下的因果力估计,但影响预防原因条件下的因果力估计。在预防原因条件下,与文字陈述和表格表征相比,图形表征会促使更多被试按效力PC理论来做因果力估计;⑶没有被试使用pCI规则;⑷被试估计因果力所使用的规则存在明显的个体差异。     

Intolerance of uncertainty and perceived threat

The present study examined the relations between dimensions of intolerance of uncertainty (i.e., desire for predictability and uncertainty paralysis) and perceptions of threat (i.e., perceptions of the probabilities and costs of future undesirable outcomes) in a sample of 239 college students. Uncertainty paralysis was positively associated with both perceived probabilities and perceived costs for negative outcomes. Desire for predictability was positively associated with perceived costs for negative outcomes but was not associated with perceived probabilities for negative outcomes. When probability estimates for low base rate outcomes were examined separately, desire for predictability was negatively associated with perceived probabilities. Finally, perceived threat partially mediated the relations between dimensions of intolerance of uncertainty and worry. The results of the present study suggest mechanisms through which dimensions of intolerance of uncertainty might lead to excessive worry.     

Empirical underidentification in estimating random utility models: The role of choice sets and standardizations

A standard approach to distinguishing people’s risk preferences is to estimate a random utility model using a power utility function to characterize the preferences and a logit function to capture choice consistency. We demonstrate that with often-used choice situations, this model suffers from empirical underidentification, meaning that parameters cannot be estimated precisely. With simulations of estimation accuracy and Kullback–Leibler divergence measures we examined factors that potentially mitigate this problem. First, using a choice set that guarantees a switch in the utility order between two risky gambles in the range of plausible values leads to higher estimation accuracy than randomly created choice sets or the purpose-built choice sets common in the literature. Second, parameter estimates are regularly correlated, which contributes to empirical underidentification. Examining standardizations of the utility scale, we show that they mitigate this correlation and additionally improve the estimation accuracy for choice consistency. Yet, they can have detrimental effects on the estimation accuracy of risk preference. Finally, we also show how repeated versus distinct choice sets and an increase in observations affect estimation accuracy. Together, these results should help researchers make informed design choices to estimate parameters in the random utility model more precisely.