We favor formal models of heuristics rather than lists of loose dichotomies: a reply to Evans and Over

In their comment on Marewski et al. (good judgments do not require complex cognition, 2009) Evans and Over (heuristic thinking and human intelligence: a commentary on Marewski, Gaissmaier and Gigerenzer, 2009) conjectured that heuristics can often lead to biases and are not error free. This is a most surprising critique. The computational models of heuristics we have tested allow for quantitative predictions of how many errors a given heuristic will make, and we and others have measured the amount of error by analysis, computer simulation, and experiment. This is clear progress over simply giving heuristics labels, such as availability, that do not allow for quantitative comparisons of errors. Evans and Over argue that the reason people rely on heuristics is the accuracy-effort trade-off. However, the comparison between heuristics and more effortful strategies, such as multiple regression, has shown that there are many situations in which a heuristic is more accurate with less effort. Finally, we do not see how the fast and frugal heuristics program could benefit from a dual-process framework unless the dual-process framework is made more precise. Instead, the dual-process framework could benefit if its two “black boxes” (Type 1 and Type 2 processes) were substituted by computational models of both heuristics and other processes.     

Speed-Accuracy Response Models: Scoring Rules based on Response Time and Accuracy

Starting from an explicit scoring rule for time limit tasks incorporating both response time and accuracy, and a definite trade-off between speed and accuracy, a response model is derived. Since the scoring rule is interpreted as a sufficient statistic, the model belongs to the exponential family. The various marginal and conditional distributions for response accuracy and response time are derived, and it is shown how the model parameters can be estimated. The model for response accuracy is found to be the two-parameter logistic model. It is found that the time limit determines the item discrimination, and this effect is illustrated with the Amsterdam Chess Test II.     

Using Response Times and Response Accuracy to Measure Fluency Within Cognitive Diagnosis Models

Psychometrika - The recent “Every Student Succeed Act" encourages schools to use an innovative assessment to provide feedback about students’ mastery level of grade-level content...     

Log-Multiplicative Association Models as Item Response Models

Log-multiplicative association (LMA) models, which are special cases of log-linear models, have interpretations in terms of latent continuous variables. Two theoretical derivations of LMA models based on item response theory (IRT) arguments are presented. First, we show that Anderson and colleagues (Anderson &; Vermunt, 2000; Anderson &; Böckenholt, 2000; Anderson, 2002), who derived LMA models from statistical graphical models, made the equivalent assumptions as Holland (1990) when deriving models for the manifest probabilities of response patterns based on an IRT approach. We also present a second derivation of LMA models where item response functions are specified as functions of rest-scores. These various connections provide insights into the behavior of LMA models as item response models and point out philosophical issues with the use of LMA models as item response models. We show that even for short tests, LMA and standard IRT models yield very similar to nearly identical results when data arise from standard IRT models. Log-multiplicative association models can be used as item response models and do not require numerical integration for estimation.     

Truth and meaning redux

In this paper, we defend Davidson's program in truth-theoretical semantics against recent criticisms by Scott Soames. We argue that Soames has misunderstood Davidson's project, that in consequence his criticisms miss the mark, that appeal to meanings as entities in the alternative approach that Soames favors does no work, and that the approach is no advance over truth-theoretic semantics.     

Stochastic Dynamic Models of Response Time and Accuracy: A Foundational Primer

A large class of statistical decision models for performance in simple information processing tasks can be described by linear, first-order, stochastic differential equations (SDEs), whose solutions are diffusion processes. In such models, the first passage time for the diffusion process through a response criterion determines the time at which an observer makes a decision about the identity of a stimulus. Because the assumptions of many cognitive models lead to SDEs that are time inhomogeneous, classical methods for solving such first passage time problems are usually inapplicable. In contrast, recent integral equation methods often yield solutions to both the one-sided and the two-sided first passage time problems, even in the presence of time inhomogeneity. These methods, which are of particular relevance to the cognitive modeler, are described in detail, together with illustrative applications. Copyright 2000 Academic Press.     

Restricted Recalibration of Item Response Theory Models

Psychometrika - In item response theory (IRT), it is often necessary to perform restricted recalibration (RR) of the model: A set of (focal) parameters is estimated holding a set of (nuisance)...     

Hierarchical Bayes Models for Response Time Data

Human response time (RT) data are widely used in experimental psychology to evaluate theories of mental processing. Typically, the data constitute the times taken by a subject to react to a succession of stimuli under varying experimental conditions. Because of the sequential nature of the experiments there are trends (due to learning, fatigue, fluctuations in attentional state, etc.) and serial dependencies in the data. The data also exhibit extreme observations that can be attributed to lapses, intrusions from outside the experiment, and errors occurring during the experiment. Any adequate analysis should account for these features and quantify them accurately. Recognizing that Bayesian hierarchical models are an excellent modeling tool, we focus on the elaboration of a realistic likelihood for the data and on a careful assessment of the quality of fit that it provides. We judge quality of fit in terms of the predictive performance of the model. We demonstrate how simple Bayesian hierarchical models can be built for several RT sequences, differentiating between subject-specific and condition-specific effects.     

Multilevel Models for Examining Individual Differences in Within-Person Variation and Covariation Over Time

Heterogeneity of variance may be more than a statistical nuisance—it may be of direct interest as a result of individual differences. In studies of short-term fluctuation, individual differences may relate to the magnitude of within-person variation as well as to level of an outcome or its covariation with other processes. Although models for heterogeneous variances have been utilized in group contexts (i.e., dispersion models), they are not usually applied in examinations of intraindividual variation. This work illustrates how an extension of the multilevel model for heterogeneous variances can be used to examine individual differences in level, between- and within-person covariation, and magnitude of within-person variation of daily positive and negative mood in persons with dementia.     

Indeterminacy,identity and counterparts: Evans reconsidered

In this paper I argue that Gareth Evans’ famous proof of the impossibility of de re indeterminate identity fails on a counterpart-theoretic interpretation of the determinacy operators. I attempt to motivate a counterpart-theoretic reading of the determinacy operators and then show that, understood counterpart-theoretically, Evans’ argument is straightforwardly invalid.     

Nested Logit Models for Multiple-Choice Item Response Data

Nested logit item response models for multiple-choice data are presented. Relative to previous models, the new models are suggested to provide a better approximation to multiple-choice items where the application of a solution strategy precedes consideration of response options. In practice, the models also accommodate collapsibility across all distractor categories, making it easier to allow decisions about including distractor information to occur on an item-by-item or application-by-application basis without altering the statistical form of the correct response curves. Marginal maximum likelihood estimation algorithms for the models are presented along with simulation and real data analyses.     

Evans on transparency: a rationalist account

Gareth Evans famously observed that he can answer the question ‘Do you think there is going to be a third world war?’ by attending to “precisely the same outward phenomena as I would attend to if I were answering the question ‘Will there be a third world war?’” (The varieties of reference, Oxford University Press, Oxford, p 225, 1982). I argue that this observation follows from two independently plausible ideas in philosophy of mind. The first is about rationality and consciousness: it is that to be rational is in part to be required to believe that you are in a conscious state if you are in one, at least if various background conditions are met. The second is about consciousness and attention: it is that consciousness in a belief state consists in its subject engaging, to a sufficient extent, in a certain sort of world-directed attention. I also argue that this suggestion is superior to others that have been made in the literature regarding Evan’s observation.