Psychometric Modeling of response speed and accuracy with mixed and conditional regression

Human performance in cognitive testing and experimental psychology is expressed in terms of response speed and accuracy. Data analysis is often limited to either speed or accuracy, and/or to crude summary measures like mean response time (RT) or the percentage correct responses. This paper proposes the use of mixed regression for the psychometric modeling of response speed and accuracy in testing and experiments. Mixed logistic regression of response accuracy extends logistic item response theory modeling to multidimensional models with covariates and interactions. Mixed linear regression of response time extends mixed ANOVA to unbalanced designs with covariates and heterogeneity of variance. Related to mixed regression is conditional regression, which requires no normality assumption, but is limited to unidimensional models. Mixed and conditional methods are both applied to an experimental study of mental rotation. Univariate and bivariate analyzes show how within-subject correlation between response and RT can be distinguished from between-subject correlation, and how latent traits can be detected, given careful item design or content analysis. It is concluded that both response and RT must be recorded in cognitive testing, and that mixed regression is a versatile method for analyzing test data.I am grateful to Rogier Donders for putting his data at my disposal.     

Likelihood inference on the underlying structure of IRT models

The assumptions underlying item response theory (IRT) models may be expressed as a set of equality and inequality constraints on the parameters of a latent class model. It is well known that the same assumptions imply that the parameters of the manifest distribution have to satisfy a more complicated set of inequality constraints which, however, are necessary but not sufficient. In this paper, we describe how the theory for likelihood-based inference under equality and inequality constraints may be used to test the underlying assumptions of IRT models. It turns out that the analysis based directly on the latent structure is simpler and more flexible. In particular, we indicate how several interesting extensions of the Rasch model may be obtained by partial relaxation of the basic constraints. An application to a data set provided by Educational Testing Service is used to illustrate the approach.We thank Dr. Gorman and Dr. Rogers of the Educational Testing Service for providing the data analyzed in Section 4. We also thank three reviewers for comments and suggestions.This revised article was published online in August 2005 with the PDF paginated correctly.     

Modeling behavior in a clinically diagnostic sequential risk-taking task

This article models the cognitive processes underlying learning and sequential choice in a risk-taking task for the purposes of understanding how they occur in this moderately complex environment and how behavior in it relates to self-reported real-world risk taking. The best stochastic model assumes that participants incorrectly treat outcome probabilities as stationary, update probabilities in a Bayesian fashion, evaluate choice policies prior to rather than during responding, and maintain constant response sensitivity. The model parameter associated with subjective value of gains correlates well with external risk taking. Both the overall approach, which can be expanded as the basic paradigm is varied, and the specific results provide direction for theories of risky choice and for understanding risk taking as a public health problem.     

Fitting Psychometric Models with Methods Based on Automatic Differentiation

Quantitative psychology is concerned with the development and application of mathematical models in the behavioral sciences. Over time, models have become more complex, a consequence of the increasing complexity of research designs and experimental data, which is also a consequence of the utility of mathematical models in the science. As models have become more elaborate, the problems of estimating them have become increasingly challenging. This paper gives an introduction to a computing tool called automatic differentiation that is useful in calculating derivatives needed to estimate a model. As its name implies, automatic differentiation works in a routine way to produce derivatives accurately and quickly. Because so many features of model development require derivatives, the method has considerable potential in psychometric work. This paper reviews several examples to demonstrate how the methodology can be applied. From the Presidential Address delivered at the 70th Annual Meeting of the Psychometric Society, Tilburg University, The Netherlands, July 5–8, 2005.     

Variability of the MAX and MIN Statistic: A Theory of the Quantile Spread as a Function of Sample Size

The maximum and minimum of a sample from a probability distribution are extremely important random variables in many areas of psychological theory, methodology, and statistics. For instance, the behavior of the mean of the maximum or minimum processing time, as a function of the number of component random processing times (n), has been studied extensively in an effort to identify the underlying processing architecture (e.g., Townsend & Ashby, 1983; Colonius & Vorberg, 1994). Little is known concerning how measures of variability of the maximum or minimum change with n. Here, a new measure of random variability, the quantile spread, is introduced, which possesses sufficient strength to define distributional orderings and derive a number of results concerning variability of the maximum and the minimum statistics. The quantile spread ordering may be useful in many venues. Several interesting open problems are pointed out. This work was supported by an NIH Grant R01 MH57717 to the first author. Some of the collaboration took place during the year 2000 while J.T. Townsend was a Fellow at the Hanse Institute for Advanced Study (HWK), sponsored by H. Colonius at Oldenburg University.     

Dispersion-weighted kappa: An integrative framework for metric and nominal scale agreement coefficients

The rater agreement literature is complicated by the fact that it must accommodate at east two different properties of rating data: the number of raters (two versus more than two) and the rating scale level (nominal versus metric). While kappa statistics are most widely used for nominal scales, intraclass correlation coefficients have been preferred for metric scales. In this paper, we suggest a dispersion-weighted kappa framework for multiple raters that integrates some important agreement statistics by using familiar dispersion indices as weights for expressing disagreement. These weights are applied to ratings identifying cells in the traditional inter-judge contingency table. Novel agreement statistics can be obtained by applying less familiar indices of dispersion in the same wayThis revised article was published online in August 2005 with the PDF paginated correctly.     

How General is Generalized Scientific Essentialism?

I look at a recent argument offered in defense of a doctrine which I will call generalized scientific essentialism. This is the doctrine according to which, not only are some facts about substance composition metaphysically necessary, but, in addition, some facts about substance behavior are metaphysically necessary. More specifically, so goes the argument, not only is water necessarily composed of H₂O and salt is necessarily composed of NaCl, but, in addition, salt necessarily dissolves in water. If this argument is sound, and if the statement that necessarily salt dissolves in water is a statement of a law of nature, then one conclusion of the argument is that there is at least one metaphysically necessary law of nature. My paper examines the extent to which this kind of argument could be generalized to provide a case for a full-blown scientific essentialism: the doctrine according to which all of the laws of nature are necessary. Or, in terms of dispositions, it is the doctrine according to which natural kinds have all of their powers, capacities and propensities as a matter of necessity.