Integrating Cognitive Process and Descriptive Models of Attitudes and Preferences

Discrete choice experiments—selecting the best and/or worst from a set of options—are increasingly used to provide more efficient and valid measurement of attitudes or preferences than conventional methods such as Likert scales. Discrete choice data have traditionally been analyzed with random utility models that have good measurement properties but provide limited insight into cognitive processes. We extend a well‐established cognitive model, which has successfully explained both choices and response times for simple decision tasks, to complex, multi‐attribute discrete choice data. The fits, and parameters, of the extended model for two sets of choice data (involving patient preferences for dermatology appointments, and consumer attitudes toward mobile phones) agree with those of standard choice models. The extended model also accounts for choice and response time data in a perceptual judgment task designed in a manner analogous to best–worst discrete choice experiments. We conclude that several research fields might benefit from discrete choice experiments, and that the particular accumulator‐based models of decision making used in response time research can also provide process‐level instantiations for random utility models.     

Measurement of Ability in Adaptive Learning and Assessment Systems when Learners Use On-Demand Hints

Adaptive learning and assessment systems support learners in acquiring knowledge and skills in a particular domain. The learners’ progress is monitored through them solving items matching their level and aiming at specific learning goals. Scaffolding and providing learners with hints are powerful tools in helping the learning process. One way of introducing hints is to make hint use the choice of the student. When the learner is certain of their response, they answer without hints, but if the learner is not certain or does not know how to approach the item they can request a hint. We develop measurement models for applications where such on-demand hints are available. Such models take into account that hint use may be informative of ability, but at the same time may be influenced by other individual characteristics. Two modeling strategies are considered: (1) The measurement model is based on a scoring rule for ability which includes both response accuracy and hint use. (2) The choice to use hints and response accuracy conditional on this choice are modeled jointly using Item Response Tree models. The properties of different models and their implications are discussed. An application to data from Duolingo, an adaptive language learning system, is presented. Here, the best model is the scoring-rule-based model with full credit for correct responses without hints, partial credit for correct responses with hints, and no credit for all incorrect responses. The second dimension in the model accounts for the individual differences in the tendency to use hints.     

Bayesian hierarchical response time modelling—A tutorial

Response time modelling is developing rapidly in the field of psychometrics, and its use is growing in psychology. In most applications, component models for response times are modelled jointly with component models for responses, thereby stabilizing estimation of item response theory model parameters and enabling research on a variety of novel substantive research questions. Bayesian estimation techniques facilitate estimation of response time models. Implementations of these models in standard statistical software, however, are still sparse. In this accessible tutorial, we discuss one of the most common response time models—the lognormal response time model—embedded in the hierarchical framework by van der Linden (2007). We provide detailed guidance on how to specify and estimate this model in a Bayesian hierarchical context. One of the strengths of the presented model is its flexibility, which makes it possible to adapt and extend the model according to researchers' needs and hypotheses on response behaviour. We illustrate this based on three recent model extensions: (a) application to non-cognitive data incorporating the distance-difficulty hypothesis, (b) modelling conditional dependencies between response times and responses, and (c) identifying differences in response behaviour via mixture modelling. This tutorial aims to provide a better understanding of the use and utility of response time models, showcases how these models can easily be adapted and extended, and contributes to a growing need for these models to answer novel substantive research questions in both non-cognitive and cognitive contexts.     

Semiparametric Thurstonian Models for Recurrent Choices: A Bayesian Analysis

We develop semiparametric Bayesian Thurstonian models for analyzing repeated choice decisions involving multinomial, multivariate binary or multivariate ordinal data. Our modeling framework has multiple components that together yield considerable flexibility in modeling preference utilities, cross-sectional heterogeneity and parameter-driven dynamics. Each component of our model is specified semiparametrically using Dirichlet process (DP) priors. The utility (latent variable) component of our model allows the alternative-specific utility errors to semiparametrically deviate from a normal distribution. This generates a robust alternative to popular Thurstonian specifications that are based on underlying normally distributed latent variables. Our second component focuses on flexibly modeling cross-sectional heterogeneity. The semiparametric specification allows the heterogeneity distribution to mimic either a finite mixture distribution or a continuous distribution such as the normal, whichever is supported by the data. Thus, special features such as multimodality can be readily incorporated without the need to overtly search for the best heterogeneity specification across a series of models. Finally, we allow for parameter-driven dynamics using a semiparametric state-space approach. This specification adds to the literature on robust Kalman filters. The resulting framework is very general and integrates divergent strands of the literatures on flexible choice models, Bayesian nonparametrics and robust time series specifications. Given this generality, we show how several existing Thurstonian models can be obtained as special forms of our model. We describe Markov chain Monte Carlo methods for the inference of model parameters, report results from two simulation studies and apply the model to consumer choice data from a frequently purchased product category. The results from our simulations and application highlight the benefits of using our semiparametric approach.     

The simplest complete model of choice response time: linear ballistic accumulation

We propose a linear ballistic accumulator (LBA) model of decision making and reaction time. The LBA is simpler than other models of choice response time, with independent accumulators that race towards a common response threshold. Activity in the accumulators increases in a linear and deterministic manner. The simplicity of the model allows complete analytic solutions for choices between any number of alternatives. These solutions (and freely-available computer code) make the model easy to apply to both binary and multiple choice situations. Using data from five previously published experiments, we demonstrate that the LBA model successfully accommodates empirical phenomena from binary and multiple choice tasks that have proven difficult for other theoretical accounts. Our results are encouraging in a field beset by the tradeoff between complexity and completeness.     

The nanoeconomics of concurrent choice behavior

We introduce to behavior analysis a way of analyzing choice behavior that exploits recent developments in nanoeconomics, financial economics, and econometrics. A response return, modeled on an economic return, is the log differenced count of responses allocated to each of two alternatives during a short time window, compared with that in the immediately preceding window. The response return is a new dependent variable which offers a novel and useful way of looking at operant behavior, especially at the molecular level of analysis. The response–return series is a near‐instantaneous measure of an organism's dynamic preferences for each of two alternatives. Analyzing such a series requires time‐analytic techniques, including Auto‐Regressive Conditional Heteroskedastic (ARCH) models. We illustrate these techniques by analyzing choices between combinations of arithmetic and exponential variable‐interval schedules with pigeon subjects. All response–return series were well‐fitted by one of three ARCH‐family models. The fitted models were differentially sensitive to transition versus steady‐state data samples. The particular insights that the ARCH analyses offer are improved understanding of the effects of the instantaneous effects of reinforcers and their absence, of how the distribution of reinforcers in time affects choice, and of the differences between choice in transition and at steady state.     

Acceleration model in the heterogeneous case of the general graded response model

A new model, called acceleration model, is proposed in the framework of the heterogenous case of the graded response model, based on processing functions defined for a finite or enumerable number of steps. The model is expected to be useful in cognitive assessment, as well as in more traditional areas of application of latent trait models. Criteria for evaluating models are proposed, and soundness and robustness of the acceleration model are discussed. Graded response models based on individual choice behavior are also discussed, and criticisms on model selection in terms of fitnesses of models to the data are also given.This research was supported by the Office of Naval Research (N00014-90-J-1456).     

The response dynamics of preferential choice

The ubiquity of psychological process models requires an increased degree of sophistication in the methods and metrics that we use to evaluate them. We contribute to this venture by capitalizing on recent work in cognitive science analyzing response dynamics, which shows that the bearing information processing dynamics have on intended action is also revealed in the motor system. This decidedly “embodied” view suggests that researchers are missing out on potential dependent variables with which to evaluate their models—those associated with the motor response that produces a choice. The current work develops a method for collecting and analyzing such data in the domain of decision making. We first validate this method using widely normed stimuli from the International Affective Picture System (Experiment 1), and demonstrate that curvature in response trajectories provides a metric of the competition between choice options. We next extend the method to risky decision making (Experiment 2) and develop predictions for three popular classes of process model. The data provided by response dynamics demonstrate that choices contrary to the maxim of risk seeking in losses and risk aversion in gains may be the product of at least one “online” preference reversal, and can thus begin to discriminate amongst the candidate models. Finally, we incorporate attentional data collected via eye-tracking (Experiment 3) to develop a formal computational model of joint information sampling and preference accumulation. In sum, we validate response dynamics for use in preferential choice tasks and demonstrate the unique conclusions afforded by response dynamics over and above traditional methods.     

Hierarchical diffusion models for two-choice response times

Two-choice response times are a common type of data, and much research has been devoted to the development of process models for such data. However, the practical application of these models is notoriously complicated, and flexible methods are largely nonexistent. We combine a popular model for choice response times-the Wiener diffusion process-with techniques from psychometrics in order to construct a hierarchical diffusion model. Chief among these techniques is the application of random effects, with which we allow for unexplained variability among participants, items, or other experimental units. These techniques lead to a modeling framework that is highly flexible and easy to work with. Among the many novel models this statistical framework provides are a multilevel diffusion model, regression diffusion models, and a large family of explanatory diffusion models. We provide examples and the necessary computer code.     

Alternative Approaches to Addressing Non-Normal Distributions in the Application of IRT Models to Personality Measures

It is generally assumed that the latent trait is normally distributed in the population when estimating logistic item response theory (IRT) model parameters. This assumption requires that the latent trait be fully continuous and the population homogenous (i.e., not a mixture). When this normality assumption is violated, models are misspecified, and item and person parameter estimates are inaccurate. When normality cannot be assumed, it might be appropriate to consider alternative modeling approaches: (a) a zero-inflated mixture, (b) a log-logistic, (c) a Ramsay curve, or (d) a heteroskedastic-skew model. The first 2 models were developed to address modeling problems associated with so-called quasi-continuous or unipolar constructs, which apply only to a subset of the population, or are meaningful at one end of the continuum only. The second 2 models were developed to address non-normal latent trait distributions and violations of homogeneity of error variance, respectively. To introduce these alternative IRT models and illustrate their strengths and weaknesses, we performed real data application comparing results to those from a graded response model. We review both statistical and theoretical challenges in applying these models and choosing among them. Future applications of these and other alternative models (e.g., unfolding, diffusion) are needed to advance understanding about model choice in particular situations.     

Diffusion versus linear ballistic accumulation: different models but the same conclusions about psychological processes?

Quantitative models for response time and accuracy are increasingly used as tools to draw conclusions about psychological processes. Here we investigate the extent to which these substantive conclusions depend on whether researchers use the Ratcliff diffusion model or the Linear Ballistic Accumulator model. Simulations show that the models agree on the effects of changes in the rate of information accumulation and changes in non-decision time, but that they disagree on the effects of changes in response caution. In fits to empirical data, however, the models tend to agree closely on the effects of an experimental manipulation of response caution. We discuss the implications of these conflicting results, concluding that real manipulations of caution map closely, but not perfectly to response caution in either model. Importantly, we conclude that inferences about psychological processes made from real data are unlikely to depend on the model that is used.     

Simple matrix methods for analyzing diffusion models of choice probability, choice response time, and simple response time

Diffusion processes (e.g., Wiener process, Ornstein-Uhlenbeck process) are powerful approaches to model human information processes in a variety of psychological tasks. Lack of mathematical tractability, however, has prevented broad applications of these models to empirical data. This tutorial explains step by step, using a matrix approach, how to construct these models, how to implement them on a computer, and how to calculate the predictions made by these models. In particular, we present models for binaries choices for unidimensional and multiattribute choice alternatives; for simple reaction time tasks; and for three alternatives choice problems.     

Eye Movements in Strategic Choice

In risky and other multiattribute choices, the process of choosing is well described by random walk or drift diffusion models in which evidence is accumulated over time to threshold. In strategic choices, level‐k and cognitive hierarchy models have been offered as accounts of the choice process, in which people simulate the choice processes of their opponents or partners. We recorded the eye movements in 2 × 2 symmetric games including dominance‐solvable games like prisoner's dilemma and asymmetric coordination games like stag hunt and hawk–dove. The evidence was most consistent with the accumulation of payoff differences over time: we found longer duration choices with more fixations when payoffs differences were more finely balanced, an emerging bias to gaze more at the payoffs for the action ultimately chosen, and that a simple count of transitions between payoffs—whether or not the comparison is strategically informative—was strongly associated with the final choice. The accumulator models do account for these strategic choice process measures, but the level‐k and cognitive hierarchy models do not. © 2015 The Authors. Journal of Behavioral Decision Making published by John Wiley & Sons Ltd.     

Estimating flexible distributions of ideal-points with external analysis of preferences

Ideal-points are widely used to model choices when preferences are single-peaked. Ideal-point choice models have been typically estimated at the individual-level, or have been based on the assumption that ideal-points are normally distributed over the population of choice makers. We propose two probabilistic ideal-point choice models for the external analysis of preferences that allow for more flexible multimodal distributions of ideal-points, thus acknowledging the existence of subpopulations with distinct preferences. The first model extends the ideal-point probit model for heterogeneous preferences to accommodate a mixture of multivariate normal distributions of ideal-points. The second model assumes that ideal-points are uniformly distributed within finite ranges of the attribute space, leading to a more simplistic formulation and a more flexible distribution. The two models are applied to simulated and actual choice data, and compared to the ideal-point probit model.This research was funded by the Dean's Fund for Faculty Research of the Owen Graduate School of Management, Vanderbilt University.     

Ignorance or integration: the cognitive processes underlying choice behavior

The fast‐and‐frugal heuristic framework assumes noncompensatory tools for human preferences (i.e., priority heuristic) and inferences (i.e., take the best heuristic). According to this framework, these heuristics predict choice behavior as well as model the cognitive processes underlying such behavior. The current paper presents two studies that juxtapose predictions derived from these two simple heuristics with alternative predictions derived from compensatory principles. Dependent measures that included reaction time, choice pattern, confidence level, and accuracy were better predicted by compensatory indices than by noncompensatory indices. These findings suggest that people do not rely on limited arguments only, but tend to integrate all acquired information into their choice processes. This tendency was replicated even when the experimental task facilitated the use of noncompensatory principles. We argue that the fast and frugal heuristics can predict the final outcome only under certain conditions, but even in these particular situations they are not applicable to the processes underlying choice behavior. An integrative model for choice behavior is proposed that better represents the data. Copyright © 2009 John Wiley & Sons, Ltd.     

Inferring latent class lexicographic rules from choice data

A lexicographic rule orders multi-attribute alternatives in the same way as a dictionary orders words. Although no utility function can represent lexicographic preference over continuous, real-valued attributes, a constrained linear model suffices for representing such preferences over discrete attributes. We present an algorithm for inferring lexicographic structures from choice data. The primary difficulty in using such data is that it is seldom possible to obtain sufficient information to estimate individual-level preference functions. Instead, one needs to pool the data across latent clusters of individuals. We propose a method that identifies latent clusters of subjects, and estimates a lexicographic rule for each cluster. We describe an application of the method using data collected by a manufacturer of television sets. We compare the predictions of the model with those obtained from a finite-mixture, multinomial-logit model.     

跨期决策中的维度差异偏好：眼动证据

在跨期决策研究领域,虽然基于维度的跨期模型得到了一些源自结果检验和过程检验的证据支持,但此类模型所假设的维度间差异比较的心理过程尚缺乏直接的过程证据。本研究通过两个眼动实验,系统考察了相关眼动指标对维度差异偏好的预测效应。结果发现,根据基于维度的权衡模型可有效拟合出个体在跨期决策中的维度差异偏好,并且反应时、眼跳注视熵和静态注视熵等指标均与维度差异偏好负相关,而基于维度的注意分配与维度差异判断正相关。这些研究发现支持了本研究所提出的跨期眼动模型的相关假设,证实了维度差异偏好与跨期决策的认知加工过程之间的联系,为基于维度的跨期模型提供了更直接的过程证据,并为今后跨期决策的眼动模型发展指明了新方向。     

A Signal Detection Model for Multiple-Choice Exams

A model for multiple-choice exams is developed from a signal-detection perspective. A correct alternative in a multiple-choice exam can be viewed as being a signal embedded in noise (incorrect alternatives). Examinees are assumed to have perceptions of the plausibility of each alternative, and the decision process is to choose the most plausible alternative. It is also assumed that each examinee either knows or does not know each item. These assumptions together lead to a signal detection choice model for multiple-choice exams. The model can be viewed, statistically, as a mixture extension, with random mixing, of the traditional choice model, or similarly, as a grade-of-membership extension. A version of the model with extreme value distributions is developed, in which case the model simplifies to a mixture multinomial logit model with random mixing. The approach is shown to offer measures of item discrimination and difficulty, along with information about the relative plausibility of each of the alternatives. The model, parameters, and measures derived from the parameters are compared to those obtained with several commonly used item response theory models. An application of the model to an educational data set is presented.     

Multivariate thurstonian models

The recent development of probabilistic Thurstonian choice models for analyzing preference data has been motivated by the need to describe both inter- and intra-individual difference, the multidimensional nature of choice objects, and the effects of similarity and comparability among choice objects. A common feature of these models is that they focus on asingle preference judgment. It is customary, however, to ask subjects not only for an overall preference judgment but also for additional paired comparison responses regarding specific attributes. This paper proposes a generalization of Thurstonian probabilistic choice models for analyzing both multiple preference responses and their relationships. The approach is illustrated by modeling data from two multivariate preference experiments.I am grateful to James, Austin, Ingo Böckenholt, and anonymous referees for helpful comments on this research. This paper is based on research presented at the Meeting of the Society for the Multivariate Analysis in the Behavioral Sciences, Groningen, The Netherlands, December 1988.