Item Response Modeling of Paired Comparison and Ranking Data

The comparative format used in ranking and paired comparisons tasks can significantly reduce the impact of uniform response biases typically associated with rating scales. Thurstone's (1927, 1931) model provides a powerful framework for modeling comparative data such as paired comparisons and rankings. Although Thurstonian models are generally presented as scaling models, that is, stimuli-centered models, they can also be used as person-centered models. In this article, we discuss how Thurstone's model for comparative data can be formulated as item response theory models so that respondents' scores on underlying dimensions can be estimated. Item parameters and latent trait scores can be readily estimated using a widely used statistical modeling program. Simulation studies show that item characteristic curves can be accurately estimated with as few as 200 observations and that latent trait scores can be recovered to a high precision. Empirical examples are given to illustrate how the model may be applied in practice and to recommend guidelines for designing ranking and paired comparisons tasks in the future.     

Analyzing the random control procedure: Effects of paired and unpaired CSs and USs on autoshaping the chick's key peck with heat reinforcement

A series of five experiments using a total of 264 subjects investigated the effects of paired and unpaired key light (CS) and heat (US) stimuli on autoshaping the chick's key peck. Experiment 1 established that paired presentations of CS and US promoted a more rapid rise in key pecking than did randomly presented CSs and USs and that the specific sequence of stimuli under the random control procedure affected key pecking performance. Experiment 2 used a trace conditioning procedure to determine the role of the CS-US interval on autoshaping and to define empirically unpaired CSs and USs. Key pecking declined as the trace delay interval was increased from 0 to 25 sec; at 25 sec, no conditioning of key pecking occurred. Experiments 3–5 assessed the effects on autoshaped key pecking of (a) number of daily CS-US pairings, (b) added unpaired CS presentations, and (c) added unpaired US presentations, since paired and random control schedules differed in all of these respects. Reduction in the number of CS-US pairings slowed the acquisition of key pecking as did the concurrent addition of nonreinforced CSs and unsignaled USs. These results support theories of association formation that stress the effects of both paired and unpaired CSs and USs.     

Maximum likelihood estimation for Luce's choice model

A terminology for general choice models based on the choice axiom is given. It applies to all kinds of choice experiments, such as confusion choice experiments, paired comparisons, triadic comparisons, directional rankings, scores on binary test items, and others. Maximum likelihood estimation for such general choice models is considered. Conditions for the uniqueness of maximum likelihood estimates are given, and it is shown that the estimates can be derived by iterative proportional fitting. This offers the opportunity of a general test of the choice axiom for all kinds of choice experiments using the likelihood ratio. The estimation and testing procedure is applied to data from a form recognition experiment, reported by W. A. Wagenaar (Nederlands Tijdschrift voor de Psychologie, 1968, 23, 96–108).     

Structural equation modeling of paired-comparison and ranking data

L. L. Thurstone's (1927) model provides a powerful framework for modeling individual differences in choice behavior. An overview of Thurstonian models for comparative data is provided, including the classical Case V and Case III models as well as more general choice models with unrestricted and factor-analytic covariance structures. A flow chart summarizes the model selection process. The authors show how to embed these models within a more familiar structural equation modeling (SEM) framework. The different special cases of Thurstone's model can be estimated with a popular SEM statistical package, including factor analysis models for paired comparisons and rankings. Only minor modifications are needed to accommodate both types of data. As a result, complex models for comparative judgments can be both estimated and tested efficiently.     

A memory-based account of automatic numerosity processing

We investigated the mechanisms responsible for the automatic processing of the numerosities represented by digits in the size congruity effect (Henik & Tzelgov, 1982). The algorithmic model assumes that relational comparisons of digit magnitudes (e.g., larger than {8,2}) create this effect. If so, congruity effects ought to require two digits. Memory-based models assume that associations between individual digits and the attributes "small" and "large" create this effect. If so, congruity effects ought only to require one digit. Contrary to the algorithmic model and consistent with memory-based models, congruity effects were just as large when subjects judged the relative physical sizes of small digits paired with letters as when they judged the relative physical sizes of two digits. This finding suggests that size congruity effects can be produced without comparison algorithms.     

Hierarchical modeling of paired comparison data

The method of paired comparisons belongs to a small group of techniques that provide explicit information about the consistency of individual and aggregated choices. This article investigates the link between the individual- and group-level judgments by extending R. D. Luce's (1959) model, which was originally developed for individual choice behavior, to a mixed-effects paired comparison model. It is shown that standard multilevel software for binary data can be used to estimate the model. The interpretation of the paired comparison parameters and statistical model tests are discussed in detail. An extensive analysis of an experimental study illustrates the usefulness of a hierarchical approach in modeling multiple pairwise judgments.     

An Alternate Method of Deriving Priorities and Related Inferences for Group Decision Making in Analytic Hierarchy Process

Some work has been carried out in the past on statistically deriving priorities in Analytic Hierarchy Process (AHP). In AHP, the aggregated worths of the alternatives, when compared with respect to several criteria, are estimated in a hierarchical comparisons model introduced by Saaty. In this setup, statistical models are used for Saaty's method of scaling in paired comparisons experiments in any level of the hierarchy. At the end, the final priority weights of the alternatives and related inferences are developed with appropriate statistical methods. Existing statistical methods in the literature assume independence of the entries of the paired comparison matrix. However, these entries are highly dependent among themselves. In this article, we propose a statistical method that allows for the dependence among the entries of the pairwise comparisons matrix. The proposed method is then illustrated with a numerical example. Copyright © 2011 John Wiley & Sons, Ltd.     

A stochastic multidimensional scaling procedure for the empirical determination of convex indifference curves for preference/choice analysis

The vast majority of existing multidimensional scaling (MDS) procedures devised for the analysis of paired comparison preference/choice judgments are typically based on either scalar product (i.e., vector) or unfolding (i.e., ideal-point) models. Such methods tend to ignore many of the essential components of microeconomic theory including convex indifference curves, constrained utility maximization, demand functions, et cetera. This paper presents a new stochastic MDS procedure called MICROSCALE that attempts to operationalize many of these traditional microeconomic concepts. First, we briefly review several existing MDS models that operate on paired comparisons data, noting the particular nature of the utility functions implied by each class of models. These utility assumptions are then directly contrasted to those of microeconomic theory. The new maximum likelihood based procedure, MICROSCALE, is presented, as well as the technical details of the estimation procedure. The results of a Monte Carlo analysis investigating the performance of the algorithm as a number of model, data, and error factors are experimentally manipulated are provided. Finally, an illustration in consumer psychology concerning a convenience sample of thirty consumers providing paired comparisons judgments for some fourteen brands of over-the-counter analgesics is discussed.     

Formal Requirements of Markov State Models for Paired Associate Learning

Discrete state learning models that make Markov assumptions are a powerful tool for the analysis and optimization of performance in paired associate tasks. We seek here to derive bounds on the complexity needed by such models in order to account for the critical effects of lag and retention intervals on paired associate learning. More specifically, after establishing that two different Markov chains are needed (one for describing the effects of trials where a paired associate is presented and one for describing the effects of trials where the paired associate is not presented), we determine the minimum number of states required in a Markov model with two chains. It is shown formally that, under certain psychologically plausible assumptions, more than three states are required. A model with two chains and four states is presented and it is shown empirically that it can account for the lag and retention effects in paired associate learning. Copyright 2001 Academic Press.     

Time- and space-order effects in timed discrimination of brightness and size of paired visual stimuli

Despite the importance of both response probability and response time for testing models of choice, there is a dearth of chronometric studies examining systematic asymmetries that occur over time- and space-orders in the method of paired comparisons. In this study, systematic asymmetries in discriminating the magnitude of paired visual stimuli are examined by way of log odds ratios of binary responses as well as by signed response speed. Hierarchical Bayesian modeling is used to map response probabilities and response speed onto constituent psychological process, and processing capacity is also assessed using response time distribution hazard functions. The findings include characteristic order effects that change systematically in magnitude and direction with changes in the magnitude and separation of the stimuli. After Hellstr?m (1979, 2000), sensation weighting (SW) model analyses show that such order effects are reflected in the weighted accumulation of noisy information about the difference between stimulus values over time, and interindividual differences in weightings asymmetries are related to the relative processing capacity of participants. An account of SW based on the use of reference level information and maximization of signal-to-noise ratios is posited, which finds support from theoretically driven analyses of behavioral data.     

Types produce types: An examination of personality development using Holland's theory

This study was designed to test Holland's hypothesis that parents encourage the development of their own personality types in their offspring. Data were collected by personal interviews from three freshmen classes at a large midwestern university. Parent's personality types (occupations) were compared to offsprings' personality types (expressed occupational choices) using the Kolmogorov-Smirnov one-sample test to determine whether or not the comparisons were due to chance. The results indicated that relationships existed between the personality types of fathers-sons, fathers-daughters and mothers-daughters while no relationship was confirmed between mothers-sons.     

The use of limit theorems in paired and triple comparison model building

The psychological framework proposed by Thompson and Singh (1967) is extended to obtain a statistical interpretation of the stimulus-response phenomenon permitting ties when comparing two stimuli. Using this interpretation, the generalized Bradley-Terry model (Rao and Kupper, 1967) for paired comparisons and the  and  for triple comparisons are derived as the appropriate asymptotic models for experiments permitting tied observations.     

Automated assembly of optimally spaced and balanced paired comparisons: Controlling order effects

To control order effects in questionnaires containing paired comparisons, Ross (1934) described an optimal ordering of the pairings. The pairs can also be balanced so that every stimulus appears equal numbers of times as the first and the second member of a pair. First, we describe and illustrate the optimally spaced, balanced ordering of pairings. Then we show how the optimally spaced, balanced order can be used to implement a matrix-sampling design or a fully incomplete design when the number of stimuli n is so large that respondents cannot reasonably be expected to judge all n(n - 1)/2 pairs. The algorithm for balancing and optimally spacing the list of pairs is described.     

A stochastic model for paired comparisons of social stimuli

A stochastic model for paired comparisons of multiattribute social stimuli is proposed where one objective is to find the relative importance of the attributes for a judge. The model can be conceived as a special strict binary utility model, i.e., a BTL-model, and is related to of the stimuli are linear combinations of functions of the attributes of the stimuli. The model neither assumes that the functions are fixed in advance nor that different judges have the same set of functions. The choice among such functions, however, is admitted only within a finite scope. Within the framework of exponential families, maximum likelihood estimators and tests are derived and applied to data coming from two psychological experiments.     

Theoretical and methodological analysis of partial knowledge

A theory of partial knowledge is proposed as an explanation of cognitive development, and methods are described for testing the theory. The theory consists of three structure-process pairs, each of which postulates a type of cognitive structure and a developmental process specific to that type. In restricted knowledge, a unitary algorithm is the cognitive structure, and amendment is the developmental process. In variable sampling, a structure of unitary substitutes is paired with a process of selection. In variable integration, modular components are paired with self-monitoring. Methods for testing the theory form a sequence of mathematical models. The first model in the sequence, called a model of double assessment, is described both verbally and mathematically. Other models in the sequence are described verbally with reference to other articles for the formal mathematics. Also described are some nonmathematical methods to be used as sequels to the double assessment model.     

Introtacts as predictors of discrimination performance in kindergarten children

Kindergarten children were given simultaneous discrimination tasks with two irrelevant dimensions varying within settings. Prior to each block of eight feed-back trials, the children were asked to attempt to provide a statement of the solution. The introtacts thus provided were found to have several of the desirable characteristics that have been reported for older children with either introtact or blank-trial probes, but which have not been found previously for kindergarten children with the blank-trial probe. Classification of the children according to the trial block on which they first verbalized the correct relevant dimension was found to account for more than 60% of the total variability in discrimination performance. The introtacts were also used to assign parameters to theoretical prediction equations, with nearly two-thirds of the total variability in discrimination performance accounted for by the theory. Extension of the method to other models is discussed.     

The role of the magazine-response contingency on signal-directed responding in pigeons

Three experiments used a procedure for directly delivering water into the thirsty pigeon's mouth to explore the role that the instrumental magazine contingency plays in autoshaped responding. Magazine training was accomplished by requiring birds to contact a “magazine” key when it was illuminated, in order to obtain intraoral injections of water. Other subjects received water injections independent of a magazine-response contingency. Subsequently, magazine-trained subjects showed a transient enhancement in responding to the illumination of another “signal” key, whether that stimulus was presented alone or paired with water delivery. The overall level of maintained autoshaped responding was little influenced by the instrumental magazine contingency, although the within-trial time course of signal-directed responding was affected. When illumination of the signal key preceded access to water only when the signal was not contacted, pigeons directed many responses toward the signal key. However, there was no evidence that the instrumental magazine contingency enhanced responding on this omission schedule of reinforcement. These results thus confirm a small, but measurable contribution of the magazine-response contingency to signal-directed responding; they fail to support the conclusion that the instrumental magazine contingency greatly affects the outcomes of autoshaping studies.     

Paired comparison,triple comparison,and ranking experiments as generalized linear models,and their implementation on GLIM

A wide variety of paired comparison, triple comparison, and ranking experiments may be viewed as generalized linear models. These include paired comparison models based on both the Bradley-Terry and Thurstone-Mosteller approaches, as well as extensions of these models that allow for ties, order of presentation effects, and the presence of covariates. Moreover, the triple comparison model of Pendergrass and Bradley, as well as models for complete rankings of more than three items, can also be represented as generalized linear models. All such models can be easily fit by maximum likelihood, using the widely available GLIM computer package. Additionally, GLIM enables the computation of likelihood ratio statistics for testing many hypotheses of interest. Examples are presented that cover a variety of cases, along with their implementation on GLIM.     

Limited information estimation and testing of Thurstonian models for paired comparison data under multiple judgment sampling

We relate Thurstonian models for paired comparisons data to Thurstonian models for ranking data, which assign zero probabilities to all intransitive patterns. We also propose an intermediate model for paired comparisons data that assigns nonzero probabilities to all transitive patterns and to some but not all intransitive patterns.There is a close correspondence between the multidimensional normal ogive model employed in educational testing and Thurstone's model for paired comparisons data under multiple judgment sampling with minimal identification restrictions. Alike the normal ogive model, Thurstonian models have two formulations, a factor analytic and an IRT formulation. We use the factor analytic formulation to estimate this model from the first and second order marginals of the contingency table using estimators proposed by Muthén. We also propose a statistic to assess the fit of these models to the first and second order marginals of the contingency table. This is important, as a model may reproduce well the estimated thresholds and tetrachoric correlations, yet fail to reproduce the marginals of the contingency table if the assumption of multivariate normality is incorrect.A simulation study is performed to investigate the performance of three alternative limited information estimators which differ in the procedure used in their final stage: unweighted least squares (ULS), diagonally weighted least squares (DWLS), and full weighted least squares (WLS). Both the ULS and DWLS show a good performance with medium size problems and small samples, with a slight better performance of the ULS estimator.This paper is based on the author's doctoral dissertation; Ulf Böckenholt, advisor. The final stages of this research took place while the author was at the Department of Statistics and Econometrics, Universidad Carlos III de Madrid. The author is indebted to Adolfo Hernández for stimulating discussions that helped improve this paper, and to Ulf Böckenholt and the Associate Editor for a number of helpfulsuggestions to a previous draft.