Attentional scanning and space errors

Using the method of paired comparisons, pairs of simultaneous horizontal or vertical lines, with one line above and one below or one on the left and one on the right of a fixation point, respectively, were presented tachistoscopically for length comparison. Space errors were found to have a pattern similar to that of time errors. The tendency to guess the comparative response from the absolute magnitude of stimuli is proposed as a basis for time and space errors. Manipulation of attentional scanning, which implies a more frequent usage of this guessing strategy for one of the two lines in a pair, was shown to affect space errors.     

Attentional scanning and space errors.

Using the method of paired comparisons, pairs of simultaneous horizontal or vertical lines, with one line above and one below or one on the left and one on the right of a fixation point, respectively, were presented tachistoscopically for length comparison. Space errors were found to have a pattern similar to that of time errors. The tendency to guess the comparative response from the absolute magnitude of stimuli is proposed as a basis for time and space errors. Manipulation of attentional scanning, which implies a more frequent usage of this guessing strategy for one of the two lines in a pair, was shown to affect space errors.     

Item difficulty parameter estimation using the idea of the graded response model and computerized adaptive testing

Abstract:  In test operations using IRT (item response theory), items are included in a test before being used to rate subjects and the response data is used to estimate their item parameters. However, this method of test operation may lead to item content leakage and an adequate test operation can become difficult. To address this problem, Ozaki and Toyoda (2005, 2006 ) developed item difficulty parameter estimation methods that use paired comparison data from the perspective of the difficulty of items as judged by raters familiar with the field. In the present paper, an improved method of item difficulty parameter estimation is developed. In this new method, an item for which the difficulty parameter is to be estimated is compared with multiple items simultaneously, from the perspective of their difficulty. This is not a one-to-one comparison but a one-to-many comparison. In the comparisons, raters are informed that items selected from an item pool are ordered according to difficulty. The order will provide insight to improve the accuracy of judgment.     

Comparison is not just subtraction: effects of time- and space-order on subjective stimulus difference

In five experiments, participants made comparative judgments of paired successive or simultaneous stimuli. Time- or space-order errors were obtained, which varied with the interstimulus interval (ISI) or stimulus duration, as well as with the stimulus level. The results, in terms of scaled subjective differences, are well described by Hellstr?m's (1979) sensation-weighting model. With successive presentation, in comparisons of line length and tone loudness, the first stimulus had the greater weight in determining the subjective difference for short ISIs, the second for longer ISIs. In comparisons of duration (auditory and visual), the second stimulus had the greater weight. For simultaneously presented line lengths, the left stimulus had the greater weight.     

Limited information estimation and testing of Thurstonian models for paired comparison data under multiple judgment sampling

We relate Thurstonian models for paired comparisons data to Thurstonian models for ranking data, which assign zero probabilities to all intransitive patterns. We also propose an intermediate model for paired comparisons data that assigns nonzero probabilities to all transitive patterns and to some but not all intransitive patterns.There is a close correspondence between the multidimensional normal ogive model employed in educational testing and Thurstone's model for paired comparisons data under multiple judgment sampling with minimal identification restrictions. Alike the normal ogive model, Thurstonian models have two formulations, a factor analytic and an IRT formulation. We use the factor analytic formulation to estimate this model from the first and second order marginals of the contingency table using estimators proposed by Muthén. We also propose a statistic to assess the fit of these models to the first and second order marginals of the contingency table. This is important, as a model may reproduce well the estimated thresholds and tetrachoric correlations, yet fail to reproduce the marginals of the contingency table if the assumption of multivariate normality is incorrect.A simulation study is performed to investigate the performance of three alternative limited information estimators which differ in the procedure used in their final stage: unweighted least squares (ULS), diagonally weighted least squares (DWLS), and full weighted least squares (WLS). Both the ULS and DWLS show a good performance with medium size problems and small samples, with a slight better performance of the ULS estimator.This paper is based on the author's doctoral dissertation; Ulf Böckenholt, advisor. The final stages of this research took place while the author was at the Department of Statistics and Econometrics, Universidad Carlos III de Madrid. The author is indebted to Adolfo Hernández for stimulating discussions that helped improve this paper, and to Ulf Böckenholt and the Associate Editor for a number of helpfulsuggestions to a previous draft.     

Item Response Modeling of Paired Comparison and Ranking Data

The comparative format used in ranking and paired comparisons tasks can significantly reduce the impact of uniform response biases typically associated with rating scales. Thurstone's (1927, 1931) model provides a powerful framework for modeling comparative data such as paired comparisons and rankings. Although Thurstonian models are generally presented as scaling models, that is, stimuli-centered models, they can also be used as person-centered models. In this article, we discuss how Thurstone's model for comparative data can be formulated as item response theory models so that respondents' scores on underlying dimensions can be estimated. Item parameters and latent trait scores can be readily estimated using a widely used statistical modeling program. Simulation studies show that item characteristic curves can be accurately estimated with as few as 200 observations and that latent trait scores can be recovered to a high precision. Empirical examples are given to illustrate how the model may be applied in practice and to recommend guidelines for designing ranking and paired comparisons tasks in the future.     

A covariance analysis of multiple paired comparisons

In the method of multiple paired comparisons the dominance of objectj over objectk is observed uponp attributes. The present paper develops a covariance analysis for these paired comparisons in terms of a linear model which includes scale, bias, and interaction effects, along withs covariants upon which the comparisons are presumably dependent. The covariance model gives rise to adjusted parameter estimates and hypothesis tests for the residual pairwise layout from which the effects of thes covariants have been removed. These estimation and testing procedures are illustrated with an analysis of political judgment data, and their relevance to the general problem of residual scaling is discussed.This research was supported by National Institute of Mental Health grants MH 12972 and MH 15506. The author would like to express his appreciation to Jerry Solomon and Wei-Ching Chang of the Oregon Research Institute for the data analysis reported at the end of the paper. Computing assistance was obtained from the Health Sciences Computing Facility, UCLA, sponsored by N.I.H. Grant FR-3.     

Hierarchical modeling of paired comparison data

The method of paired comparisons belongs to a small group of techniques that provide explicit information about the consistency of individual and aggregated choices. This article investigates the link between the individual- and group-level judgments by extending R. D. Luce's (1959) model, which was originally developed for individual choice behavior, to a mixed-effects paired comparison model. It is shown that standard multilevel software for binary data can be used to estimate the model. The interpretation of the paired comparison parameters and statistical model tests are discussed in detail. An extensive analysis of an experimental study illustrates the usefulness of a hierarchical approach in modeling multiple pairwise judgments.     

An application of a biased version of the Bradley-Terry-Luce model to professional basketball results

Some extensions of the Bradley-Terry-Luce model for paired comparisons are presented and used to assess the existence of a home court advantage for professional basketball teams. It is shown how maximum likelihood estimates for these models can be obtained from a simple iterative scaling algorithm. Won-loss records for the regular season are used to make predictions for playoff series. The models are extended to include possible differences among divisions. These models can be applied in many other situations to incorporate group effects and order effects into analysis of paired comparisons.     

Identification and Small Sample Estimation of Thurstone's Unrestricted Model for Paired Comparisons Data

The interpretation of a Thurstonian model for paired comparisons where the utilities' covariance matrix is unrestricted proved to be difficult due to the comparative nature of the data. We show that under a suitable constraint the utilities' correlation matrix can be estimated, yielding a readily interpretable solution. This set of identification constraints can recover any true utilities' covariance matrix, but it is not unique. Indeed, we show how to transform the estimated correlation matrix into alternative correlation matrices that are equally consistent with the data but may be more consistent with substantive theory. Also, we show how researchers can investigate the sample size needed to estimate a particular model by exploiting the simulation capabilities of a popular structural equation modeling statistical package.     

An individual difference model for the multidimensional analysis of preference data

A model for the analysis of paired comparison data is presented which combines features of the BTL-model with features of the Unfolding model. The model is metric, mathematically tractable, and has an exact algebraic solution. Since it is multidimensional and allows for individual differences, it is thought to be more realistic for some choice situations than either the Thurstone model or the BTL-model. No claim is made that the present model will be appropriate for all conceivable choice situations. Rather, it is argued that the fact that it is explicitly falsifiable is a point in its favor.This work was supported by a grant of the United States Public Health Service (1 R03 MH19139 01 MSM) and a David Ross (XR) grant of the Purdue Research Foundation (PRF 2132), both of which are gratefully acknowledged. We also owe thanks to Dr. Terry Cooper, whose careful scrutiny of an earlier draft of this paper allowed us to reduce the number of errors from n to n-1.     

A stochastic multidimensional scaling procedure for the empirical determination of convex indifference curves for preference/choice analysis

The vast majority of existing multidimensional scaling (MDS) procedures devised for the analysis of paired comparison preference/choice judgments are typically based on either scalar product (i.e., vector) or unfolding (i.e., ideal-point) models. Such methods tend to ignore many of the essential components of microeconomic theory including convex indifference curves, constrained utility maximization, demand functions, et cetera. This paper presents a new stochastic MDS procedure called MICROSCALE that attempts to operationalize many of these traditional microeconomic concepts. First, we briefly review several existing MDS models that operate on paired comparisons data, noting the particular nature of the utility functions implied by each class of models. These utility assumptions are then directly contrasted to those of microeconomic theory. The new maximum likelihood based procedure, MICROSCALE, is presented, as well as the technical details of the estimation procedure. The results of a Monte Carlo analysis investigating the performance of the algorithm as a number of model, data, and error factors are experimentally manipulated are provided. Finally, an illustration in consumer psychology concerning a convenience sample of thirty consumers providing paired comparisons judgments for some fourteen brands of over-the-counter analgesics is discussed.     

The use of limit theorems in paired and triple comparison model building

The psychological framework proposed by Thompson and Singh (1967) is extended to obtain a statistical interpretation of the stimulus-response phenomenon permitting ties when comparing two stimuli. Using this interpretation, the generalized Bradley-Terry model (Rao and Kupper, 1967) for paired comparisons and the  and  for triple comparisons are derived as the appropriate asymptotic models for experiments permitting tied observations.     

Modelling the reliability of paired comparisons

This paper considers paired comparison experiments for which weights describing the reliability of the comparisons are available. Such a weight can be represented by a belief function as defined by G. Shafer in A Mathematical Theory of Evidence (Princeton, NJ: Princeton Univ. Press, 1976). We develop a belief function analysis for this case, and discuss the interpretation of such an analysis and the conditions for its validity.     

On the importance of distinguishing between within- and between-subject effects in intransitive intertemporal choice

Most probabilistic paired comparison models treat inconsistent choices as caused by independent and random errors in the pairwise judgments. In this paper, we argue that this assumption is too restrictive for the analysis of paired comparison data obtained from multiple judges when transitivity violations are systematic. We present a new framework that contains the random error assumption as a special case but also allows for systematic changes in an option's utility assessments over the pairwise comparisons. Accounting for both between- and within-judge sources of variability, we demonstrate in an application on intertemporal choice that the proposed framework can capture systematic transitivity violations as well as individual taste differences.     

Timed magnitude comparisons of numerical and nonnumerical expressions of uncertainty

Two experiments involving paired comparisons of numerical and nonnumerical expressions of uncertainty are reported. Subjects were timed under two opposing sets of instructions ("choose higher probability" vs. "choose lower probability"). Numerical comparisons were consistently faster and easier than their nonnumerical counterparts. Consistent distance and congruity effects were obtained, illustrating that both numerical and nonnumerical expressions of uncertainty contain subjective magnitude information, and suggesting that similar processes are employed in manipulating and comparing numerical and verbal terms. To account for the general pattern of results obtained, Holyoak's reference point model (1978) was generalized by explicitly including the vagueness of the nonnumerical expressions. This generalized model is based on the notion that probability expressions can be represented by membership functions (Wallsten, Budescu, Rapoport, Zwick, & Forsyth, 1986) from which measures of location for each word, and measures of overlap for each pair can be derived. A good level of fit was obtained for this model at the individual level.     

An Alternate Method of Deriving Priorities and Related Inferences for Group Decision Making in Analytic Hierarchy Process

Some work has been carried out in the past on statistically deriving priorities in Analytic Hierarchy Process (AHP). In AHP, the aggregated worths of the alternatives, when compared with respect to several criteria, are estimated in a hierarchical comparisons model introduced by Saaty. In this setup, statistical models are used for Saaty's method of scaling in paired comparisons experiments in any level of the hierarchy. At the end, the final priority weights of the alternatives and related inferences are developed with appropriate statistical methods. Existing statistical methods in the literature assume independence of the entries of the paired comparison matrix. However, these entries are highly dependent among themselves. In this article, we propose a statistical method that allows for the dependence among the entries of the pairwise comparisons matrix. The proposed method is then illustrated with a numerical example. Copyright © 2011 John Wiley & Sons, Ltd.     

The scaling of test scores by the method of paired comparisons

The purpose of this study is to investigate, by the method of paired comparisons, a possible scaling of individuals who have made certain test scores, such that the additive property will be satisfied and such that a stability in scaling will be maintained,—in other words, a scaling such that the scaled score of an individual will remain relatively the same regardless of the grouping of individuals in which he may be placed. The results show that it is possible to utilize psychophysical methods in psychological and educational test situations. Among the major findings are that Case V of the Law of Comparative Judgment is applicable to the data in this problem, the method of dividing the intermediate category equally between the greater and the less was the best of three possible methods, internal consistency was satisfied, and, finally, when a new test of stability was applied, it was found that the distances between the hypothetical individuals remain the same.The writer wishes to express appreciation for the invaluable help and guidance of Professor Harold O. Gulliksen and also to Professor Marion W. Richardson who suggested the problem and made valuable suggestions.     

A model for repeat paired comparison preference tests

A model is proposed that describes subject behavior on repeat paired comparison preference tests. The model extends prior work in this area in that it explicitly allows for abstentions and permits the derivation of individual true scores for discrimination ability as well as conditional estimates of proportionate preference. With these results, the properties of a paired comparison test can be thoroughly explored. An empirical example is presented, and test design issues are considered. In particular, repeat paired comparison preference tests are shown to be inherently less efficient discrimination tests than are pick 1 of 2 tests.