Hierarchical modeling of paired comparison data

The method of paired comparisons belongs to a small group of techniques that provide explicit information about the consistency of individual and aggregated choices. This article investigates the link between the individual- and group-level judgments by extending R. D. Luce's (1959) model, which was originally developed for individual choice behavior, to a mixed-effects paired comparison model. It is shown that standard multilevel software for binary data can be used to estimate the model. The interpretation of the paired comparison parameters and statistical model tests are discussed in detail. An extensive analysis of an experimental study illustrates the usefulness of a hierarchical approach in modeling multiple pairwise judgments.     

A stochastic multidimensional scaling procedure for the empirical determination of convex indifference curves for preference/choice analysis

The vast majority of existing multidimensional scaling (MDS) procedures devised for the analysis of paired comparison preference/choice judgments are typically based on either scalar product (i.e., vector) or unfolding (i.e., ideal-point) models. Such methods tend to ignore many of the essential components of microeconomic theory including convex indifference curves, constrained utility maximization, demand functions, et cetera. This paper presents a new stochastic MDS procedure called MICROSCALE that attempts to operationalize many of these traditional microeconomic concepts. First, we briefly review several existing MDS models that operate on paired comparisons data, noting the particular nature of the utility functions implied by each class of models. These utility assumptions are then directly contrasted to those of microeconomic theory. The new maximum likelihood based procedure, MICROSCALE, is presented, as well as the technical details of the estimation procedure. The results of a Monte Carlo analysis investigating the performance of the algorithm as a number of model, data, and error factors are experimentally manipulated are provided. Finally, an illustration in consumer psychology concerning a convenience sample of thirty consumers providing paired comparisons judgments for some fourteen brands of over-the-counter analgesics is discussed.     

An internal consistency check for scale values determined by the method of successive intervals

The method of successive intervals is a psychological scaling procedure in which stimuli are classified into successive intervals according to the degree of some defined attribute which they are judged to possess. A psychological continuum is defined and the scale values are then taken as the medians of the distributions of judgments on the psychological continuum. It is assumed that the distributions of judgments for each stimulus are normal on the psychological continuum as defined.An internal consistency check indicates that the cumulative distributions of empirical judgments for the various stimuli can be reproduced by means of a limited number of parameters with an average error that compares favorably with that usually reported for paired comparison data. Furthermore, the scale values obtained by successive interval scaling, for the data reported, are shown to be linearly related to those obtained by the method of paired comparisons.This paper was written while the first author was a post-doctoral Research Training Fellow of the Social Science Research Council studying at the University of Chicago. It reports research undertaken in cooperation with the Quartermaster Food and Container Institute for the Armed Forces. The views or conclusions contained in this report are those of the authors. They are not to be construed as necessarily reflecting the views or endorsement of the Department of Defense.     

A new scaling technique for absolute judgments

The results of an experiment using the method of absolute judgments can be viewed as a matrix of conditional probabilities in which the rows represent stimuli and the columns responses. The cosine of the angle between two row vectors is a measure of the similarity of the corresponding stimuli. This cosine provides the basis for a method of scaling the stimuli. Unlike the method of paired comparisons, this new technique does not require arbitrary fixing of a unit of measurement. A numerical example is given.     

A least squares solution for paired comparisons with incomplete data

A precise and rapid procedure has been devised for dealing with a matrix of incomplete data in paired comparisons. This method should increase the general applicability of paired comparisons since experiments involving large numbers of stimuli may now be shortened to feasible experimental proportions. Also, we may now use sets of stimuli which cover a wide range, resulting in a considerable number of 100 per cent vs. 0 per cent judgments, and still give a precise solution depending equally on each of the observations.This study was supported in part by the Office of Naval Research Contract N6onr-270-20 with Princeton University and by the National Science Foundation Grant G-642. The opinions expressed are, of course, those of the author and do not represent attitudes or policies of the Office of Naval Research or of the National Science Foundation.The author wishes to acknowledge helpful suggestions and comments received from Frederic M. Lord, Warren S. Torgerson, and Ledyard R Tucker. Thanks are also due to Mrs. Gertrude Diederich for some of the tabulating and computing for the illustrative problem.     

DISCRIMINATION OF LATERALLY DISPLAYED HANDWRITING AND HANDWRITTEN WORDS*

Four highly discriminable words each written in four highly discriminable natural kinds of handwriting were paired and laterally displayed for 180 msec. Thirty-two normal females made same/different judgments for the words and for handwriting; there were 136 trials in each condition. Accuracy and manual reaction times of correct responses were recorded. The usual right-field (left cerebral hemisphere) superiority was observed for word comparisons, but no clear asymmetry emerged for handwriting comparisons. An interference of the words in the comparisons of handwriting was observed mainly in the right field, and interference of the handwriting in the judgments on words occurred equally in both fields.     

Structural equation modeling of paired-comparison and ranking data

L. L. Thurstone's (1927) model provides a powerful framework for modeling individual differences in choice behavior. An overview of Thurstonian models for comparative data is provided, including the classical Case V and Case III models as well as more general choice models with unrestricted and factor-analytic covariance structures. A flow chart summarizes the model selection process. The authors show how to embed these models within a more familiar structural equation modeling (SEM) framework. The different special cases of Thurstone's model can be estimated with a popular SEM statistical package, including factor analysis models for paired comparisons and rankings. Only minor modifications are needed to accommodate both types of data. As a result, complex models for comparative judgments can be both estimated and tested efficiently.     

Modelling intransitive preferences: A random-effects approach

Current models for individual-level paired comparison data are based on the three assumptions that (1) pairwise judgments are independent, (2) the utility of an item remains invariant across trials, and (3) pair-specific variability can account for intransitive choice behaviour. All three assumptions seem strong and likely to be violated in empirical applications. This paper introduces a new framework for the analysis of paired comparison data which relaxes these three assumptions and considers the utilities associated with the same item across trials to be neither independent nor identical, but related. The proposed approach provides new insights about the reliability and consistency of paired comparison judgments and can account for systematic violations of transitivity. An application, based on a replication of Tversky's [(1969). Intransitivity of preference. Psychological Review, 76, 31-48] gamble study, illustrates the usefulness of the new approach in modelling both transitive and intransitive preferences.     

Two logit models for external analysis of preferences

A logit vector model and a logit ideal point model are presented for external analysis of paired comparison preference judgments aggregated over a homogeneous group. The logit vector model is hierarchically nested within the logit ideal point model so that statistical tests are available to distinguish between these two models. Generalized least squares estimation procedures are developed to account for heteroscedastic sampling error variances and specification error variances. Two numerical illustrations deal with judgments concerning employee compensation plans and preferences for salt and sugar in the brine of canned green beans.     

Classical Psychophysics and the Assessment of Agreement and Accuracy In Judgments of Personality

ABSTRACT The just noticeable difference (jnd) unit of classical psycho-physics is introduced as a new way to describe accuracy and agreement in observer evaluations of personality. A formula for estimating jnd's from typically available summary statistics is derived from Thurstone's law of comparative judgment. A study examining four traits judged in 10 samples of subjects where the design permitted the calculation of jnd's by the method of paired comparisons indicated that the formula predicted the empirically derived jnd's associated with the mean judge ratings with considerable precision. Jnd's of criterion measures were also predicted, and while fit was somewhat less impressive in this case, there was still appreciable convergence between predicted and empirical values. The implications of jnd measures of agreement and accuracy are discussed. These implications include (a) possibilities for increased understanding of bias in observer judgments, (b) a new recognition that equal correlations to external criteria do not necessarily imply equal accuracy, and (c) alternative ways of describing the magnitude of effect in psychological research.     

On the importance of distinguishing between within- and between-subject effects in intransitive intertemporal choice

Most probabilistic paired comparison models treat inconsistent choices as caused by independent and random errors in the pairwise judgments. In this paper, we argue that this assumption is too restrictive for the analysis of paired comparison data obtained from multiple judges when transitivity violations are systematic. We present a new framework that contains the random error assumption as a special case but also allows for systematic changes in an option's utility assessments over the pairwise comparisons. Accounting for both between- and within-judge sources of variability, we demonstrate in an application on intertemporal choice that the proposed framework can capture systematic transitivity violations as well as individual taste differences.     

Modeling within-subject dependencies in ordinal paired comparison data

This paper presents two probabilistic models based on the logistic and the normal distribution for the analysis of dependencies in individual paired comparison judgments. It is argued that a core assumption of latent class choice models, independence of individual decisions, may not be well-suited for the analysis of paired comparison data. Instead, the analysis and interpretation of paired comparison data may be much simplified by allowing for within-person dependencies that result from repeated evaluations of the same options in different pairs. Moreover, by relating dependencies among the individual-level responses to (in)consistencies in the judgmental process, we show that the proposed graded paired comparison models reduce to ranking models under certain conditions. Three applications are presented to illustrate the approach.This research was partially supported by NSF grant SBR-9409531. The authors are grateful to the reviewers, Alan Agresti and Herbert Hoijtink for their helpful comments on this research.     

Was Life Better in the “Good Old Days”? Intertemporal Judgments of Life Satisfaction

Intertemporal judgments are paired comparisons between the present time and some other time (e.g. “How satisfied are you with your life these days compared to five years ago?”). These judgments can provide evidence on the question, “Is life satisfaction in developed nations increasing, decreasing, or remaining constant?” This paper provides the first review of intertemporal judgments of life satisfaction, and reports a meta-analysis of 71 such studies from 9 developed countries. Results show that in every survey that asks people how happy their own lives are now compared to some past time, a majority says they are happier now than in the past. The meta-analysis also shows that this question must be carefully distinguished from asking about quality of life of the average person, which shows a majority believing that life was better for the “average person” in the past. These two beliefs are logically inconsistent, since the average respondent actually says their satisfaction is higher now. We consider several sources of bias that may influence these results, including cognitive processing heuristics, self-appraisal (desirability) bias, and information bias in media news.     

A dissociation between causal judgment and outcome recall

It has been suggested that causal learning in humans is similar to Pavlovian conditioning in animals. According to this view, judgments of cause reflect the degree to which an association exists between the cause and the effect. Inferential accounts, by contrast, suggest that causal judgments are reasoning based rather than associative in nature. We used a direct measure of associative strength, identification of the outcome with which a cause was paired (cued recall), to see whether associative strength translated directly into causal ratings. Causal compounds AB+ and CD+ were intermixed with A+ and C- training. Cued-recall performance was better for cue B than for cue D; thus, associative strength was inherited by cue B from the strongly associated cue A (augmentation). However, the reverse was observed on the causal judgment measure: Cue B was judged to be less causal than D (cue competition). These results support an inferential over an associative account of causal judgments.     

Asymptotic judgment of cause in a relative validity paradigm

We report three experiments in which we tested asymptotic and dynamic predictions of the Rescorla-Wagner (R-W) model and the asymptotic predictions of Cheng's probabilistic contrast model (PCM) concerning judgments of causality when there are two possible causal candidates. We used a paradigm in which the presence of a causal candidate that is highly correlated with an effect influences judgments of a second, moderately correlated or uncorrelated cause. In Experiment 1, which involved a moderate outcome density, judgments of a moderately positive cause were attenuated when it was paired with either a perfect positive or perfect negative cause. This attenuation was robust over a large set of trials but was greater when the strong predictor was positive. In Experiment 2, in which there was a low overall density of outcomes, judgments of a moderately correlated positive cause were elevated when this cause was paired with a perfect negative causal candidate. This elevation was also quite robust over a large set of trials. In Experiment 3, estimates of the strength of a causal candidate that was uncorrelated with the outcome were reduced when it was paired with a perfect cause. The predictions of three theoretical models of causal judgments are considered. Both the R-W model and Cheng's PCM accounted for some but not all aspects of the data. Pearce's model of stimulus generalization accounts for a greater proportion of the data.     

Comparative Optimism and Event Skewness

Comparative judgment biases—wherein a majority of people report being above‐ or below‐average in their abilities, traits, or future events—are a robust phenomenon in psychology. A recent explanation for these biases has focused on people's awareness that many comparative judgment domains form skewed distributions, and, hence, a majority of people can feasibly be above or below average. Indeed, this prior research found that comparative biases for abilities emerged more for skewed (vs. normal) distributions. In the current research, we attempted to (i) conceptually replicate this finding in a comparative likelihood context and (ii) provide evidence of an alternative explanation for the prior results. Replicating prior research, three correlational studies and one experimental study found that event skewness was related to direct comparative likelihood judgments for health events, such that comparative optimism emerged more for events judged or manipulated to come from positively skewed distributions than from negatively skewed distributions. However, event skewness was unrelated to indirect comparisons (absolute self minus absolute other). Moreover, consistent with an egocentric‐processes account, absolute self‐judgments were more predictive of direct comparisons than were absolute other judgments and showed the same association with event skewness as direct comparisons. Implications for explaining and interpreting comparative judgment biases are discussed. Copyright © 2016 John Wiley & Sons, Ltd.     

Remarks on the test of significance for the method of paired comparisons

A three-component model for comparative judgment which allows for individual differences in preference is proposed. An implication of the model is that errors in the observed proportions due to sampling individuals in paired comparisons experiments are correlated. By neglecting this correlation, Mosteller's test for the method of paired comparisons tends to accept falsely the goodness of fit of the Case V solution. It is shown that bounds may be set for the correlation effect which make a valid test possible in some cases and provide useful standard errors for the estimated affective values.Preparation of this paper has been supported in part by the Quartermaster Food and Container Institute for the Armed Forces. Views and conclusions expressed herein are those of the author and do not necessarily reflect the views or endorsement of the Department of Defense. Comments of one of the reviewers, which substantially improved an earlier version of this paper, are gratefully acknowledged.     

A new technique for investigating the feeling of knowing

Previous investigations of the feeling of knowing (FOK) have relied on absolute FOK judgements rather than on relative FOK judgments. This has resulted in a confounding of (1) the subject's metamemorial knowledge of nonrecalled items with (2) the subject's know/don't-know threshold. The new technique replaces the absolute FOK judgments with relative FOK judgments in which the subject generates (via paired comparisons) a FOK rank order of nonrecalled items in terms of the predicted likelihood of recognition. This new technique eliminates the aforementioned confounding, provides a richer data base, and yields separate estimates of FOK validity and FOK reliability.     

Modelling the reliability of paired comparisons

This paper considers paired comparison experiments for which weights describing the reliability of the comparisons are available. Such a weight can be represented by a belief function as defined by G. Shafer in A Mathematical Theory of Evidence (Princeton, NJ: Princeton Univ. Press, 1976). We develop a belief function analysis for this case, and discuss the interpretation of such an analysis and the conditions for its validity.     

Comparison is not just subtraction: effects of time- and space-order on subjective stimulus difference

In five experiments, participants made comparative judgments of paired successive or simultaneous stimuli. Time- or space-order errors were obtained, which varied with the interstimulus interval (ISI) or stimulus duration, as well as with the stimulus level. The results, in terms of scaled subjective differences, are well described by Hellstr?m's (1979) sensation-weighting model. With successive presentation, in comparisons of line length and tone loudness, the first stimulus had the greater weight in determining the subjective difference for short ISIs, the second for longer ISIs. In comparisons of duration (auditory and visual), the second stimulus had the greater weight. For simultaneously presented line lengths, the left stimulus had the greater weight.