The scaling of test scores by the method of paired comparisons

The purpose of this study is to investigate, by the method of paired comparisons, a possible scaling of individuals who have made certain test scores, such that the additive property will be satisfied and such that a stability in scaling will be maintained,—in other words, a scaling such that the scaled score of an individual will remain relatively the same regardless of the grouping of individuals in which he may be placed. The results show that it is possible to utilize psychophysical methods in psychological and educational test situations. Among the major findings are that Case V of the Law of Comparative Judgment is applicable to the data in this problem, the method of dividing the intermediate category equally between the greater and the less was the best of three possible methods, internal consistency was satisfied, and, finally, when a new test of stability was applied, it was found that the distances between the hypothetical individuals remain the same.The writer wishes to express appreciation for the invaluable help and guidance of Professor Harold O. Gulliksen and also to Professor Marion W. Richardson who suggested the problem and made valuable suggestions.     

A covariance analysis of multiple paired comparisons

In the method of multiple paired comparisons the dominance of objectj over objectk is observed uponp attributes. The present paper develops a covariance analysis for these paired comparisons in terms of a linear model which includes scale, bias, and interaction effects, along withs covariants upon which the comparisons are presumably dependent. The covariance model gives rise to adjusted parameter estimates and hypothesis tests for the residual pairwise layout from which the effects of thes covariants have been removed. These estimation and testing procedures are illustrated with an analysis of political judgment data, and their relevance to the general problem of residual scaling is discussed.This research was supported by National Institute of Mental Health grants MH 12972 and MH 15506. The author would like to express his appreciation to Jerry Solomon and Wei-Ching Chang of the Oregon Research Institute for the data analysis reported at the end of the paper. Computing assistance was obtained from the Health Sciences Computing Facility, UCLA, sponsored by N.I.H. Grant FR-3.     

Remarks on the method of paired comparisons: III. A test of significance for paired comparisons when equal standard deviations and equal correlations are assumed

A test of goodness of fit is developed for Thurstone's method of paired comparisons, Case V. The test involves the computation of , wheren is the number of observations per pair, and and are the angles obtained by applying the inverse sine transformation to the fitted and the observed proportions respectively. The number of degrees of freedom is (k–1) (k–2)/2.This research was performed in the Laboratory of Social Relations under a grant made available to Harvard University by the RAND Corporation under the Department of the Air Force, Project RAND.     

Remarks on the test of significance for the method of paired comparisons

A three-component model for comparative judgment which allows for individual differences in preference is proposed. An implication of the model is that errors in the observed proportions due to sampling individuals in paired comparisons experiments are correlated. By neglecting this correlation, Mosteller's test for the method of paired comparisons tends to accept falsely the goodness of fit of the Case V solution. It is shown that bounds may be set for the correlation effect which make a valid test possible in some cases and provide useful standard errors for the estimated affective values.Preparation of this paper has been supported in part by the Quartermaster Food and Container Institute for the Armed Forces. Views and conclusions expressed herein are those of the author and do not necessarily reflect the views or endorsement of the Department of Defense. Comments of one of the reviewers, which substantially improved an earlier version of this paper, are gratefully acknowledged.     

Modelling the reliability of paired comparisons

This paper considers paired comparison experiments for which weights describing the reliability of the comparisons are available. Such a weight can be represented by a belief function as defined by G. Shafer in A Mathematical Theory of Evidence (Princeton, NJ: Princeton Univ. Press, 1976). We develop a belief function analysis for this case, and discuss the interpretation of such an analysis and the conditions for its validity.     

A note on the use of triads for paired comparisons

When scaling a large number of stimuli from comparative judgments, considerable savings in time and labor may be realized if stimuli are presented in triad form rather than in pairs. If, forN stimuli, the proper configuration of triads can be assembled so that all possible pairs appear once, the paired judgment matrix may be reproduced with one-third fewer judgments and two-thirds fewer presentations than would be required with complete pairing. A simple procedure is described for enumerating triad configurations for whichN is an odd multiple of three.     

A stochastic model for paired comparisons of social stimuli

A stochastic model for paired comparisons of multiattribute social stimuli is proposed where one objective is to find the relative importance of the attributes for a judge. The model can be conceived as a special strict binary utility model, i.e., a BTL-model, and is related to of the stimuli are linear combinations of functions of the attributes of the stimuli. The model neither assumes that the functions are fixed in advance nor that different judges have the same set of functions. The choice among such functions, however, is admitted only within a finite scope. Within the framework of exponential families, maximum likelihood estimators and tests are derived and applied to data coming from two psychological experiments.     

Successive intervals scaling of paired comparisons

In this paper a method is developed for analyzing data resulting from the use of a modification of the usual paired comparisons procedure which allows for ratings of size of difference. The scaling model is essentially an extension of the Thurstonian successive intervals model. The method is applied to scaling of a set of nine immoral actions and the results agree rather well with those from conventional successive intervals scaling of the same stimuli.This study was supported by a grant from the Swedish Council for Social Science Research.     

A technique for the nonmetric analysis of paired comparisons of psychological intervals

A nonmetric coordinate adjustment technique is developed which determines scale values for objects whose interobject intervals (differences in subjective value) have been directly compared. In Monte Carlo simulations, the degree of metric determinancy of the scale values is shown to be quite high even when the amount of error is relatively high. This robustness under high-error conditions permitted the analysis of individual subject data in experiments on the direct comparison of loudness differences and loudness ratios where only one judgment per interval comparison was obtained per subject.This research was supported by a grant from the National Research Council of Canada.     

A least squares solution for paired comparisons with incomplete data

A precise and rapid procedure has been devised for dealing with a matrix of incomplete data in paired comparisons. This method should increase the general applicability of paired comparisons since experiments involving large numbers of stimuli may now be shortened to feasible experimental proportions. Also, we may now use sets of stimuli which cover a wide range, resulting in a considerable number of 100 per cent vs. 0 per cent judgments, and still give a precise solution depending equally on each of the observations.This study was supported in part by the Office of Naval Research Contract N6onr-270-20 with Princeton University and by the National Science Foundation Grant G-642. The opinions expressed are, of course, those of the author and do not represent attitudes or policies of the Office of Naval Research or of the National Science Foundation.The author wishes to acknowledge helpful suggestions and comments received from Frederic M. Lord, Warren S. Torgerson, and Ledyard R Tucker. Thanks are also due to Mrs. Gertrude Diederich for some of the tabulating and computing for the illustrative problem.     

A general procedure for obtaining paired comparisons from multiple rank orders

From a theoretical point of view, paired comparisons and the law of comparative judgment provide an excellent approach to the problem of psychological measurement. However, if a reasonably large number of stimuli are to be investigated, paired comparisons become extremely time-consuming and fatiguing to the subjects. A balanced incomplete block design, requiring multiple rank order judgments for each subject, provides an efficient experimental method for obtaining paired comparisons judgments. Features of the analysis proposed for this design are discussed in detail. A program for the analysis is available for the IBM 650 electronic computer.Prepared in connection with research done under Office of Naval Research Contract Nonr 1858-(15), Project Designation NR 150-088, and National Science Foundation Grant G-3407. Reproduction of any part of this material is permitted for any purpose of the United States Government.     

Determining the degree of inconsistency in a set of paired comparisons

Consistency in paired comparison data is defined. Two types of inconsistency which may arise are defined. Computational formulas for these types of inconsistency are derived, and examples illustrating the use of these formulas are presented.These ideas were developed while the first author was on the staff of the Research Center for Human Relations. The work was made possible by the ONR contract NONR 285(10). The authors are indebted to Jack Moshman for his helpful critical suggestions. The United States Government is authorized to reprint this article in whole or in part.     

The rational side of egocentrism in social comparisons

Prior work has found that when people compare themselves with others they egocentrically focus on their own strengths and achievements more than on the (equally relevant) strengths and achievements of the comparison group. As a consequence, people tend to overestimate their comparative standing when absolute standing is high and underestimate their comparative standing when absolute standing is low. The present research investigated a rational discounting explanation of this bias—namely, that people weight the target of the comparison (the self) more than the referent of the comparison (others) because they typically have more knowledge about the former than the latter. In three studies, we found that the tendency to focus on the target in social comparisons—and the over and underestimation of relative standing that tendency engenders—was reduced (but not eliminated) as people’s knowledge about the comparison group increased. These results suggest that there may be a rational side to egocentrism in social comparisons.     

Appearance-related social comparisons: the role of contingent self-esteem and self-perceptions of attractiveness

Two studies examined contingent self-esteem (CSE) and responses to appearance-related social comparisons. Study 1 was an experimental study in which women rated a series of advertisements from popular women's magazines. Study 2 employed an event-contingent diary recording procedure. In Study 1, women who were higher in CSE and lower in self-perceptions of attractiveness (SPA) experienced greater decreases in positive affect and greater increases in negative affect following the ad-rating task. Study 2 results supported a mediation model in which women who were higher in CSE felt worse after social comparisons because they made primarily upward comparisons. Overall, results suggest that appearance-related comparisons are more distressing for those who base their self-worth on contingencies and have lower self-perceived attractiveness.