A general procedure for obtaining paired comparisons from multiple rank orders

From a theoretical point of view, paired comparisons and the law of comparative judgment provide an excellent approach to the problem of psychological measurement. However, if a reasonably large number of stimuli are to be investigated, paired comparisons become extremely time-consuming and fatiguing to the subjects. A balanced incomplete block design, requiring multiple rank order judgments for each subject, provides an efficient experimental method for obtaining paired comparisons judgments. Features of the analysis proposed for this design are discussed in detail. A program for the analysis is available for the IBM 650 electronic computer.Prepared in connection with research done under Office of Naval Research Contract Nonr 1858-(15), Project Designation NR 150-088, and National Science Foundation Grant G-3407. Reproduction of any part of this material is permitted for any purpose of the United States Government.     

Automated assembly of optimally spaced and balanced paired comparisons: Controlling order effects

To control order effects in questionnaires containing paired comparisons, Ross (1934) described an optimal ordering of the pairings. The pairs can also be balanced so that every stimulus appears equal numbers of times as the first and the second member of a pair. First, we describe and illustrate the optimally spaced, balanced ordering of pairings. Then we show how the optimally spaced, balanced order can be used to implement a matrix-sampling design or a fully incomplete design when the number of stimuli n is so large that respondents cannot reasonably be expected to judge all n(n - 1)/2 pairs. The algorithm for balancing and optimally spacing the list of pairs is described.     

A covariance analysis of multiple paired comparisons

In the method of multiple paired comparisons the dominance of objectj over objectk is observed uponp attributes. The present paper develops a covariance analysis for these paired comparisons in terms of a linear model which includes scale, bias, and interaction effects, along withs covariants upon which the comparisons are presumably dependent. The covariance model gives rise to adjusted parameter estimates and hypothesis tests for the residual pairwise layout from which the effects of thes covariants have been removed. These estimation and testing procedures are illustrated with an analysis of political judgment data, and their relevance to the general problem of residual scaling is discussed.This research was supported by National Institute of Mental Health grants MH 12972 and MH 15506. The author would like to express his appreciation to Jerry Solomon and Wei-Ching Chang of the Oregon Research Institute for the data analysis reported at the end of the paper. Computing assistance was obtained from the Health Sciences Computing Facility, UCLA, sponsored by N.I.H. Grant FR-3.     

Optimal scaling of paired comparison and rank order data: An alternative to guttman's formulation

A formulation, which is different from Guttman's is presented. The two formulations are both called the optimal scaling approach, and are proven to provide identical scale values. The proposed formulation has at least two advantages over Guttman's. Namely, (i) the former serves to clarify close relations of the optimal scaling approach to those of Slater and the vector model of preferential choice, and (ii) in addition to the stimulus scale values, it provides scores for the subjects, which indicate the degrees of response consistency (transitivity), relative to the optimum solution. The method is assumption-free and capable of multidimensional analysis.This study was partly supported by the National Research Council Grant (No. A4581) to S. Nishisato. The author is indebted to Dr. Bert F. Green, Jr., Mr. Tomoichi Ishizuka, and anonymous reviewers for their valuable comments on an earlier draft.     

A stochastic model for paired comparisons of social stimuli

A stochastic model for paired comparisons of multiattribute social stimuli is proposed where one objective is to find the relative importance of the attributes for a judge. The model can be conceived as a special strict binary utility model, i.e., a BTL-model, and is related to of the stimuli are linear combinations of functions of the attributes of the stimuli. The model neither assumes that the functions are fixed in advance nor that different judges have the same set of functions. The choice among such functions, however, is admitted only within a finite scope. Within the framework of exponential families, maximum likelihood estimators and tests are derived and applied to data coming from two psychological experiments.     

Successive intervals scaling of paired comparisons

In this paper a method is developed for analyzing data resulting from the use of a modification of the usual paired comparisons procedure which allows for ratings of size of difference. The scaling model is essentially an extension of the Thurstonian successive intervals model. The method is applied to scaling of a set of nine immoral actions and the results agree rather well with those from conventional successive intervals scaling of the same stimuli.This study was supported by a grant from the Swedish Council for Social Science Research.     

Modelling the reliability of paired comparisons

This paper considers paired comparison experiments for which weights describing the reliability of the comparisons are available. Such a weight can be represented by a belief function as defined by G. Shafer in A Mathematical Theory of Evidence (Princeton, NJ: Princeton Univ. Press, 1976). We develop a belief function analysis for this case, and discuss the interpretation of such an analysis and the conditions for its validity.     

A least squares solution for paired comparisons with incomplete data

A precise and rapid procedure has been devised for dealing with a matrix of incomplete data in paired comparisons. This method should increase the general applicability of paired comparisons since experiments involving large numbers of stimuli may now be shortened to feasible experimental proportions. Also, we may now use sets of stimuli which cover a wide range, resulting in a considerable number of 100 per cent vs. 0 per cent judgments, and still give a precise solution depending equally on each of the observations.This study was supported in part by the Office of Naval Research Contract N6onr-270-20 with Princeton University and by the National Science Foundation Grant G-642. The opinions expressed are, of course, those of the author and do not represent attitudes or policies of the Office of Naval Research or of the National Science Foundation.The author wishes to acknowledge helpful suggestions and comments received from Frederic M. Lord, Warren S. Torgerson, and Ledyard R Tucker. Thanks are also due to Mrs. Gertrude Diederich for some of the tabulating and computing for the illustrative problem.     

A note on the use of triads for paired comparisons

When scaling a large number of stimuli from comparative judgments, considerable savings in time and labor may be realized if stimuli are presented in triad form rather than in pairs. If, forN stimuli, the proper configuration of triads can be assembled so that all possible pairs appear once, the paired judgment matrix may be reproduced with one-third fewer judgments and two-thirds fewer presentations than would be required with complete pairing. A simple procedure is described for enumerating triad configurations for whichN is an odd multiple of three.     

A rational origin obtained by the method of contingent paired comparisons

A new paired comparison method, based upon choices between lotteries, is developed for the measurement of utilities of objects with respect to the utility of receiving nothing, i.e., the status quo. The method is used to estimate the utilities of four birthday gifts. These objects had also been studied in an earlier experiment which used choices between single objects and pairs of objects to determine a rational origin. A comparison of the results of the two experiments indicates that both methods scale objects with respect to the same rational origin and unit of measurement.This research was supported under contract AF 49(638)-724 with the Air Force Office of Scientific Research and contract DA 19-129-QM-1045 with the Quartermaster Research and Engineering Command.     

A multilevel approach to the relationship between birth order and intelligence

Many studies show relationships between birth order and intelligence but use cross-sectional designs or manifest other threats to internal validity. Multilevel analyses with a control variable show that when these threats are removed, two major results emerge: (a) birth order has no significant influence on children's intelligence and (b) earlier reported birth order effects on intelligence are attributable to factors that vary between, not within, families. Analyses on 7- to 8 - and 13- to 14-year-old children from the National Longitudinal Survey of Youth support these conclusions. When hierarchical data structures, age variance of children, and within-family versus between-family variance sources are taken into account, previous research is seen in a new light.     

Remarks on the method of paired comparisons: III. A test of significance for paired comparisons when equal standard deviations and equal correlations are assumed

A test of goodness of fit is developed for Thurstone's method of paired comparisons, Case V. The test involves the computation of , wheren is the number of observations per pair, and and are the angles obtained by applying the inverse sine transformation to the fitted and the observed proportions respectively. The number of degrees of freedom is (k–1) (k–2)/2.This research was performed in the Laboratory of Social Relations under a grant made available to Harvard University by the RAND Corporation under the Department of the Air Force, Project RAND.     

A technique for the nonmetric analysis of paired comparisons of psychological intervals

A nonmetric coordinate adjustment technique is developed which determines scale values for objects whose interobject intervals (differences in subjective value) have been directly compared. In Monte Carlo simulations, the degree of metric determinancy of the scale values is shown to be quite high even when the amount of error is relatively high. This robustness under high-error conditions permitted the analysis of individual subject data in experiments on the direct comparison of loudness differences and loudness ratios where only one judgment per interval comparison was obtained per subject.This research was supported by a grant from the National Research Council of Canada.     

Covariation assessment in rank order data

Two experiments were conducted to investigate how individuals assess covariation with rank order data. In both studies, subjects were given sets of rank order data, each set consisting of ten items ranked on two characteristics, and were asked to estimate the degree of relationship for each set. Contrary to previous research, subjects' estimates of covariation in this task were quite sensitive to actual levels of correlation in the data and remained unaffected by simple variations in the way rank order data were presented. More importantly, it appeared that this sensitivity to covariation was due likely to the use of a simple heuristic referred to here as the total discrepancy heuristic. These findings are discussed in terms of the availability of simple heuristics in rank-ordered versus other types of data and the consequences of using such heuristics in decision-making contexts.     

The relationship between paired associate learning and phonological skills in normally developing readers

Seventy-five 6- to 11-year-old children were administered tests of phonological awareness, verbal short term memory (STM), and visual-verbal paired associate learning (PA learning) to investigate their relationship with word recognition and decoding skills. Phonological awareness was a stronger concurrent predictor of word recognition than verbal STM, and phonological awareness but not verbal STM was a predictor of learning in the PA learning task. Importantly, measures of phonological awareness and PA learning both accounted for independent variance in word reading, even when decoding skill was controlled. The results suggest that PA learning and phonological awareness tasks tap two separate mechanisms involved in learning to read. The results are discussed in relation to current theories of reading development.