The scaling of test scores by the method of paired comparisons

The purpose of this study is to investigate, by the method of paired comparisons, a possible scaling of individuals who have made certain test scores, such that the additive property will be satisfied and such that a stability in scaling will be maintained,—in other words, a scaling such that the scaled score of an individual will remain relatively the same regardless of the grouping of individuals in which he may be placed. The results show that it is possible to utilize psychophysical methods in psychological and educational test situations. Among the major findings are that Case V of the Law of Comparative Judgment is applicable to the data in this problem, the method of dividing the intermediate category equally between the greater and the less was the best of three possible methods, internal consistency was satisfied, and, finally, when a new test of stability was applied, it was found that the distances between the hypothetical individuals remain the same.The writer wishes to express appreciation for the invaluable help and guidance of Professor Harold O. Gulliksen and also to Professor Marion W. Richardson who suggested the problem and made valuable suggestions.     

A technique for the nonmetric analysis of paired comparisons of psychological intervals

A nonmetric coordinate adjustment technique is developed which determines scale values for objects whose interobject intervals (differences in subjective value) have been directly compared. In Monte Carlo simulations, the degree of metric determinancy of the scale values is shown to be quite high even when the amount of error is relatively high. This robustness under high-error conditions permitted the analysis of individual subject data in experiments on the direct comparison of loudness differences and loudness ratios where only one judgment per interval comparison was obtained per subject.This research was supported by a grant from the National Research Council of Canada.     

Modelling the reliability of paired comparisons

This paper considers paired comparison experiments for which weights describing the reliability of the comparisons are available. Such a weight can be represented by a belief function as defined by G. Shafer in A Mathematical Theory of Evidence (Princeton, NJ: Princeton Univ. Press, 1976). We develop a belief function analysis for this case, and discuss the interpretation of such an analysis and the conditions for its validity.     

A covariance analysis of multiple paired comparisons

In the method of multiple paired comparisons the dominance of objectj over objectk is observed uponp attributes. The present paper develops a covariance analysis for these paired comparisons in terms of a linear model which includes scale, bias, and interaction effects, along withs covariants upon which the comparisons are presumably dependent. The covariance model gives rise to adjusted parameter estimates and hypothesis tests for the residual pairwise layout from which the effects of thes covariants have been removed. These estimation and testing procedures are illustrated with an analysis of political judgment data, and their relevance to the general problem of residual scaling is discussed.This research was supported by National Institute of Mental Health grants MH 12972 and MH 15506. The author would like to express his appreciation to Jerry Solomon and Wei-Ching Chang of the Oregon Research Institute for the data analysis reported at the end of the paper. Computing assistance was obtained from the Health Sciences Computing Facility, UCLA, sponsored by N.I.H. Grant FR-3.     

A dual scaling analysis for paired compositions

A paired composition is a response (upon a dependent variable) to the ordered pair <j, k> of stimuli, treatments, etc. The present paper develops an alternative analysis for the paired compositions layout previously treated by Bechtel's [1967] scaling model. The alternative model relaxes the previous one by including row and column scales that provide an expression of bias foreach pair of objects. The parameter estimation and hypothesis testing procedures for this model are illustrated by means of a small group analysis, which represents a new approach to pairwise sociometrics and personality assessment.This study was supported by Grant Nos. MH 12972, MH 10822, and MH 15506 from the National Institute of Mental Health, U. S. Public Health Service. Computing assistance was obtained from the Health Sciences Computing Facility, UCLA, sponsored by N.I.H. special research resources Grant RR-3.The motivation of this effort has been enhanced by Gerald Patterson and his associates, who have kindly provided the illustrative data at the end of the paper. The author would also like to express his appreciation to Wei-Ching Chang of the Oregon Research Institute and the University of Oregon for his substantive contributions to the paper, and to William Chaplin and Mark Layman of the Oregon Research Institute for the programming of the data analytic method. FORTRAN IV program listings for this analysis are available upon request to the author.     

A stochastic model for paired comparisons of social stimuli

A stochastic model for paired comparisons of multiattribute social stimuli is proposed where one objective is to find the relative importance of the attributes for a judge. The model can be conceived as a special strict binary utility model, i.e., a BTL-model, and is related to of the stimuli are linear combinations of functions of the attributes of the stimuli. The model neither assumes that the functions are fixed in advance nor that different judges have the same set of functions. The choice among such functions, however, is admitted only within a finite scope. Within the framework of exponential families, maximum likelihood estimators and tests are derived and applied to data coming from two psychological experiments.     

A note on the use of triads for paired comparisons

When scaling a large number of stimuli from comparative judgments, considerable savings in time and labor may be realized if stimuli are presented in triad form rather than in pairs. If, forN stimuli, the proper configuration of triads can be assembled so that all possible pairs appear once, the paired judgment matrix may be reproduced with one-third fewer judgments and two-thirds fewer presentations than would be required with complete pairing. A simple procedure is described for enumerating triad configurations for whichN is an odd multiple of three.     

An axiomatic formulation and generalization of successive intervals scaling

A formal set of axioms is presented for the method of successive intervals, and directly testable consequences of the scaling assumptions are derived. Then by a systematic modification of basic axioms the scaling model is generalized to non-normal stimulus distributions of both specified and unspecified form.     

Determining the degree of inconsistency in a set of paired comparisons

Consistency in paired comparison data is defined. Two types of inconsistency which may arise are defined. Computational formulas for these types of inconsistency are derived, and examples illustrating the use of these formulas are presented.These ideas were developed while the first author was on the staff of the Research Center for Human Relations. The work was made possible by the ONR contract NONR 285(10). The authors are indebted to Jack Moshman for his helpful critical suggestions. The United States Government is authorized to reprint this article in whole or in part.     

A least squares solution for paired comparisons with incomplete data

A precise and rapid procedure has been devised for dealing with a matrix of incomplete data in paired comparisons. This method should increase the general applicability of paired comparisons since experiments involving large numbers of stimuli may now be shortened to feasible experimental proportions. Also, we may now use sets of stimuli which cover a wide range, resulting in a considerable number of 100 per cent vs. 0 per cent judgments, and still give a precise solution depending equally on each of the observations.This study was supported in part by the Office of Naval Research Contract N6onr-270-20 with Princeton University and by the National Science Foundation Grant G-642. The opinions expressed are, of course, those of the author and do not represent attitudes or policies of the Office of Naval Research or of the National Science Foundation.The author wishes to acknowledge helpful suggestions and comments received from Frederic M. Lord, Warren S. Torgerson, and Ledyard R Tucker. Thanks are also due to Mrs. Gertrude Diederich for some of the tabulating and computing for the illustrative problem.     

Remarks on the method of paired comparisons: III. A test of significance for paired comparisons when equal standard deviations and equal correlations are assumed

A test of goodness of fit is developed for Thurstone's method of paired comparisons, Case V. The test involves the computation of , wheren is the number of observations per pair, and and are the angles obtained by applying the inverse sine transformation to the fitted and the observed proportions respectively. The number of degrees of freedom is (k–1) (k–2)/2.This research was performed in the Laboratory of Social Relations under a grant made available to Harvard University by the RAND Corporation under the Department of the Air Force, Project RAND.     

The measurement of loudness using direct comparisons of sensory intervals

Subjects were required in each trial to directly compare two pairs of tones and indicate which pair of tones had the greater loudness difference. Ten 1200 Hz tones differing only in intensity were employed. Subjects made binary comparisons among the 45 tone pairs which can be formed from the set of ten tones. The subjects' binary comparisons of the tone pairs were found to satisfy the transitivity and monotonicity requirements of a positive difference structure. These comparisons of loudness intervals were used to construct a rank order of loudness difference. A loudness scale was constructed from a nonmetric analysis of the rank order of loudness difference for the 45 tone pairs and indicated that loudness was a power function of sound pressure with an exponent of 0.26.     

A rational origin obtained by the method of contingent paired comparisons

A new paired comparison method, based upon choices between lotteries, is developed for the measurement of utilities of objects with respect to the utility of receiving nothing, i.e., the status quo. The method is used to estimate the utilities of four birthday gifts. These objects had also been studied in an earlier experiment which used choices between single objects and pairs of objects to determine a rational origin. A comparison of the results of the two experiments indicates that both methods scale objects with respect to the same rational origin and unit of measurement.This research was supported under contract AF 49(638)-724 with the Air Force Office of Scientific Research and contract DA 19-129-QM-1045 with the Quartermaster Research and Engineering Command.     

The validity of the successive intervals method of psychometric scaling

The degree to which scale values computed by the method of successive intervals diverge from theoretically true values is seen to be due to three types of error: error due to inequalities in variances of the distributions from which the scale values are computed, error due to non-normality of the distributions, and sampling error. The contribution of each type of error to the total error is evaluated; the latter is seen to be surprisingly small under appropriate conditions. Certain aspects of the formal methodology underlying scaling procedures are also briefly considered.This paper reports research undertaken in cooperation with the Quartermaster Food and Container Institute for the Armed Forces, and has been assigned number 475 in the series of papers approved for publication. The views or conclusions contained in this report are those of the authors. They are not to be construed as necessarily reflecting the views or indorsement of the Department of Defense.     

Individual loudness functions determined from direct comparisons of loudness intervals

Five subjects were required in each trial to directly compare two pairs of tones and indicate which pair of tones had the greater loudness difference. Ten 1,200-Hz tones differing only in intensity were employed. Subjects made binary comparisons among the 45 tone pairs that can be formed from these 10 tones. The loudness difference comparisons of each subject were found to satisfy two properties (transitivity and monotonicity) that are required for an interval scale representation of loudness. Therefore, individual loudness scales were constructed using a nonmetric scaling technique designed for comparisons of sensory intervals. These loudness scales differed significantly from subject to subject. Since a nonnumerical scaling procedure was employed, these individual differences could not be attributed to biases in the way in which observers use numbers or numerical concepts to describe the loudness of tones. Hence, they suggest strong individual differences in the coding of sound intensity.