A simple computational alternative to analysis of variance formulas for estimating the k-judge reliability

Formulas requiring the computation of only three standard deviations are presented for computing the interjudge reliability coefficient for any number of judges. These formulas yield coefficients identical to those obtained from a one-way repeated-measures analysis of variance. Even researchers with small handheld calculators can use this simple approach.     

Some thoughts regarding the sensitivity of t to error in estimated within-group variance

A. J. Riopelle (2003) has eloquently demonstrated that the null hypothesis assessed by the t test involves not only mean differences but also error in the estimation of the within-group standard deviation, s. He is correct in his conclusion that the precision of the interpretation of a significant t and the null hypothesis tested is complex, particularly when sample sizes are small. In this article, the author expands on Riopelle's thoughts by comparing t with some equivalent or closely related tests that make the reliance of t on the accurate estimation of error perhaps more salient and by providing a simulation that may address more directly the magnitude of the interpretational problem.     

Test reliability and practice effects

The purpose of this paper is to develop mathematical formulations of test reliability when the estimation of reliability requires successive observations and these observations are not independent of the process (or act) of measurement. The specific application of concern arises in performance testing where performance varies as a function of practice and where practice accrues as a result of measurement. Repeated measures are experimentally nonindependent in such cases. Formulations for the estimation of reliability are developed for each of two cases. The first of these assumes a linear model for acquisition; the second assumes a nonlinear (quadratic) model. Illustrative data are presented whereby insights may be gained as to the inadequacy of the models.This investigation was supported by research grant (G14437) from National Science Foundation to Oregon Research Institute.The analysis of much of the data referred to in this report was made possible through the facilities of the Western Data Processing Center at the University of California, Los Angeles.     

Regression analysis of linear composite variance

The problem of defining and determining the effective contribution of a component variable to the variance of a composite is briefly reviewed. Another method of dealing with this problem is proposed and illustrated with analysis of a three-component problem. The proposed method is based on systematic application of a series of regressions and is restricted in application to positive manifold systems.This study was performed at Personnel Laboratory, WADD, Lackland AFB, Texas, under ARDC Project 7717, Task 87003. The opinions expressed in this paper are those of the authors and do not necessarily represent the official policy of either the National Academy of Sciences or of the USAF.     

Test reliability and correction for attenuation

Evidence is cited to show that specificity, or lack of equivalence, in the comparable forms of tests has a tendency to lower the value of reliability coefficients but has no tendency to lower the value of observed trait coefficients. This implies that the greater the lack of equivalence, the higher will be coefficients corrected for attenuation. Errors of measurement are supposed to reduce the magnitude of observed trait coefficients. Since specificity does not lower the correlation between two tests and since the split-half and equivalent-form reliability coefficients treat specificity as error, it follows that these two coefficients cannot legitimately be used in Spearman's correction-for-attenuation formula.     

Test reliability and effective test length

Measures of effective test length are developed for speeded and power tests, which are independent of the number of items in the test or of the time required for administration. These measures are used in determining reliability for (1) speeded and power tests, where a separately timed short parallel form is administered in addition to the full-length test; (2) power tests, where a subset of items is imbedded within the total test, parallel to the total test; and (3) power tests, where the subset of items is correlated with the complementary parallel subset in the test.     

The reliability of the Thematic Apperception Test

Controversy over the TAT's reliability may stem largely from the mis-application of traditional psychometric measures, which are inappropriate to this test. The TAT is implicitly based on a multiple regression model, for which coefficient alpha is not appropriate. Also, test-retest correlations may be adversely affected by the standard instructions to write a "creative" story. In a test-retest study, 47 high school students retook the TAT after a year with instructions designed to break the implicit set to produce a new and different story from that previously written. The test-retest correlations were r = .48 (need for affiliation) and .56 (need for intimacy), or approximately the same as those for, e.g., the MMPI, 16PF, and CPI, It was demonstrated that this high stability over time was not due to subjects' recalling and repeating previous responses. Finally, it was shown that alpha considerably underestimated the test-retest reliability, contrary to assumptions of classical psychometrics.     

The relation of item difficulty and inter-item correlation to test variance and reliability

Under assumptions that will hold for the usual test situation, it is proved that test reliability and variance increase (a) as the average inter-item correlation increases, and (b) as the variance of the item difficulty distribution decreases. As the average item variance increases, the test variance will increase, but the test reliability will not be affected. (It is noted that as the average item variance increases, the average item difficulty approaches .50). In this development, no account is taken of the effect of chance success, or the possible effect on student attitude of different item difficulty distributions. In order to maximize the reliability and variance of a test, the items should have high intercorrelations, all items should be of the same difficulty level, and the level should be as near to 50% as possible.The desirability of determining this relationship has been indicated by previous writers. Work on the present paper arose out of some problems raised by Dr. Herbert S. Conrad in connection with an analysis of aptitude tests.On leave for Government war research from the Psychology Department, University of Chicago.     

Correcting an analysis of variance for clustering

A great deal of educational and social data arises from cluster sampling designs where clusters involve schools, classrooms, or communities. A mistake that is sometimes encountered in the analysis of such data is to ignore the effect of clustering and analyse the data as if it were based on a simple random sample. This typically leads to an overstatement of the precision of results and too liberal conclusions about precision and statistical significance of mean differences. This paper gives simple corrections to the test statistics that would be computed in an analysis of variance if clustering were (incorrectly) ignored. The corrections are multiplicative factors depending on the total sample size, the cluster size, and the intraclass correlation structure. For example, the corrected F statistic has Fisher's F distribution with reduced degrees of freedom. The corrected statistic reduces to the F statistic computed by ignoring clustering when the intraclass correlations are zero. It reduces to the F statistic computed using cluster means when the intraclass correlations are unity, and it is in between otherwise. A similar adjustment to the usual statistic for testing a linear contrast among group means is described.     

A multipurpose four-dimensional analysis of variance program

A computer program is described for a four-dimensional analysis of variance using the IBM 360/40 OS, or larger, machine.     

The analysis of variance and pairwise scaling

Variance analyses are presented for two data layouts—each corresponding to the class of all ordered pairs from a single finite set. The analysis of the dominance layout is in terms of a fixed effects linear model which includes parameters representing the scale values of the elements of the set, response bias, and pairwise interactions. A parallel parametrization is carried out for the composition layout for which corresponding point estimates and hypothesis tests are given. A joint treatment of concurrently observed dominance and composition layouts is suggested and illustrative data are presented.This research was supported in part by National Institutes of Health Grant MH-04439-06. The author would like to express his appreciation to Richard Beatty of the University of Toronto, James Baker and William Carroll of Oregon Research Institute, and J. E. Keith Smith of the University of Michigan, for their helpful comments concerning aspects of this work. Computing assistance was obtained from the Health Sciences Computing Facility, UCLA, sponsored by NIH Grant FR-3.     

Unbiased estimators of ability parameters,of their variance,and of their parallel-forms reliability

Given known item parameters, unbiased estimators are derived i) for an examinee's ability parameter and for his proportion-correct true score, ii) for the variances of and across examinees in the group tested, and iii) for the parallel-forms reliability of the maximum likelihood estimator .This work was supported in part by contract N00014-80-C-0402, project designation NR 150-453 between the Office of Naval Research and Educational Testing Service. Reproduction in whole or in part is permitted for any purpose of the United States Government.     

The role of correlation in analysis of variance

A test of the significance of a row or column agent in an analysis of variance may be expressed in the form of correlation between the agent and the variate. A test of the significance of interaction variance may be expressed in the form of correlation between the agents. These expressions are principally of theoretical interest in that the degree of significance in anF test or the value of a correlation coefficient may be controlled at will, or inadvertently, within certain limits.This study was made possible by the Bureau of Psychological Services, Institute for Human Adjustment, Horace H. Rackham School of Graduate Studies, University of Michigan.     

Latin square analysis of variance using summary statistics

The method for computing a Latin square analysis of variance using summary statistics is presented.