Exploring the Relationship Between Interrater Correlations and Validity of Peer Ratings

A field study was conducted to investigate the relationship between interrater correlations and validity estimates of peer ratings. Validity coefficients and interrater correlations were calculated for 281 work units in a large law enforcement organization in Israel. The main result was a weak positive linear relationship between these two variables. Furthermore, in some of the analyses conducted, a nonlinear quadratic component in the relationship between these measures was evident. Validity was low only when interrater correlation was very low (r = .4 and less). Above this level, validity was stable and almost did not change as interrater correlation increased. This finding, together with other studies (Borman, 1975 Borman, W. C. 1975. Effects of instructions to avoid halo error on reliability and validity of performance evaluation ratings. Journal of Applied Psychology, 60: 556–560. [Crossref], [Web of Science ®] , [Google Scholar]; Buckner, 1959 Buckner, D. N. 1959. The predictability of ratings as a function of interrater agreement. Journal of Applied Psychology, 43: 60–64.  [Google Scholar]; Freeberg, 1969 Freeberg, N. E. 1969. Relevance of rater-ratee acquaintance in the validity and reliability of ratings. Journal of Applied Psychology, 53: 518–524.  [Google Scholar]; Weekley & Gier, 1989 Weekley, J. A. and Gier, J. A. 1989. Ceilings in the reliability and validity of performance ratings: The case of expert raters. Academy of Management Review, 32: 213–222.  [Google Scholar]), cast doubt on the assertion that interrater correlation in the field of performance rating is a proper measurement of reliability.