BEHAVIORAL OBSERVATION SCALES AND THE EVALUATION OF PERFORMANCE APPRAISAL EFFECTIVENESS

An earlier article by the present authors (Bernardin and Kane, 1980) pointed out a serious flaw in the conceptual basis of Behavioral Observation Scales (BOS). The present article explains this flaw in more detail and shows that its solution would make BOS indistinguishable from other methods already in existence. The focus then shifts from the conception of appraisal methods to their evaluation with the presentation of a discussion of the special problems of error capitalization that arise in the use of item analysis in the development of multi-item rating scales. This discussion proceeds to describe the correct approach to removing the effects of error capitalization in the evaluation of the psychometric properties of such rating scales.     

THE EFFECT OF CORRELATED ARTIFACTS AND TRUE VALIDITY ON THE ACCURACY OF PARAMETER ESTIMATION IN VALIDITY GENERALIZATION

The effect of non-zero intercorrelations among the three artifacts (criterion reliability, predictor reliability, and range restriction on the predictor) and true validity on the accuracy of estimation in two validity generalization models (Model 1and Model 2) was investigated. Only the Thylor Series Approximation 1(TSA1) procedure from Model 1 and the procedure with complete sample-based artifact data from Model 2 were included in this study. Six intercorrelation conditions (zero, low positive, medium positive, low negative, low negative and positive, and medium negative and positive) and two distributions of artifacts were used in this investigation. The two major conclusions from this study are: (a) Both models yielded reasonably accurate estimates of the mean of true validities (M_ρ) and (b) only Model 2 produced accurate estimates of the variance of true validities (V_ρ). The need for additional research for accurately estimating V_ρ when the three artifacts and true validity are correlated is recommended.