IMPLICATIONS OF THE MULTIDIMENSIONAL NATURE OF JOB PERFORMANCE FOR THE VALIDITY OF SELECTION TESTS: MULTIVARIATE FRAMEWORKS FOR STUDYING TEST VALIDITY

Although most studies of criterion-related validity focus on univariate relationships, the complex and multidimensional nature of the performance construct and the widespread use of multiple selection devices argue in favor of multivariate frameworks for evaluating validity. Using a Monte Carlo simulation we estimated the validity of general cognitive ability tests and personality tests in predicting "job performance," where performance is conceptualized as a composite of multiple performance measures (i.e., individual job task performance and organizational citizenship behaviors). The validity of a selection battery varies substantially as a function of the relative weight given to both predictors and criteria; the 95% confidence interval for validities ranged from .20 to .78. The effective weights given to performance dimensions accounted for 34% of the variance in selection battery validities; depending on precisely how "performance" is defined, the same test battery can have relatively high or relatively low levels of validity. Our model suggests that the way an organization defines job performance is a source of true and important variability in validities, and that the validity of selection tests for predicting complex performance criteria may show considerably less generalizability that current metaanalysis of univariate validities would suggest.     

TOWARD NARROWING THE RESEARCH-PRACTICE GAP IN PERFORMANCE APPRAISAL

Recently, performance appraisal researchers have adopted a cognitive approach to analyzing judgment processes in performance appraisal. While this approach allows researchers to tap a wealth of knowledge applicable to the appraisal context, this line of research is likely to widen the already existing gap between research and practice. We argue that coordination of the talents of researchers and practitioners is essential for narrowing the gap. Specifically, we suggest researchers focus their research on the best methods of ensuring use of relevant and valid data in appraisal, given organizational constraints. We also suggest practitioners focus on determining observable and measurable aspects of performance, and thus, specify appropriate appraisal content. We also note that cognitive process research has promise for increasing raters' ability to judge accurately, but that this approach does not necessarily address the rater's willingness to provide accurate ratings.     

COST-BENEFIT CONSIDERATIONS IN CHOOSING AMONG CROSS-VALIDATION METHODS

There are two general methods of cross-validation: (a) empirical estimation, and (b) formula estimation. In choosing a specific cross-validation procedure, one should consider both costs (eg. inefficient use of available data in estimating regression parameters) and benefits (eg. accuracy in estimating population cross-validity). Empirical cross-validation methods involve significant costs, since they are typically laborious and wasteful of data, but under conditions represented in Monte Carlo studies, they are generally not more accurate than formula estimates. Consideration of costs and benefits suggests that empirical estimation methods are typically not worth the cost, except in a limited number of cases in which Monte Carlo sampling assumptions are not met in the derivation sample. Designs which use multiple samples to estimate the cross-validity of a single regression equation are clearly preferable to single-sample designs; the latter are never expected to be more accurate than formula estimates and thus are never worth the cost. Multi-equation designs are more accurate than single equation designs, but they appear to estimate the wrong parameter, and thus are difficult to interpret.     

MODELING THE EFFECTS OF BANDING IN PERSONNEL SELECTION

Selection outcomes under banding are affected by characteristics of the selection system and the applicant pool. This study examined the effects of eight parameters on the proportions hired from higher- and lower-scoring groups: (a) selection ratio; (b) reliability; (b) fixed vs. sliding bands; (d) top-down vs. random within-band selection; (e) preferential vs. nonpreferential selection; (f) mean differences; (g) standard deviation differences; and (h) proportion of applicants from the lower-scoring group. Simulation results were analyzed in a fully-crossed eight-way ANOVA. Higher-order interactions among selection system and applicant pool characteristics had virtually no effect on selection outcomes; the proportion of the applicant pool from the lower-scoring group accounted for nearly half the variance in out-comes. Other important effects are, in order, the effects of standard deviation differences, mean differences, preferential hiring, and the selection ratio. Applicant pool characteristics have considerably more influence on selection outcomes than do selection system characteristics.