效度概化：预测效度元分析的30年成果述评

效度概化是通过元分析技术对普遍化的预测效度的估计。元分析是研究效度概化问题的技术,是对具有“预测因子-效标”特征的相关数据进行定量综合的方法。效度概化促进了预测因子与效标之间关系的理论研究和应用研究,是应用心理学领域近30年（1977~2007）来最重要的进展之一。30年来的效度概化研究表明,认知能力测验、知识和技能测验、人格测验、结构化面试和评价中心技术等的预测效度具有鲁棒性、对应性和联合增值性     

EXPLORING OPTIONS FOR SUPPORTING TEST USE IN SITUATIONS PRECLUDING LOCAL VALIDATION

This paper examines some of the options available to practitioners interested in supporting the use of selection measures in an organization, including test transportability, validity generalization (VG), and synthetic validation, reviewing some of the advantages, disadvantages and requirements of each approach. Results of four unpublished, proprietary validation studies are reported which compare validity estimates provided by the job component validation (JCV; a type of synthetic validation) routine inherent in the Position Analysis Questionnaire (PAQ) with observed validity coefficients for a variety of criterion measures. We then examine the accuracy of the JCV procedure in predicting validity coefficients for 51 clerical classifications extracted from an existing PAQ job evaluation database of a large utility company. Predicted JCVs are compared to mean observed validity coefficients for five DOT clerical categories provided by Pearlman, Schmidt, and Hunter (1980). The VG and JCV methods provided highly similar and converging estimates of the validity of cognitive ability tests for predicting performance in clerical occupations. Implications for practice are discussed, particularly the need to use multiple, converging lines of evidence to support test use.     

Personality Testing in Personnel Selection: Adverse impact and differential hiring rates

Personality tests are often used in selection and have demonstrated predictive validity across a variety of occupational groups and performance criteria. Although different selection decision methods can be used to make selection decisions (e.g., compensatory top down, compensatory with sliding bands, noncompensatory) from personality test results, there is a paucity of research addressing the influence of these different selection decision methods on issues such as, adverse impact and differential hiring rates. This gap in the literature is redressed in the current study. Results from 398 bus operator candidates indicated that there may be adverse impact and differential hiring rate issues depending on the selection decision method used and the designated group being assessed. Implications and future research directions are discussed.     

A meta-analytic study of general mental ability validity for different occupations in the European community

A comprehensive meta-analysis of the validity of general mental ability (GMA) measures across 12 occupational categories in the European Community (EC) is presented. GMA measures showed that there is validity generalization and large operational validities for job performance and training success in 11 occupational groups. Results also showed that job complexity moderated the magnitude of the operational validity of GMA tests across three levels of job complexity: low, medium, and high. In general, results were similar to those found in the United States, although the European findings showed a slightly larger magnitude of operational validity in some cases. Theoretical and practical implications of these findings for personnel selection are discussed. ((c) 2003 APA, all rights reserved)     

INTERNATIONAL VALIDITY GENERALIZATION OF GMA AND COGNITIVE ABILITIES: A EUROPEAN COMMUNITY META-ANALYSIS

This article reports on a series of meta-analyses into the criterion validity of general menatl ability (GMA) and specific cognitive ability tests for predicting job performance ratings and training success in the European Community (EC). Meta-analyses were computed on a large EC database examining the operational validity of GMA and other specific cognitive abilities, including verbal, numerical, spatial-mechanical, perceptual and memory ( N ranged from 946 to 16,065) across 10 EC member countries. The results showed that tests of GMA and specific cognitive ability are very good predictors of job performance and training success across the EC. Evidence for the international validity generalization of GMA and specific cognitive abilities was presented. The results for the EC meta-analyses showed a larger operational validity than previous meta-analyses in the U.S. for predicting job performance. For training success, the European and American results are very similar. Implications for the international generalizability of GMA test validities, practical use of cognitive ability tests for personnel selection, and directions for future research are discussed.     

Selling Oneself: Construct and Criterion-Related Validity of Impression Management in Structured Interviews

Interviewee impression management has been a long-standing concern in the interview literature. Yet recent insights into the impact of impression management on interviewee performance in structured interviews suggest that interviewee impression management may be more than just a source of bias and a nuisance. Rather, impression management should possess construct-related validity and contribute to the interviews' criterion-related validity. These hypotheses were tested with 129 participants using a simulated selection interview aimed at university graduates. Results confirmed most of the hypotheses. In particular, interviewee impression management behavior showed construct-related validity across different structured interview types and correlated positively with interviewees' performance on subsequent typical and maximum performance proxy criteria. Implications and directions for future research are discussed.     

Validity generalization: Some critical remarks on the Schmidt-Hunter procedure*

This article presents some critical comments on the validity generalization procedure which has been presented by Schmidt, Hunter and others. They have put forward a method for testing the hypothesis that the variance in validity coefficients across situations for job-test combinations is due to what they consider to be statistical artifacts. The Schmidt-Hunter approach is criticized on the following points: the compilation of validity data, the use of criterion measures, and the test of the hypothesis of no situational specificity. Further, the relation between the concepts ‘situational specificity’ and ‘validity generalization’ is considered. In addition, it is noted that Schmidt, Hunter and others have defined the concept ‘situation’ in a different way than classical writers. It is concluded that the Schmidt-Hunter approach to validity generalization shows fundamental shortcomings. As a consequence their far-reaching conclusions for the practice of personnel selection should be considered premature.     

The Within‐Setting Variability of Validity in Cognitive Ability Tests

This study was directed to check the second prediction of the situational specificity hypothesis (SSH). According to this prediction, if the setting is held constant, then validity coefficients will not vary from study to study. This prediction was tested using data from a series of studies in which cognitive ability tests were used. These tests are subjected to a cultural bias that could affect the validity variability. The results are contrary to the second prediction of the SSH because a great variability was found in the validity coefficients within‐setting. However, the findings are consistent with the hypothesis of validity generalization (VGH) that predicts similar variability in both across settings and within‐setting. Also, the cultural context does not seem to affect the variability of validity and this variability is similar to that observed in previous studies of validity generalization.     

IDENTIFYING OPTIMAL PREDICTOR COMPOSITES AND TESTING FOR GENERALIZABILITY ACROSS JOBS AND PERFORMANCE FACTORS

The initial examination of validity generalization in the Army Selection and Classification Project used data from a concurrent validation sample of 4,039 job incumbents drawn from a representative sample of nine jobs. The available data consisted of 24 predictor scores and five job performance factor scores on each individual. The major objectives were to determine (a) the degree of validity generalization across the major components of performance, with the job held constant, and (b) the degree of validity generalization across jobs within each major performance factor. After reducing the predictor set by eliminating variables that added no information, a modified confirmatory analysis was used to test the hypotheses that one equation would fit the data from all performance components and that one equation would fit the data from all jobs, given a particular performance component. The major findings were that different predictor equations were needed for each of the five criterion factors. For generalization across jobs, within each criterion factor, one equation fit the data for four of the five performance components. Different prediction equations were required for the component that reflects proficiency on the technical tasks specific to each job.     

Cognitive and GMA Testing in the European Community: Issues and Evidence

This article reviews the evidence on General Mental Ability (GMA) and cognitive ability tests in connection with employment in the European Community (EC). Five themes are reviewed: prevalence, applicant reactions, testing standards, criterion validity, and recent advances. The first section shows that GMA and cognitive ability tests are used more often in Europe than in America. The second section, regarding applicant reactions, shows that the studies carried out in the EC have shown that there are no differences between Europe and America. The third section shows that there is an initiative to harmonize the standards used across the European countries. In the fourth section, we report on a validity generalization study carried out using the primary studies conducted in Great Britain and Spain. The results showed that GMA and cognitive tests are valid predictors of job performance and training success in both countries and they generalize validity across occupations. Furthermore, the size of the observed validity estimates is similar to those found in the American meta-analyses. Finally, two recent advances are mentioned: the British "Project B" and the use of virtual reality technology for predicting job performance.     

Work Sample Testing

The general evidence supporting the use of work sample tests (good criterion — related validity, positive applicant reaction and job preview capability) is reviewed, limitations are also noted. The extent to which work sample tests may be used to limit adverse impact in selection is reviewed and the extent to which work sample tests may be used to complement other selection instruments, in particular, cognitive ability tests, is discussed. Work sample tests are based on attempts to embody key tasks within the job in a selection instrument. Five distinctive (though not mutually exclusive) types of work sample tests are identified and validity evidence is reviewed. Key dimensions (bandwidth, fidelity, task specificity, necessary experience, type of tasks and mode of presentation and response) are identified and used to compare different types of work sample tests.     

A WITHIN SETTING EMPIRICAL TEST OF THE SITUATIONAL SPECIFICITY HYPOTHESIS IN PERSONNEL SELECTION

The situational specificity hypothesis of selection procedure validity makes two predictions. The first is that variation in observed validities across settings is caused by real differences in what constitutes job performance. Validity generalization studies to date have provided disconfirming evidence for this prediction by showing that the observed variation is due to statistical and measurement artifacts. The second prediction is that if the situation (i.e., the organization, the setting, the job, the test and the criterion) is held constant, then validity findings will not vary from study to study. This article tests this prediction empirically using data from a unique series of studies reported by Bender and Loveless (1958). The results are contrary to the situational specificity hypothesis and consistent with the hypothesis that variation in observed validities is due to statistical and measurement artifacts that are unrelated to situations and settings. It is concluded that both major predictions of the situational specificity hypothesis have now been empirically disconfirmed.     

FURTHER TESTS OF THE SCHMIDT-HUNTER BAYESIAN VALIDITY GENERALIZATION PROCEDURE1

Using an improved version of the Bayesian validity generalization model presented in Schmidt and Hunter (1977), this study showed that most of the between-study variation in observed validity coefficients of various aptitude and ability tests for two types of clerical work and for the job of first-line supervisor is artifactual in nature. These results cast additional doubt on the traditional belief that employment test validities are situationally specific. In addition, the results showed that generalization of validities to new settings was justified even where the hypothesis of situational specificity could not be rejected under the standards used in this study. Implications for the use of aptitude and ability tests in selection and for theory development in industrial-organizational psychology are discussed.     

Maintaining responding during stimulus generalization testing in extinction

Resistance to extinction and generalization gradients were studied following training with a long-adjusting-interval schedule. One large reinforcer occurred at the end of each daily training session. Sessions varied in length from 20 sec to 42.66 min, but were usually the latter. Repeated generalization tests were subsequently conducted for these subjects and subjects trained with a more conventional short-random-interval schedule. The long-adjusting-interval schedule produced generalization gradients that were not qualitatively different from those produced by the conventional procedure. However, the advantages of the long-adjusting-interval schedule are: (1) greater resistance to extinction both within and across generalization tests and (2) more stable gradient slopes within and across tests.     

Meta‐Analysis of Clerical Performance Predictors: Still stable after all these years

Pearlman, Schmidt, and Hunter (1980) published a seminal paper on the validity of various measures for predicting performance in clerical jobs. They concluded that for both job and training performance criteria, 10 types of tests (e.g., perceptual speed, clerical aptitude, verbal ability) predicted performance across 5 clerical job families. This paper describes a psychometric meta‐analysis of validity studies using similar measures for clerical jobs conducted since 1980 to examine the stability of Pearlman and colleagues' validity estimates. This paper contributes to the literature by investigating the long‐term stability of validity generalization estimates over a period of several decades. Clerical jobs provide a compelling case study because these jobs have changed considerably due to the increased use of computers and technology in the office environment. Results showed that the mean population estimates in the present study were consistent with, or higher than, those obtained by Pearlman and colleagues, thus demonstrating the long‐term stability of meta‐analytic estimates of validity. The relative stability of the validity estimates also supports the robustness of g as a predictor, even as jobs change over time.     

The Validity of Job Knowledge Measures

The validity of written job knowledge tests as predictors of job performance and training success was examined. Based on a sample of 363,528 persons and 502 coefficients, meta-analyses were performed to investigate the extent of validity generalization and the effects of two hypothesized moderator variables: content similarity and job complexity. Corrected mean validities were 0.45 for studies predicting job performance and 0.47 for studies of training success. Support was found for both moderators since validities were higher for high complexity jobs and when job-test content similarity was high. The implications of this study for personnel selection and research in the area of work performance are discussed.     

Understanding how and why adding valid predictors can decrease the validity of selection composites: A generalization of Sackett,Dahlke, Shewach,and Kuncel (2017)

It is usually assumed that adding more valid predictors will increase the predictive power of a selection test battery. Sackett, Dahlke, Shewach, and Kuncel showed that when selection tests are combined using unit weights, adding a valid predictor can lead to a decrease in validity. Situating the Sackett et al. approach in a more general multivariate framework I show how: (a) it is the tradeoff between predictor validity and predictor intercorrelations, and not the differences in predictor validities that determines whether adding a valid predictor to a composite will cause the validity of that composite to increase or decrease; and (b) this same dynamic applies across a wide range of non‐optimal schemes for weighting predictors and/or criteria.     

Some Comments on Pareto Thinking,Test Validity,and Adverse Impact: When ‘and’ is optimal and ‘or’ is a trade‐off

De Corte, Lievens, and Sackett add to the literature on selection test validity and adverse impact (AI). Their Pareto‐based weighting scheme essentially asks organizations if they are willing to give up some validity to hopefully achieve some reduction in AI. We considered their approach and conclusions in relation to the regression weighting method we used, and we offer five points that reflect our observations as well as our shared goals. We hope our comments, like their work in this field, will invigorate the pursuit of new ways of examining, and one day resolving, the persistent concern regarding the AI associated with valid selection tests.     

Renewed behavior produced by context change and its implications for treatment maintenance: A review

Behavioral treatment gains established in one setting do not always maintain in other settings. The present review examines the relevance of basic and translational research to understanding failures to maintain treatment gains across settings. Specifically, studies of the renewal effect examine how transitioning away from a treatment setting could evoke a return of undesirable behavior, rather than newly trained appropriate behavior. Studies of renewal typically arrange three phases, with a response trained and reinforced under a particular set of contextual stimuli in the first phase. Next, that response is extinguished, often under a different set of contextual stimuli. Finally, that response returns despite extinction remaining in effect upon returning to the original training context or transitioning to a novel context. Thus, removing the extinction context is sufficient to produce a recurrence of the response. The findings suggest treatment effects can become specific to the context in which the treatment was delivered. This literature offers promising methods for systematically assessing the factors contributing to treatment maintenance and improving generalization of treatment gains across contexts. Therefore, the present review suggests basic and translational research on renewal provides an empirical literature to bring greater conceptual systematization to understanding generalization and maintenance of behavioral treatment.