VALIDATING SYNTHETIC VALIDATION: COMPARING TRADITIONAL AND SYNTHETIC VALIDITY COEFFICIENTS

We describe a unique application of a synthetic validation technique to a selection system development project in a large organization. Job analysis data were collected from 4,725 job incumbents and 619 supervisors, and were used to identify 11 job families and 27 job components. We developed 12 tests to predict performance on these job components and conducted a concurrent validation study to collect test and job component data for 1,926 incumbents. We created a test composite for each job component and then chose a test battery for each job family based on its relevant job components. Synthetic validity coefficients were computed for each test battery and compared to traditional validity coefficients that were computed within job families with large sample sizes. The synthetic validity coefficient was very close to the within-family validity coefficient for most job families and was within the bounds of sampling error for all job families. Validities tended to be highest when predictors were weighted according to the number of job components to which they were relevant and job component criterion measures were unit weighted.     

THE IMPACT OF JOB ANALYSIS ON EMPLOYMENT TEST VALIDATION FOR MINORITY AND NONMINORITY ACCOUNTING PERSONNEL

The purposes of the present study were (a) to examine the comparative validity of a written job knowledge test constructed on the basis of a systematic job analysis with that of a commercial employment test selected in the absence of a prior job analysis for accounting positions and (b) to determine the fairness of each test for minority and nonminority job applicants. Results indicated that the job knowledge test was a valid and unbiased predictor of relevant criteria of job performance while the commercial employment test produced adverse impact and lacked validity. Implications of the results for future research studies and test validation efforts involving differential prediction are discussed.     

PREDICTION OF TRADES APPRENTICES'PERFORMANCE ON JOB SAMPLE CRITERIA

A criterion-related validation was conducted to assess the validity of four aptitude tests and five tests of content taken directly from job tasks in predicting job sample performance of apprentices in eight skilled trades. Observed validities were above .40 (corrected for range restriction, validities averaged .52). Though there were large subgroup mean differences on both predictor and criterion measures, there was no evidence of significant differential prediction.     

SYNTHETIC VALIDITY: PAST, PRESENT, AND FUTURE

Practitioners in the area of personnel selection can be confronted with situations in which the use of traditional validation strategies is not feasible. This situation can result from small numbers of incumbents, rapidly changing jobs, insufficient resources, or the lack of high-quality criterion data. Although traditional strategies are not possible, a set of techniques falling under the rubric of synthetic validity can be used to estimate predictor validity in these situations. Synthetic validity is a logical process of inferring validity on the basis of the relationships between components of a job and tests of the attributes that are needed to perform the job components. Unfortunately, synthetic validity is infrequently used and not well understood. This article provides an overview and conceptualization of synthetic validity, reviews the historical approaches to synthetic validity, describes the recent developments and trends, and suggests future directions for synthetic validity application and research. The relationship of synthetic validity with other validity concepts, the advantages, the limitations, and the legal issues are considered throughout.     

Ethnic and Gender Differences in Applicants' Decision‐Making Processes: An Application of the Theory of Reasoned Action

Although a growing proportion of the new entrants into the workforce consist of women and ethnic minorities, relatively little is known about the recruitment and job choice processes of these applicant groups. Therefore, this study investigated cultural and gender differences in job application decision processes among 191 job seekers looking for temporary employment. The theory of reasoned action (TRA) was found to be a valid framework to explain job application decisions, although gender differences existed in the strength of the relations. Job attractiveness and perceived person–organization fit added to the prediction of job application intention over and beyond the TRA variables.     

EXPLORING OPTIONS FOR SUPPORTING TEST USE IN SITUATIONS PRECLUDING LOCAL VALIDATION

This paper examines some of the options available to practitioners interested in supporting the use of selection measures in an organization, including test transportability, validity generalization (VG), and synthetic validation, reviewing some of the advantages, disadvantages and requirements of each approach. Results of four unpublished, proprietary validation studies are reported which compare validity estimates provided by the job component validation (JCV; a type of synthetic validation) routine inherent in the Position Analysis Questionnaire (PAQ) with observed validity coefficients for a variety of criterion measures. We then examine the accuracy of the JCV procedure in predicting validity coefficients for 51 clerical classifications extracted from an existing PAQ job evaluation database of a large utility company. Predicted JCVs are compared to mean observed validity coefficients for five DOT clerical categories provided by Pearlman, Schmidt, and Hunter (1980). The VG and JCV methods provided highly similar and converging estimates of the validity of cognitive ability tests for predicting performance in clerical occupations. Implications for practice are discussed, particularly the need to use multiple, converging lines of evidence to support test use.     

COURT STANDARDS FOR JOB ANALYSIS IN TEST VALIDATION

Selected Federal court are reviewed and analyzed to determine the criteria used by the courts in their assessment of job analyses in the development and validation of selection tests. A set of standards which delineates the components and characteristics of a job analysis necessary to withstand legal scrutiny is presented. Implications are discussed.     

JOB-RELATEDNESS OF A POSTTRAINING JOB KNOWLEDGE CRITERION USE TO ASSESS VALIDITY AND TEST FAIRNESS

Significant job-relatedness was found for a posttraining job knowledge test criterion using an application of Lawshe's content validity method. The aide test was used as a criterion to assess the predictive validity of a vocabulary test and a civil service test with samples of black ( N = 43) and white ( N = 62) psychiatric aides. Significant validities were found on both tests, but a vocabulary test proved to be the better predictor of the criterion in both samples. The obtained validities were discussed in terms of differential validity, test fairness, and sample size. This study demonstrated that a content validity method could be applied to criteria as well as selection tests. It was concluded that content validity methods may be able to help solve the problem of criterion relevance in validation research by providing quantitative evidence of the job-relatedness of criteria.     

Investigating differential prediction by gender in employment-oriented personality measures

Gender-based differential prediction of job performance in employment-oriented personality measures has been left virtually unexamined. The use of 3 personality composites from the U.S. Army's instrument to predict 5 dimensions of job performance across 9 military jobs was investigated. Differential prediction, occurring in one third of the cases, was predominantly in the form of overprediction of female performances (i.e., higher male intercepts); slope differences were not found at above-chance levels. Female performance on the Effort and Leadership dimension was overpredicted in 90% of all predictor-criterion-job combinations, suggesting the measurement of this performance dimension as the source of the differential prediction rather than bias in the personality measures. Findings of overprediction of female performance parallel those of research investigating differential prediction by race in the ability domain.     

The Validity of Unstructured Panel Interviews: More than Meets the Eye?

When considered at the aggregate level, a weak level of validity was found for unstructured panel interviews in the prediction of the job performance and training success of corrections officers. Aggregate analyses also revealed that panel judgments made a small incremental contribution to the prediction of job performance relative to paper credentials and that the two sources of information were only weakly related. However, considerable variation in simple and incremental validity was found at the level of individual panel members and among subgroups of panels. The findings suggest that aggregate analyses underestimate the validity of the typical unstructured panel interview.     

The Structure of Life History: Implications for the Construct Validity of Background Data Scales

Background data measures have proven to be effective predictors of a variety of criteria. Little attention, however, has been given to the substantive princi- ples underlying their application. In this article, we present a model for un- derstanding the structure of differential life history. This model was used subsequently to generate a substantial framework for applying background data measures. These principles were used then to address various issues bear- ing on construct definition, item generation, and performance prediction. It was concluded that systematic application of construct validation principles may do much to enhance the utility of background data scales.     

Mind the Gap: Between‐group differences and fair test use

This paper integrates recent meta‐analytical findings regarding group differences in job‐ and educational‐related criteria and cognitive ability measures used as predictors in personnel selection and selection to higher education institutions. The findings suggest that cognitive ability measures reveal much higher group differences than the corresponding between‐group differences in job‐ and educational‐related criteria. One possible explanation for these differential gaps is that cognitive ability measures are objective and standardized while the typical measures used as job‐ and‐educational related criteria are non‐standardized subjective evaluations of job performance and academic achievement. While these findings are consistent with unbiased prediction or over‐prediction for lower scoring groups, they imply that selection is biased against them. Implications and future research are discussed.     

Out-of-bag Prediction Error: A Cross Validation Index for Generalized Structured Component Analysis

Cross validation is a useful way of comparing predictive generalizability of theoretically plausible a priori models in structural equation modeling (SEM). A number of overall or local cross validation indices have been proposed for existing factor-based and component-based approaches to SEM, including covariance structure analysis and partial least squares path modeling. However, there is no such cross validation index available for generalized structured component analysis (GSCA) which is another component-based approach. We thus propose a cross validation index for GSCA, called Out-of-bag Prediction Error (OPE), which estimates the expected prediction error of a model over replications of so-called in-bag and out-of-bag samples constructed through the implementation of the bootstrap method. The calculation of this index is well-suited to the estimation procedure of GSCA, which uses the bootstrap method to obtain the standard errors or confidence intervals of parameter estimates. We empirically evaluate the performance of the proposed index through the analyses of both simulated and real data.     

DEVELOPMENTS IN THE CRITERION‐RELATED VALIDATION OF SELECTION PROCEDURES: A CRITICAL REVIEW AND RECOMMENDATIONS FOR PRACTICE

The use of validated employee selection and promotion procedures is critical to workforce productivity and to the legal defensibility of the personnel decisions made on the basis of those procedures. Consequently, there have been numerous scholarly developments that have considerable implications for the appropriate conduct of criterion‐related validity studies. However, there is no single resource researchers can consult to understand how these developments impact practice. The purpose of this article is to summarize and critically review studies published primarily within the past 10 years that address issues pertinent to criterion‐related validation. Key topics include (a) validity coefficient correction procedures, (b) the evaluation of multiple predictors, (c) differential prediction analyses, (d) validation sample characteristics, and (e) criterion issues. In each section, we discuss key findings, critique and note limitations of the extant research, and offer conclusions and recommendations for the planning and conduct of criterion‐related studies. We conclude by discussing some important but neglected validation issues for which more research is needed.     

Development and validation of the Job Security Index and the Job Security Satisfaction scale: A classical test theory and IRT approach

This study describes the development and validation of two scales measuring job security: the Job Security Index (JSI), measuring an individual's cognitive appraisal of the future of his or her job with respect to the perceived level of stability and continuance of that job, and the Job Security Satisfaction (JSS) scale, measuring employee satisfaction with a perceived level of job security. Item response theory and classical test theory analyses indicate that the scales are highly reliable and exhibit good discriminant and criterion‐related validity. Future researchers are advised to apply confirmatory factor analysis (CFA) to the scales to confirm their factor structures.     

SYNTHETIC VALIDITY AND ITS APPLICATION TO THE UNIFORM GUIDELINES VALIDATION REQUIREMENTS

The Uniform Guidelines for Employee Selection Procedures have served to create an urgent need for efficient validation methods that can be generalized to a class of occupations. The one method currently authorized for such a purpose by the Guidelines is synthetic validation. (The Guidelines erroneously describe the synthetic validity paradigm as construct validity.) Approaches to synthetic validity employed by Lawshe, Guion, McCormick, and Primoff are described. Their extent of conformance to the Guidelines validation requirements is noted. Primoff's J -Coefficient approach is recommended for two reasons; it meets the Guidelines requirements and under certain circumstances it permits the test user to estimate the traditional validity coefficient. An illustrated example of Primoff's method is presented.     

A factorial approach to job families

This is an experimental study of the application of factor analysis to a new domain—the formation of job families. Correlations between jobs are computed from the formula based on the number of common elements between two variables and the job analyses provide the basic data on the presence or absence of the elements. A first-order general factor and four common factors are obtained in a small sample of twenty occupations. Tentative interpretations are made and implications for job analysis and the formation of job families are pointed out.The analysis the results of which are reported here was made possible by the Bureau of Psychological Services, Institute for Human Adjustment, Horace H. Rackham School of Graduate Studies, University of Michigan.     

Changes at work and employee reactions: organizational elements, job insecurity, and short-term stress as predictors for employee health and safety

The objective was to identify focus areas for possible reduction of job insecurity and its outcomes. A model was specified and tested as a prediction model for health and safety. First, a parsimonious model was specified. The model consisted of perceived job insecurity (as a stressor), organizational factors (information quality, leadership style, work task administration), and short-term stress reactions (job dissatisfaction, reduced work motivation). Second, the model was tested as a prediction model in three separate path analyses, in order to examine the model's contribution in explaining (1) physical health complaints, (2) mental health complaints, and (3) risk taking behavior. A quota sample of Norwegian employees ( N = 1,002) was obtained by means of a self-completion questionnaire survey. The results of the structural equation modeling (path analyses) supported the hypothesized model. Mental health complaints and employee risk taking behavior were significantly predicted (not physical health complaints).     

THE INCREMENTAL VALIDITY OF INTERVIEW SCORES OVER AND ABOVE COGNITIVE ABILITY AND CONSCIENTIOUSNESS SCORES

Recent research has suggested that scores on measures of cognitive ability, measures of Conscientiousness, and interview scores are positively correlated with job performance. There remains, however, a question of incremental validity: To what extent do interviews predict above and beyond cognitive ability and Conscientiousness? This question was addressed in this paper by (a) conducting meta‐analyses of the relationships among cognitive ability, Conscientiousness, and interviews, (b) combining these results with predictive validity results from previous meta‐analyses to form a “meta‐correlation matrix” representing the relationships among cognitive ability, Conscientiousness, interviews, and job performance, and (c) performing 9 hierarchical regressions to examine the incremental validity of 3 levels of structured interviews in best, actual, and worst case scenarios for prediction. Results suggested that interview scores contribute to the prediction of job performance over and above cognitive ability and Conscientiousness to the extent that they are structured, with scores from highly structured interviews contributing substantially to prediction. Directions for future research are discussed.     

Job demands, job resources and individual innovation at work: going beyond Karasek's model?

The job demands-control model is one of the most recognized models in occupational stress research. It has, however, provided contradictory results, and the active learning hypothesis derived from this model has been under-researched in comparison with research on the stress hypothesis. The main aim of this study is to test the Job Demands Resources Model in the prediction of individual innovation at work as an active coping strategy. Results with hierarchical multiple regression analyses provide empirical support for this model. We found a positive relationship between job demands and individual innovation in situations characterized by high job resources. Finally, we discuss the limitations and practical implications of this study.