Predictor content matters for knowledge testing: Evidence supporting content validation

Organizations often rely on the match between job requirements and test content to justify test use. This practice has been questioned on the grounds that content validation has little relevance to criterion-related validation due to positive manifold among predictors. We analyze two large databases to assess the implications of test content for (a) test interchangeability and (b) criterion-related validity. Analyses of 15 knowledge tests administered (N = 80,394) as part of Project Talent demonstrate that test content is related to predictor interchangeability. Analyses of SAT and Advanced Placement test data compare correlations among predictors and criteria drawn from matched and unmatched content domains. We conclude that test-criterion content match is likely to result in stronger criterion-related validity.     

Webcam testing: Validation of an innovative open-ended multimedia test

A modern test that takes advantage of the opportunities provided by advancements in computer technology is the multimedia test. The purpose of this study was to investigate the criterion-related validity of a specific open-ended multimedia test, namely a webcam test, by means of a concurrent validity study. In a webcam test a number of work-related situations are presented and participants have to respond as if these were real work situations. The responses are recorded with a webcam. The aim of the webcam test which we investigated is to measure the effectiveness of social work behaviour. This first field study on a webcam test was conducted in an employment agency in The Netherlands. The sample consisted of 188 consultants who participated in a certification process. For the webcam test, good interrater reliabilities and internal consistencies were found. The results showed the webcam test to be significantly correlated with job placement success. The webcam test scores were also found to be related to job knowledge. Hierarchical regression analysis demonstrated that the webcam test has incremental validity up to and beyond job knowledge in predicting job placement success. The webcam test, therefore, seems a promising type of instrument for personnel selection.     

Evaluation of video-based assessment in transit operator selection

The use of video testing is relatively new in personnel selection. Initial research has shown that the visual presentation of behaviorial incidents to job applicants may be a practical alternative to paper-and-pencil selection tests. Little research, however, has been conducted on the psychometric properties of video testing. To address this shortcoming in the personnel selection literature, this study reports the results of a validation study conducted on a video test for transit operators conducted in a large Canadian transit authority. The test, the Metro Seattle Video Test (MSVT), was designed to assess the interpersonal skills required of transit operators. The results show that, although content validation evidence was supportive, other psychometric evidence (i.e., reliability, criterion-related validation evidence, and construct-oriented validation evidence) is not consistent with an adequate selection test of interpersonal skills. Recommendations are made for future development and use of video testing in transit operator selection, as well as for improving the reliability and validity of video-based assessment of interpersonal skills.We gratefully acknowledge the assistance of Kevin Kelloway and Maury Getkate for their assistance with this paper, as well as Rick Hackett and Peter Hausdorf for their involvement in the data collection phase of this study.     

PREDICTION OF TRADES APPRENTICES'PERFORMANCE ON JOB SAMPLE CRITERIA

A criterion-related validation was conducted to assess the validity of four aptitude tests and five tests of content taken directly from job tasks in predicting job sample performance of apprentices in eight skilled trades. Observed validities were above .40 (corrected for range restriction, validities averaged .52). Though there were large subgroup mean differences on both predictor and criterion measures, there was no evidence of significant differential prediction.     

THE VALIDATION OF A MINICOURSE FOR TELEPHONE COMPANY SWITCHING TECHNICIANS1

The introduction of electronic switching equipment has changed the nature of the telephone company switching job. A lengthy and complex training program must be completed before an employee can perform the electronic switching job. Because of the high cost of this training a more elaborate, second-stage selection procedure was developed. The ESS Minicourse was designed to be a self-paced content valid sample of ESS training which would be suitable for use with job candidates without any previous telephone company experience. A criterion-related validity study was undertaken to provide further evidence of validity as well as data helpful in setting a cutting score. Results showed that a combination of time to complete the Minicourse and performance on the objective tests was predictive of time to complete self-paced training in electronic switching. Cross- validated estimates of validity were used to develop estimates of u'tility given different selection ratios.     

SITUATIONAL JUDGMENT TESTS: CONSTRUCTS ASSESSED AND A META-ANALYSIS OF THEIR CRITERION-RELATED VALIDITIES

Situational judgment tests (SJTs) are a measurement method that may be designed to assess a variety of constructs. Nevertheless, many studies fail to report the constructs measured by the situational judgment tests in the extant literature. Consequently, a construct-level focus in the situational judgment test literature is lacking, and researchers and practitioners know little about the specific constructs typically measured. Our objective was to extend the efforts of previous researchers (e.g., McDaniel, Hartman, Whetzel, & Grubb, 2007 ; McDaniel & Ngyuen, 2001 ; Schmitt & Chan, 2006 ) by highlighting the need for a construct focus in situational judgment test research. We identified and classified the construct domains assessed by situational judgment tests in the literature into a content-based typology. We then conducted a meta-analysis to determine the criterion-related validity of each construct domain and to test for moderators. We found that situational judgment tests most often assess leadership and interpersonal skills and those situational judgment tests measuring teamwork skills and leadership have relatively high validities for overall job performance. Although based on a small number of studies, we found evidence that (a) matching the predictor constructs with criterion facets improved criterion-related validity; and (b) video-based situational judgment tests tended to have stronger criterion-related validity than pencil-and-paper situational judgment tests, holding constructs constant. Implications for practice and research are discussed.     

Validation of intelligence and mechanical aptitude tests in selecting employees for manufacturing jobs

One hundred ninety-three manufacturing employees who produce electro-mechanical components participated in a concurrent criterion-related validity study. The employees were administered three tests: The Bennett Mechanical Comprehension Test (Form S); The Flanagan Aptitude Classification Test-Mechanics; and the Thurstone Test of Mental Alertness (Form A). Job performance was measured by a supervisor rating of fifteen job dimensions, assessed at two points in time separated by 60 days. Correlational and multiple regression analyses were used to assess the relationship between test scores and job performance ratings. The results revealed that the Bennett Mechanical Comprehension test was the best single predictor of job performance (uncorrectedr =.38), and the incremental gain in predictability from additional tests was not significant. The results were discussed in the context of the changing nature of manufacturing jobs and the inadequacy of conventional mechanical aptitude tests to be sensitive to these changes.     

The Effects of Social Desirability and Faking on Personality and Integrity Assessment for Personnel Selection

A review of the extant literature and new empirical research suggests that social desirability is not much of a concern in personality and integrity testing for personnel selection. In particular, based on meta-analytically derived evidence, it appears that social desirability influences do not destroy the convergent and discriminant validity of the Big Five dimensions of personality (Emotional Stability, Extraversion, Openness to Experience, Agreeableness, and Conscientiousness). We also present new empirical evidence regarding gender and age differences in socially desirable re- sponding. Although social desirability predicts a number of important work variables such as job satisfaction, organizational commitment, and supervisor ratings of training success, social desirability does not seem to be a predictor of overall job performance and is only very weakly related to specific dimensions of job performance such as technical proficiency (r = -.07) and personal discipline ( r = .05). Large sample investigations of the moderating influences of social desirability in actual work settings indicate that social desirability does not moderate the criterion-related validities of personality variables or integrity tests. The criterion-related validity of integrity tests for overall job performance with applicant samples in predictive studies is .41. Controlling for social desirability in integrity or personality test scores leaves the operational validities intact, thereby suggesting that social desirability functions neither as a mediator nor as a suppressor variable in personality-performance.     

The Five-Factor model of personality and work behaviour in Greece

This study explores the validity of the five-factor model of personality (FFM) in occupational settings in Greece, examining its relationship to employees' overall job performance, job satisfaction, organizational citizenship behaviour, and generic work competencies. Two hundred and twenty-seven employees from various Greek SMEs participated in the study completing a personality and a job satisfaction measure. Their supervisors completed three questionnaires assessing their performance and their work competencies. Some of the most significant results of this study were the strong links identified between personality and job satisfaction and the moderating effect of job type on the criterion-related validity of some personality dimensions. These results are discussed in terms of the FFM literature taking into consideration the strong effect of Greek culture. The theoretical and practical implications for research and practice in Greece are also discussed.     

Incremental validity of person-organization fit over the Big Five personality measures

Few studies have provided the validity evidence of a measure of objective person-organization fit (P-O fit) as a selection tool. The present study used a concurrent validation design to examine the criterion-related validity and the incremental validity of a P-O fit measure beyond the validity of the Big Five personality test for predicting job performance (task performance and organizational citizenship behavior) and employee commitment (organizational commitment and supervisory commitment) for a group of high-tech professional employees in Taiwan. Results showed that P-O fit predicted the contextual component of overall job performance and was significantly related to two types of employee commitment. Moreover, P-O fit had an incremental validity beyond that of the personality measures for predicting some of our outcome variables.     

Can I retake it? Exploring subgroup differences and criterion-related validity in promotion retesting

Despite recent interest in the practice of allowing job applicants to retest, surprisingly little is known about how retesting affects 2 of the most critical factors on which staffing procedures are evaluated: subgroup differences and criterion-related validity. We examined these important issues in a sample of internal candidates who completed a job-knowledge test for a within-job promotion. This was a useful context for these questions because we had job-performance data on all candidates (N = 403), regardless of whether they passed or failed the promotion test (i.e., there was no direct range restriction). We found that retest effects varied by subgroup, such that females and younger candidates improved more upon retesting than did males and older candidates. There also was some evidence that Black candidates did not improve as much as did candidates from other racial groups. In addition, among candidates who retested, their retest scores were somewhat better predictors of subsequent job performance than were their initial test scores (rs = .38 vs. .27). The overall results suggest that retesting does not negatively affect criterion-related validity and may even enhance it. Furthermore, retesting may reduce the likelihood of adverse impact against some subgroups (e.g., female candidates) but increase the likelihood of adverse impact against other subgroups (e.g., older candidates).     

On the myth and the reality of the temporal validity degradation of general mental ability test scores

Claims of changes in the validity coefficients associated with general mental ability (GMA) tests due to the passage of time (i.e., temporal validity degradation) have been the focus of an on-going debate in applied psychology. To evaluate whether and, if so, under what conditions this degradation may occur, we integrate evidence from multiple sub-disciplines of psychology. The temporal stability of construct validity is considered in light of the evidence regarding the differential stability of g and the invariance of measurement properties of GMA tests over the adult life-span. The temporal stability of criterion-related validity is considered in light of evidence from long-term predictive validity studies in educational and occupational realms. The evidence gained from this broad-ranging review suggests that temporal degradation of the construct- and criterion-related validity of ability test scores may not be as ubiquitous as some have previously concluded. Rather, it appears that both construct and criterion-related validity coefficients are reasonably robust over time and that any apparent degradation of criterion-related validity coefficients has more to do with changes in the determinants of task performance and changes in the nature of the criterion domain rather temporal degradation per se (i.e., the age of the test scores). A key exception to the conclusion that temporal validity degradation is more myth than reality concerns decision validity. Although the evidence is sparse, it is likely that the utility of a given GMA test score for making diagnostic decisions about an individual deteriorates over time. Importantly, we also note several areas in need of additional and more rigorous research before strong conclusions can be supported.     

JOB-RELATEDNESS OF A POSTTRAINING JOB KNOWLEDGE CRITERION USE TO ASSESS VALIDITY AND TEST FAIRNESS

Significant job-relatedness was found for a posttraining job knowledge test criterion using an application of Lawshe's content validity method. The aide test was used as a criterion to assess the predictive validity of a vocabulary test and a civil service test with samples of black ( N = 43) and white ( N = 62) psychiatric aides. Significant validities were found on both tests, but a vocabulary test proved to be the better predictor of the criterion in both samples. The obtained validities were discussed in terms of differential validity, test fairness, and sample size. This study demonstrated that a content validity method could be applied to criteria as well as selection tests. It was concluded that content validity methods may be able to help solve the problem of criterion relevance in validation research by providing quantitative evidence of the job-relatedness of criteria.     

Personality and job performance: the Big Five revisited

Prior meta-analyses investigating the relation between the Big 5 personality dimensions and job performance have all contained a threat to construct validity, in that much of the data included within these analyses was not derived from actual Big 5 measures. In addition, these reviews did not address the relations between the Big 5 and contextual performance. Therefore, the present study sought to provide a meta-analytic estimate of the criterion-related validity of explicit Big 5 measures for predicting job performance and contextual performance. The results for job performance closely paralleled 2 of the previous meta-analyses, whereas analyses with contextual performance showed more complex relations among the Big 5 and performance. A more critical interpretation of the Big 5-performance relationship is presented, and suggestions for future research aimed at enhancing the validity of personality predictors are provided.     

A counterintuitive hypothesis about employment interview validity and some supporting evidence

This study found mixed support for the hypothesis that the difference in criterion-related validity between unstructured and structured employment interviews is due solely to the greater reliability of structured interviews. Using data from prior meta-analyses, this hypothesis was tested in 4 data sets by using standard psychometric procedures to remove the effects of measurement error in interview scores from correlations with rated job performance and training performance. In the 1st data set. support was found for this hypothesis. However, in a 2nd data set structured interviews had higher true score correlations with performance ratings, and in 2 other data sets unstructured interviews had higher true score correlations. We also found that averaging across 3 to 4 independent unstructured interviews provides the same level of validity for predicting job performance as a structured interview administered by a single interviewer. Practical and theoretical implications are discussed.     

PERSONALITY DIMENSIONS EXPLAINING RELATIONSHIPS BETWEEN INTEGRITY TESTS AND COUNTERPRODUCTIVE BEHAVIOR: BIG FIVE,OR ONE IN ADDITION?

Although the criterion-related validity of integrity tests is well established, there has not been enough research examining which personality constructs contribute to their criterion-related validity. Moreover, evidence of how well findings on integrity tests in North America generalize to non-English speaking countries is virtually absent. This research addressed these issues with data obtained from employees and students in Canada and Germany (total N = 853). Specifically, we tested the hypotheses that (a) Honesty–Humility, as specified in the HEXACO model of personality, is relatively more important than the Big 5 dimensions of personality in accounting for the criterion-related validity of overt integrity tests, whereas (b) the Big 5 are relatively more important in explaining the validity of personality-based integrity tests. These predictions were tested using 2 criteria (counterproductive work behavior and counterproductive academic behavior) as well as 2 overt and 2 personality-based integrity tests. We found evidence of the expected differences between types of integrity tests largely regardless of culture of the sample, specific test, criterion, or population under research, pointing to some degree of generalizability of findings in integrity testing research. Implications include theoretical refinements in research on integrity testing and encouragement of practical applications beyond North America.     

Measuring Job Interview Anxiety: Beyond Weak Knees and Sweaty Palms

A multidimensional measure of interview anxiety, called the Measure of Anxiety in Selection Interviews (MASI), was developed using a student sample  ( N = 212)  and tested using a sample of job applicants in a field setting  ( N = 276)  . The MASI goes beyond the measurement of "weak knees" and "sweaty palms" by providing an assessment of 5 interview anxiety dimensions: Communication, Appearance, Social, Performance, and Behavioral. The psychometric properties of the scales were strong and confirmatory factor analyses supported the a priori structure. In addition, substantial evidence for the concurrent, discriminant, criterion-related, and incremental validity of the MASI was obtained. Moreover, a multiple correlation of .34 was found for the 5 MASI scales in the prediction of interview performance. The development of the MASI has important implications for the field, as it may provide the foundation for future research on job interview anxiety, guide interview anxiety treatment programs, and promote the enhancement of job interview validity.     

PUBLICATION BIAS: A CASE STUDY OF FOUR TEST VENDORS

This article has 2 goals. First, we discuss publication bias and explain why it presents a potential problem for industrial and organizational psychology. After reviewing the traditional failsafe N, or file drawer analysis, we introduce a more sophisticated method of publication bias analysis (trim and fill), which has been developed in the medical literature but is largely unfamiliar to industrial and organizational psychology researchers. Second, we demonstrate trim and fill by applying it to validity information reported in the technical manuals of 4 test vendors. In doing so, we assess the likelihood that criterion-related validity information provided by test publishers may overestimate test validity. In our analysis of 18 validity distributions, we found evidence of either no or minimal bias for 2 of the vendors' distributions and evidence of moderate-to-severe bias in at least 1 distribution from each of the other 2 vendors. In both cases in which publication bias was found, we noted instances in which the publishers tended to report only statistically significant correlations and that this practice was detected using publication bias methodology.     

Modeling the impact of test anxiety and test familiarity on the criterion-related validity of cognitive ability tests

The assessment of cognitive abilities, whether it is for purposes of basic research or applied decision making, is potentially susceptible to both facilitating and debilitating influences. However, relatively little research has examined the degree to which these factors might moderate the criterion-related validity of cognitive ability tests. To address this gap, we use Classical Test Theory formulas to articulate how test anxiety and test familiarity can influence observed scores, observed score variance, and most importantly, the criterion-related validity of observed scores. The resulting equations reveal that understanding the influence of test anxiety and test familiarity on criterion-related validity coefficients requires the consideration of a number of additional parameters. To elucidate the implications of the model, we present a Monte Carlo simulation. Results show that anxiety and familiarity can have a significant negative effect on the observed criterion-related validity, but also show that this effect is highly variable. In particular, the effect depends heavily upon the relation between these factors and the criterion variable. Additionally, we note that the equations we develop highlight important gaps in the literature; there are few clear empirical estimates of several of the parameters in our formulas. We call for future research to better examine these additional relations.     

EMPLOYMENT TEST VALIDATION FOR MINORITY AND NONMINORITY PRODUCTION WORKERS

The purposes of the present study were (a) to examine the validity of selected employment tests potentially useful in selecting production workers engaged in the construction of boxboard containers and (b) to evaluate the applicability of the tests to minority and non-minority workers. Using data collected from 100 production workers employed by the same company but located in two different geographical regions, it was found that a short test battery was potentially useful in selecting production employees without necessarily introducing unfair racial bias. Implications of the results for future research studies and test validation efforts involving differential and single-group validity are discussed.