The need for an evolving concept of validity in industrial and personnel psychology: Psychometric, legal, and emerging issues

The Trinitarian view of validity (namely content, criterion-related, and construct) has dominated psychology for almost a half century. In industrial and personnel psychology especially, content and criterion-related (in particular) forms of validity have served as workhorses. Such a limited focus has lead to a stagnation of the area and the inability to incorporate more theory into the research and practice of personnel psychology. Therefore, we review the evolution of the concept of validity in the psychology discipline, with a particular emphasis on industrial and personnel psychology. We then review the legal and statutory history within industrial and personnel psychology that has helped to shape the concept of validity over the years. Finally, we examine some emerging trends in industrial and personnel psychology, including the issue of graduate training, which may have important consequences on how we conceptualize validity in the future in both our research and practice of personnel psychology.     

Cognitive Tests Used in Selection Can Have Content Validity as Well as Criterion Validity: A broader research review and implications for practice

Many industrial/organizational (I/O) psychologists, both academics and practitioners, believe that the content validity model is not appropriate for cognitive ability measures used in personnel selection. They believe that cognitive tests can have criterion validity and construct validity but not content validity. Based on a review of the broader differential psychology research literature on cognitive skills, aptitudes, and abilities, this article demonstrates that with the proper content validity procedures, cognitive ability measures, including, ultimately, some de facto measures of general cognitive ability, can have content validity in addition to criterion and construct validity. Finally, the article considers, critiques, and refutes the specific arguments contending that content validity is inappropriate for use with cognitive skills and abilities. These research facts have implications for I/O practice, professional standards, and legal defensibility of selection programs.     

On the myth and the reality of the temporal validity degradation of general mental ability test scores

Claims of changes in the validity coefficients associated with general mental ability (GMA) tests due to the passage of time (i.e., temporal validity degradation) have been the focus of an on-going debate in applied psychology. To evaluate whether and, if so, under what conditions this degradation may occur, we integrate evidence from multiple sub-disciplines of psychology. The temporal stability of construct validity is considered in light of the evidence regarding the differential stability of g and the invariance of measurement properties of GMA tests over the adult life-span. The temporal stability of criterion-related validity is considered in light of evidence from long-term predictive validity studies in educational and occupational realms. The evidence gained from this broad-ranging review suggests that temporal degradation of the construct- and criterion-related validity of ability test scores may not be as ubiquitous as some have previously concluded. Rather, it appears that both construct and criterion-related validity coefficients are reasonably robust over time and that any apparent degradation of criterion-related validity coefficients has more to do with changes in the determinants of task performance and changes in the nature of the criterion domain rather temporal degradation per se (i.e., the age of the test scores). A key exception to the conclusion that temporal validity degradation is more myth than reality concerns decision validity. Although the evidence is sparse, it is likely that the utility of a given GMA test score for making diagnostic decisions about an individual deteriorates over time. Importantly, we also note several areas in need of additional and more rigorous research before strong conclusions can be supported.     

PUBLICATION BIAS: A CASE STUDY OF FOUR TEST VENDORS

This article has 2 goals. First, we discuss publication bias and explain why it presents a potential problem for industrial and organizational psychology. After reviewing the traditional failsafe N, or file drawer analysis, we introduce a more sophisticated method of publication bias analysis (trim and fill), which has been developed in the medical literature but is largely unfamiliar to industrial and organizational psychology researchers. Second, we demonstrate trim and fill by applying it to validity information reported in the technical manuals of 4 test vendors. In doing so, we assess the likelihood that criterion-related validity information provided by test publishers may overestimate test validity. In our analysis of 18 validity distributions, we found evidence of either no or minimal bias for 2 of the vendors' distributions and evidence of moderate-to-severe bias in at least 1 distribution from each of the other 2 vendors. In both cases in which publication bias was found, we noted instances in which the publishers tended to report only statistically significant correlations and that this practice was detected using publication bias methodology.     

MODERATOR RESEARCH AND THE LAW OF SMALL NUMBERS1

The thesis of this paper is that many proposed moderators in personnel psychology are probably illusory, having been created solely by belief in the law of small numbers. Evidence is presented that race as a moderator of test validity is one such illusory moderator. In addition, a model for validity generalization is described which, in addition to eliminating the need for criterion-related validity studies under certain circumstances, strongly calls into question the idea that situations moderate test validity, i.e., the traditional doctrine of situational specificity of test validities. Calculations are presented which show that adequate statistical power in moderator research requires much larger sample sizes than have typically been employed. This requirement is illustrated empirically using validity data for the Army Classification Battery for 35 jobs and 21,000 individuals. These analyses show that (1) even when a moderator is generally assumed to be large, large samples are required to gauge its effect reliably and (2) large sample research may show that moderators that appear plausible and important a priori are nonexistent or trivial in magnitude. The practice of pooling across numerous small sample studies to obtain statistical power equivalent to that of large sample studies is recommended. In light of the evidence that many proposed moderators may not exist, the authors hypothesize that the true structure of underlying relationships in personnel psychology is considerably simpler than personnel psychologists have generally imagined it to be.     

单题项测量：质疑、回应及建议

研究者关于单题项测量的争论已久。支持者认为单题项测量具有时间和效率上的优势, 而反对者则认为单题项测量的信效度均无法得到保障。通过定性和定量的回顾, 归纳了单题项测量的优缺点, 剖析了以往研究对单题项测量的种种质疑并逐一进行回应。通过系统梳理, 发现单题项测量具有可以接受的信度和效度水平, 且多题项测量的效标关联效度并没有显著优于单题项测量。最后, 指出了单题项测量开发和使用过程中应该注意的事项。尽管多题项测量仍是当前研究界的主流测量方法, 但未来研究者应当更加客观地看待单题项测量。学界应当充分理解单题项测量潜在的优点和适用范围, 从而使单题项测量在管理心理学和社会科学研究中发挥其应有的作用。     

Modeling the impact of test anxiety and test familiarity on the criterion-related validity of cognitive ability tests

The assessment of cognitive abilities, whether it is for purposes of basic research or applied decision making, is potentially susceptible to both facilitating and debilitating influences. However, relatively little research has examined the degree to which these factors might moderate the criterion-related validity of cognitive ability tests. To address this gap, we use Classical Test Theory formulas to articulate how test anxiety and test familiarity can influence observed scores, observed score variance, and most importantly, the criterion-related validity of observed scores. The resulting equations reveal that understanding the influence of test anxiety and test familiarity on criterion-related validity coefficients requires the consideration of a number of additional parameters. To elucidate the implications of the model, we present a Monte Carlo simulation. Results show that anxiety and familiarity can have a significant negative effect on the observed criterion-related validity, but also show that this effect is highly variable. In particular, the effect depends heavily upon the relation between these factors and the criterion variable. Additionally, we note that the equations we develop highlight important gaps in the literature; there are few clear empirical estimates of several of the parameters in our formulas. We call for future research to better examine these additional relations.     

IN SUPPORT OF PERSONALITY ASSESSMENT IN ORGANIZATIONAL SETTINGS

Personality constructs have been demonstrated to be useful for explaining and predicting attitudes, behaviors, performance, and outcomes in organizational settings. Many professionally developed measures of personality constructs display useful levels of criterion-related validity for job performance and its facets. In this response to Morgeson et al. (2007) , we comprehensively summarize previously published meta-analyses on (a) the optimal and unit-weighted multiple correlations between the Big Five personality dimensions and behaviors in organizations, including job performance; (b) generalizable bivariate relationships of Conscientiousness and its facets (e.g., achievement orientation, dependability, cautiousness) with job performance constructs; (c) the validity of compound personality measures; and (d) the incremental validity of personality measures over cognitive ability. Hundreds of primary studies and dozens of meta-analyses conducted and published since the mid 1980s indicate strong support for using personality measures in staffing decisions. Moreover, there is little evidence that response distortion among job applicants ruins the psychometric properties, including criterion-related validity, of personality measures. We also provide a brief evaluation of the merits of alternatives that have been offered in place of traditional self-report personality measures for organizational decision making. Given the cumulative data, writing off the whole domain of individual differences in personality or all self-report measures of personality from personnel selection and organizational decision making is counterproductive for the science and practice of I-O psychology.     

Multilevel modeling in industrial and personnel psychology

Many of the topics of interest in the social and behavior sciences are often hierarchical or multilevel in nature. These multiple levels (e.g., individual versus group) create problems for researchers related to the choice of measurement and analysis. Recent innovations in statistical analysis have made it possible to account for the hierarchical nature of observations. Therefore, in this article we begin with a review of multilevel analysis techniques and discuss advances that have been made in the social sciences using multilevel models. Next, we summarize contemporary research specific to the organizational psychology literature that uses multilevel analysis. Possible applications for industrial and personnel psychology are then discussed. Guidelines for determining if multilevel analysis is appropriate for a given applied research project are provided. We conclude with a summary and call for increased use of multilevel analysis in industrial and personnel psychology.     

Personnel selection in the digital age: a review of validity and applicant reactions,and future research challenges

ABSTRACT

We present a targeted review of recent developments and advances in digital selection procedures (DSPs) with particular attention to advances in internet-based techniques. By reviewing the emergence of DSPs in selection research and practice, we highlight five main categories of methods (online applications, online psychometric testing, digital interviews, gamified assessment and social media). We discuss the evidence base for each of these DSP groups, focusing on construct and criterion validity, and applicant reactions to their use in organizations. Based on the findings of our review, we present a critique of the evidence base for DSPs in industrial, work and organizational psychology and set out an agenda for advancing research. We identify pressing gaps in our understanding of DSPs, and ten key questions to be answered. Given that DSPs are likely to depart further from traditional non-digital selection procedures in the future, a theme in this agenda is the need to establish a distinct and specific literature on DSPs, and to do so at a pace that reflects the speed of the underlying technological advancement. In concluding, we, therefore, issue a call to action for selection researchers in work and organizational psychology to commence a new and rigorous multidisciplinary programme of scientific study of DSPs.     

Multilevel modeling in industrial and personnel psychology

Many of the topics of interest in the social and behavior sciences are often hierarchical or multilevel in nature. These multiple levels (e.g., individual versus group) create problems for researchers related to the choice of measurement and analysis. Recent innovations in statistical analysis have made it possible to account for the hierarchical nature of observations. Therefore, in this article we begin with a review of multilevel analysis techniques and discuss advances that have been made in the social sciences using multilevel models. Next, we summarize contemporary research specific to the organizational psychology literature that uses multilevel analysis. Possible applications for industrial and personnel psychology are then discussed. Guidelines for determining if multilevel analysis is appropriate for a given applied research project are provided. We conclude with a summary and call for increased use of multilevel analysis in industrial and personnel psychology.     

Predictor content matters for knowledge testing: Evidence supporting content validation

Organizations often rely on the match between job requirements and test content to justify test use. This practice has been questioned on the grounds that content validation has little relevance to criterion-related validation due to positive manifold among predictors. We analyze two large databases to assess the implications of test content for (a) test interchangeability and (b) criterion-related validity. Analyses of 15 knowledge tests administered (N = 80,394) as part of Project Talent demonstrate that test content is related to predictor interchangeability. Analyses of SAT and Advanced Placement test data compare correlations among predictors and criteria drawn from matched and unmatched content domains. We conclude that test-criterion content match is likely to result in stronger criterion-related validity.     

Transparency in Structured Interviews: Consequences for Construct and Criterion-Related Validity

Although researchers agree on the use of structured interviews in personnel selection, past research has been undecided on whether these interviews need to be conducted nontransparently (i.e., without giving interviewees any indication of the evaluated criteria) or transparently (i.e., by revealing to interviewees the dimensions assessed in the interview). This article presents two independent studies examining the effects of interview transparency on interviewees' performance and on the interview's construct and criterion-related validity in the context of an application training program. Results from both Study 1 (N = 123) and Study 2 (N = 269) indicate an improvement in interviewees' performance under transparent interview conditions. Both studies further support the assumption that transparent interviews show satisfactory construct validity, whereas nontransparent interviews do not. Moreover, Study 2 showed no significant difference between the interview's criterion-related validity under transparent versus nontransparent conditions. Implications and directions for future research are discussed.     

The Effects of Social Desirability and Faking on Personality and Integrity Assessment for Personnel Selection

A review of the extant literature and new empirical research suggests that social desirability is not much of a concern in personality and integrity testing for personnel selection. In particular, based on meta-analytically derived evidence, it appears that social desirability influences do not destroy the convergent and discriminant validity of the Big Five dimensions of personality (Emotional Stability, Extraversion, Openness to Experience, Agreeableness, and Conscientiousness). We also present new empirical evidence regarding gender and age differences in socially desirable re- sponding. Although social desirability predicts a number of important work variables such as job satisfaction, organizational commitment, and supervisor ratings of training success, social desirability does not seem to be a predictor of overall job performance and is only very weakly related to specific dimensions of job performance such as technical proficiency (r = -.07) and personal discipline ( r = .05). Large sample investigations of the moderating influences of social desirability in actual work settings indicate that social desirability does not moderate the criterion-related validities of personality variables or integrity tests. The criterion-related validity of integrity tests for overall job performance with applicant samples in predictive studies is .41. Controlling for social desirability in integrity or personality test scores leaves the operational validities intact, thereby suggesting that social desirability functions neither as a mediator nor as a suppressor variable in personality-performance.     

ASSESSING THE CANDIDATE AS A WHOLE: A HISTORICAL AND CRITICAL ANALYSIS OF INDIVIDUAL PSYCHOLOGICAL ASSESSMENT FOR PERSONNEL DECISION MAKING

Although individual assessment is a thriving area of professional practice in industry, it receives little, if any, attention from textbooks on industrial psychology or personnel management. This article is an attempt to establish individual assessment's place in the history of personnel selection, and to examine why the practice has survived despite receiving little attention in research and graduate training. It is argued that the clinical, holistic approach that has characterized individual-assessment practice has survived primarily because the "elementalistic" testing approach, focusing on traits and abilities, has often been dismissed as inadequate for addressing the complexities of the executive profile. Moreover, public displeasure with standard paper-and-pencil testing in the 1960s and 1970s made the holistic approach to assessment an attractive, alternative. The article contrasts individual assessment practice with the current state of knowledge on psychological assessment and personnel decision making. Like psychotherapy in the 1950s, individual psychological assessment appears to have achieved the status of functional autonomy within psychology.     

SITUATIONAL JUDGMENT TESTS: CONSTRUCTS ASSESSED AND A META-ANALYSIS OF THEIR CRITERION-RELATED VALIDITIES

Situational judgment tests (SJTs) are a measurement method that may be designed to assess a variety of constructs. Nevertheless, many studies fail to report the constructs measured by the situational judgment tests in the extant literature. Consequently, a construct-level focus in the situational judgment test literature is lacking, and researchers and practitioners know little about the specific constructs typically measured. Our objective was to extend the efforts of previous researchers (e.g., McDaniel, Hartman, Whetzel, & Grubb, 2007 ; McDaniel & Ngyuen, 2001 ; Schmitt & Chan, 2006 ) by highlighting the need for a construct focus in situational judgment test research. We identified and classified the construct domains assessed by situational judgment tests in the literature into a content-based typology. We then conducted a meta-analysis to determine the criterion-related validity of each construct domain and to test for moderators. We found that situational judgment tests most often assess leadership and interpersonal skills and those situational judgment tests measuring teamwork skills and leadership have relatively high validities for overall job performance. Although based on a small number of studies, we found evidence that (a) matching the predictor constructs with criterion facets improved criterion-related validity; and (b) video-based situational judgment tests tended to have stronger criterion-related validity than pencil-and-paper situational judgment tests, holding constructs constant. Implications for practice and research are discussed.     

SITUATIONAL JUDGMENT TESTS, RESPONSE INSTRUCTIONS, AND VALIDITY: A META-ANALYSIS

Situational judgment tests (SJTs) are personnel selection instruments that present job applicants with work-related situations and possible responses to the situations. There are typically 2 types of instructions: behavioral tendency and knowledge. Behavioral tendency instructions ask respondents to identify how they would likely behave in a given situation. Knowledge instructions ask respondents to evaluate the effectiveness of possible responses to a given situation. Results showed that response instructions influenced the constructs measured by the tests. Tests with knowledge instructions had higher correlations with cognitive ability. Tests with behavioral tendency instructions showed higher correlations with personality constructs. Results also showed that response instructions had little moderating effect on criterion-related validity. Supplemental analyses showed that the moderating effect of response instructions on construct validity was not due to systematic differences in item content. SJTs have incremental validity over cognitive ability, the Big 5, and over a composite of cognitive ability and the Big 5.     

Science, ethnicity, and bias: where have we gone wrong?

The quality, quantity, and funding of ethnic minority research have been inadequate. One factor that has contributed to this inadequacy is the practice of scientific psychology. Although principles of psychological science involve internal and external validity, in practice psychology emphasizes internal validity in research studies. Because many psychological principles and measures have not been cross-validated with different populations, those conducting ethnic minority research often have a more difficult time demonstrating rigorous internal validity. Thus, psychology's overemphasis of internal as opposed to external validity has differentially hindered the development of ethnic minority research. To develop stronger research knowledge on ethnic minority groups, it is important that (a) all research studies address external validity issues and explicitly specify the populations to which the findings are applicable; (b) different research approaches, including the use of qualitative and ethnographic methods, be appreciated; and (c) the psychological meaning of ethnicity or race be examined in ethnic comparisons.     

The Importance of Exercise and Dimension Factors in Assessment Centers: Simultaneous Examinations of Construct-Related and Criterion-Related Validity

This study presents a simultaneous examination of multiple evidential bases of the validity of assessment center (AC) ratings. In particular, we combine both construct-related and criterion-related validation strategies in the same sample to determine the relative importance of exercises and dimensions. We examine the underlying structure of ACs in terms of exercise and dimension factors while directly linking these factors to a work-related criterion (salary). Results from an AC (N = 753) showed that exercise factors not only explained more variance in AC ratings than dimension factors but also were more important in predicting salary. Dimension factors explained a smaller albeit significant portion of the variance in AC ratings and had lower validity for predicting salary. The implications of these findings for AC theory, practice, and research are discussed.     

They're Right,Well ... Mostly Right: Research Evidence and an Agenda to Rescue Personality Testing From 1960s Insights

We comment on the major points made by Murphy and Dzieweczynski (this issue) and Hogan (this issue), and although we agree with much of what they say, we go on to provide solid reasons for hope in the development and use of personality tests in personnel selection settings. We review current theorizing and research in this area, which suggests that the pessimistic view-that personality tests necessarily yield low-to-no criterion-related validity-is unwarranted and untrue. The points of caution offered by Murphy, Dzieweczynski, and Hogan should therefore also be viewed as points of scientific challenge, and to that end we conclude by outlining an agenda of major topics for future research.