SYNTHETIC VALIDITY AND ITS APPLICATION TO THE UNIFORM GUIDELINES VALIDATION REQUIREMENTS

The Uniform Guidelines for Employee Selection Procedures have served to create an urgent need for efficient validation methods that can be generalized to a class of occupations. The one method currently authorized for such a purpose by the Guidelines is synthetic validation. (The Guidelines erroneously describe the synthetic validity paradigm as construct validity.) Approaches to synthetic validity employed by Lawshe, Guion, McCormick, and Primoff are described. Their extent of conformance to the Guidelines validation requirements is noted. Primoff's J -Coefficient approach is recommended for two reasons; it meets the Guidelines requirements and under certain circumstances it permits the test user to estimate the traditional validity coefficient. An illustrated example of Primoff's method is presented.     

SYNTHETIC VALIDITY: PAST, PRESENT, AND FUTURE

Practitioners in the area of personnel selection can be confronted with situations in which the use of traditional validation strategies is not feasible. This situation can result from small numbers of incumbents, rapidly changing jobs, insufficient resources, or the lack of high-quality criterion data. Although traditional strategies are not possible, a set of techniques falling under the rubric of synthetic validity can be used to estimate predictor validity in these situations. Synthetic validity is a logical process of inferring validity on the basis of the relationships between components of a job and tests of the attributes that are needed to perform the job components. Unfortunately, synthetic validity is infrequently used and not well understood. This article provides an overview and conceptualization of synthetic validity, reviews the historical approaches to synthetic validity, describes the recent developments and trends, and suggests future directions for synthetic validity application and research. The relationship of synthetic validity with other validity concepts, the advantages, the limitations, and the legal issues are considered throughout.     

GENERALIZING PHYSICAL ABILITY TEST VALIDITY: A CASE STUDY USING TEST TRANSPORTABILITY, VALIDITY GENERALIZATION, AND CONSTRUCT-RELATED VALIDATION EVIDENCE

This paper describes a field study using Position Analysis Questionnaire (PAQ; McCormick, Jeanneret, & Mecham, 1972) data to support inferences of physical ability test (PAT) validity at a large utility company. The project developed a test transportability procedure using worker-oriented job analysis, and was based on synthesizing the findings of an earlier consortium project (American Gas Association, 1990), recent validity generalization evidence, and findings from a variety of physical ability construct-related research projects published or presented since the completion of the original consortium study. Selected PAQ divisional dimension scores for 95 jobs were analyzed via cluster analysis to develop five physical demand job families. Statistically derived job families were modified very slightly based on reviews by subject matter experts who found PAQ-derived job family structures quite reasonable (agreement of 95% between statistical and rational procedures). PAQ strength attributes were used to estimate the physical demands underlying job families. The statistical and analytic procedures developed and reported here could be applied in other organizational settings, and allow broader applicability of the PAT than would a task-based transportability model.     

RELIABILITY IS NOT VALIDITY AND VALIDITY IS NOT RELIABILITY

Interrater correlations do provide an index of reliability of job performance ratings. We show that the arguments presented by Murphy and DeShon (2000) lead to the radical conclusion that traditional measurement models–both classical theory and generalizability theory models–can be used neither with job performance ratings nor with other measures used in I-O and other areas of psychology and the social sciences. We show that this untenable conclusion is based on confusion of validity issues and questions with reliability issues and questions. It is also based on the incorrect belief that classical measurement models are capable of addressing only random response measurement error and cannot address other forms of measurement error. We also show that the solution Murphy and DeShon offer to the problem of measurement error in ratings, as they define this problem, cannot work. Properly understood, the position taken by Murphy and DeShon leaves us with the nihilistic conclusion that no appropriate measurement models are possible in psychological research, thus making meaningful research impossible.     

INTERVIEW NOTES AND VALIDITY

Three studies were conducted investigating the effects of notetaking behavior and the content of notes on validity in a selection interview. Overall, results indicate that when notetaking was voluntary, notetakers made more valid ratings than non-notetakers. But, when note-taking was manipulated non-notetakers made ratings that were just as valid as raters told to take behavioral notes, and actually more valid than those raters instructed to take general notes. With regard to the content of notes, behavioral-type notes were found to contribute to greater validity. In addition, in the first and third study, procedural type notes resulted in lower validity. These results may have important implications for interviewer training for notetaking.     

COMPARING THE TORTOISE AND THE HARE:

Abstract— Recent research suggests that dynamic spatial reasoning tasks show more robust gender differences than static spatial reasoning tasks. These differences have implications for selection procedures based on aptitude test scores. Two experiments were therefore designed to examine the locus of such gender differences. In Experiment 1, 82 males and 82 females performed two separate tasks: judging the relative velocity of moving objects and judging their relative distances from target destinations. Significant gender differences occurred only for relative velocity judgments and were partially related to prior experience. A second experiment therefore tested the effects of practice and feedback on performance judging relative velocity. Male and female performance differences were again found, and performance improved equally as a result of feedback. Consistent with contemporary views of aptitude and assessment, the data suggest that equitable assessment of dynamic spatial abilities requires a determination of experiential history and learning rates as well as absolute performance levels.     

EDUCATIONAL REQUIREMENTS: LEGALITY AND VALIDITY1

Educational achievements (e.g., high school diploma, college degree) are frequently used as selection requirements for a wide variety of jobs. Despite this, the legality and validity of this “selection test” has been a neglected subject of study by personnel psychologists. In this article, two important issues concerning educational requirements are discussed. First, court cases are reviewed to gain insight into judicial opinions. Next, validity studies of educational requirements are summarized. The article concludes with suggestions for the proper use and defense of educational requirements.     

病前智力估计:效度与局限

病前智力的估计是甄别、诊断智能衰退的必要条件,有重要的临床和科研意义。依据学业成绩和工作成就等背景资料可综合评估病前智力的水平与结构,但其定性的结论不利于科研,且难免主观。依赖人口统计学变量的回归公式能客观给出确切的、具有一定效度的估计数,但会高估或低估极端智商。依据当前测验成绩的估计有“保持”和最佳作业两种模式,这种估计个体特异高,但被试在选用测验上的当前表现能否代表病前智力水平尚有争议。联合不同方法能提高预测效度。     

ASSESSMENT CENTERS AND CONTENT VALIDITY: SOME NEGLECTED ISSUES

Literature on the use of content validity to establish the job-relatedness of assessment centers is reviewed. Most approaches to content validity are shown to focus on job-analysis-based exercise construction and dimension selection as the bases for determining the content validity of an assessment center. The article attempts to show that content validity requires more than the careful construction of stimulus materials (i.e., exercises); how these stimulus materials are presented to candidates and how responses to these stimuli are evaluated are also critical considerations in making judgments about content validity. Variability in the way these issues are dealt with is documented.     

INTERVIEWER VALIDITY AND RELIABILITY: AN INDIVIDUAL ANALYSIS APPROACH

Interviewer reliability, validity, and strategy for information integration were studied by analysis of data across interviewers and, also, by within, individual interviewer analysis. Candidates ( N = 412) for selection to a military division of a national defense organization were interviewed by 10 female interviewers and assessed on nine behaviorally anchored dimensions. Candidates ( N = 131) subsequently admitted to officers' training school were evaluated, for the purposes of this study, on 19 dimensions and on an overall evaluation taken at six and twelve week points. Results of analyses of data across interviewers indicated that interviewers functioned in a similar fashion, using few of the dimensions in their decisions whereas analyses of individual interviewers indicated higher reliability and individual differences among interviewers' strategy formation. Analysis across interviewers of the relationship of the interview decision to six and twelve week training performance evaluations indicated no validity for the interview decision. Analysis of individual interview strategies revealed differences among the interviewers only at the six week point. Results are discussed with regard to methodological problems, interview strategy differences, criteria dimensionality, fruitfulness of individual, within interviewer analyses, and purpose of the interview.     

PERSONALITY AND JOB PERFORMANCE: EVIDENCE OF INCREMENTAL VALIDITY

It is suggested that personality variables are significant predictors of job performance when carefully matched with the appropriate occupation and organization. The present study investigates the relationship between specific personality variables and job performance in a sample ( N = 43) of accountants. The results indicate that even with the effects of cognitive ability taken into account, three personality scales (orientation towards work; degree of ascendancy; and degree and quality of interpersonal orientation) are significantly related to important aspects of job performance. It is suggested that the overall validity of selection strategies might be improved with the addition of measures of relevant personality dimensions when appropriately matched to an occupation and organization.     

SIMULATION-BASED ASSESSMENT OF MANAGERIAL COMPETENCE: RELIABILITY AND VALIDITY

Prior work based on complexity theory has attempted, with some success, to predict general and managerial performance in complex, uncertain, and fluid task settings. The present paper evaluates a quasi-experimental simulation technique that was specifically designed to measure the impact of individual differences in a number of managerial styles (including a style reflecting cognitive complexity) on executive performance. Twenty simulation-based measures were tested for reliability and validity. On the basis of the data obtained from two separate samples, it was concluded that this quasi-experimental simulation technology may be useful for assessing a number of managerial styles that are not currently tapped by other measurement methods. Research results reported in this paper, as well as results obtained in other concurrent efforts, are summarized.     

SELF-DISCLOSURE AND INTERPERSONAL SOLIDARITY: MEASUREMENT,VALIDATION, AND RELATIONSHIPS

This study investigated the relationship of self-disclosure and interpersonal solidarity. Major emphasis was devoted to the refinement of a multidimensional measure of reported self-disclosure and development of a preliminary measure of solidarity. Results replicated in two studies indicated that (1) the self-disclosure scales could be improved to acceptable levels of reliability, (2) the solidarity measure validly distinguished between closer and more distant relationships, (3) self-disclosure and solidarity were positively related, and (4) meaningfully higher levels of self-disclosure were associated with high solidarity relations than with low solidarity relations.     

EXPERIENCE-BASED AND SITUATIONAL INTERVIEW QUESTIONS: STUDIES OF VALIDITY

This research compared the validity of two different types of structured interview questions (i.e., experience-based and situational) under tightly controlled conditions. The experience-based interview questions required that 108 study participants relate how they had handled situations in the past requiring skills and abilities necessary for effective performance on the job. Situational questions, administered to another group of 108 study participants, provided interviewees with hypothetical job-relevant situations and asked them how they would respond if they were confronted with these problems. The experience-based interview questions yielded higher levels of validity than the situational questions. Additional analyses showed that the interview added incrementally to the prediction of performance beyond the variance accounted for by a cognitive ability test. There were small differences in subgroup performance (White, Black, Hispanic, male, and female) on the experience-based interview, though it was equally valid for all subgroups.     

《青少年心理健康素质调查表》动力系统分量表的编制

本文旨在介绍青少年心理健康素质动力系统量表的编制及其标准化过程。量表编制的重要理论基础为马斯洛的五级动机理论和弗兰克尔的自我超越动机理论。研究的测量工具有:自尊量表,自我超越生命意义量表,指向自我生命意义量表和对意义的追寻量表。被试为北京市某中学的334名初中高中学生。结果显示:青少年心理健康素质动力系统量表的内部一致性Cronbach′sα=0.90,效标效度较理想。研究结论:该量表可以作为测量青少年心理健康素质动力系统的一个较为可靠的工具。     

THE ROLE OF THE TRADITIONAL RESEARCH DESIGN IN UNDERESTIMATING THE VALIDITY OF THE EMPLOYMENT INTERVIEW

Building upon the suggestions of Arvey and Campion (1982) and Zedeck, Tziner, and Middlestadt (1983), this paper calls into question the traditional approach to validating the employment interview. A general review of the literature reveals that individuals are likely to differ in their ability to provide accurate predictions of employee behavior and that interviewers often differ in their constant tendency to make favorable or unfavorable ratings. A review of published validation studies shows that most researchers fail to consider these tendencies and, therefore, may be inappropriately collapsing data across multiple interviewers. Through analytic argument we conclude that reviews of the existing literature will likely underestimate the usefulness of this widely used technique.     

THE DIVERSITY–VALIDITY DILEMMA: OVERVIEW AND LEGAL CONTEXT

Due to racioethnic and sex subgroup differences on predictor scores in many selection procedures, it is difficult for organizations to simultaneously maximize the validity of their selection procedures and hire a diverse workforce. One response to this diversity–validity dilemma is to revise the selection procedures, an approach developed by Ployhart and Holtz (this issue, 2008) . A second possible response is to use affirmative action to increase workforce diversity, an approach developed by Kravitz (this issue, 2008) . This paper briefly presents the legal context that motivates and constrains these approaches. We begin by defining key terms, describing adverse impact, and outlining the burden of proof in adverse impact cases. We then turn to the use of racioethnic minority and female preferences, summarizing some key court decisions and the conditions under which private and public employers may use preferences.