The criterion-related validity of integrity tests: an updated meta-analysis

Integrity tests have become a prominent predictor within the selection literature over the past few decades. However, some researchers have expressed concerns about the criterion-related validity evidence for such tests because of a perceived lack of methodological rigor within this literature, as well as a heavy reliance on unpublished data from test publishers. In response to these concerns, we meta-analyzed 104 studies (representing 134 independent samples), which were authored by a similar proportion of test publishers and non-publishers, whose conduct was consistent with professional standards for test validation, and whose results were relevant to the validity of integrity-specific scales for predicting individual work behavior. Overall mean observed validity estimates and validity estimates corrected for unreliability in the criterion (respectively) were .12 and .15 for job performance, .13 and .16 for training performance, .26 and .32 for counterproductive work behavior, and .07 and .09 for turnover. Although data on restriction of range were sparse, illustrative corrections for indirect range restriction did increase validities slightly (e.g., from .15 to .18 for job performance). Several variables appeared to moderate relations between integrity tests and the criteria. For example, corrected validities for job performance criteria were larger when based on studies authored by integrity test publishers (.27) than when based on studies from non-publishers (.12). In addition, corrected validities for counterproductive work behavior criteria were larger when based on self-reports (.42) than when based on other-reports (.11) or employee records (.15).     

Internal consistency reliability of the self-report antisocial process screening device

The self-report version of the Antisocial Process Screening Device (APSD) has become a popular measure for assessing psychopathic features in justice-involved adolescents. However, the internal consistency reliability of its component scales (Narcissism, Callous-Unemotional, and Impulsivity) has been questioned in several studies. This study evaluates the internal consistency reliability of the self-report APSD by examining various indices (e.g., Cronbach's alpha, mean interitem correlation) across 11 studies. Whereas the Narcissism and Impulsivity scales display moderate to good reliability, internal consistency indices for the Callous-Unemotional scale are consistently poor across studies. Suggestions are made for revisions to this scale that would substantially improve its internal consistency reliability.     

The use of person-organization fit in employment decision making: an assessment of its criterion-related validity

Because measures of person-organization (P-O) fit are accountable to the same psychometric and legal standards used for other employment tests when they are used for personnel decision making, the authors assessed the criterion-related validity of P-O fit as a predictor of job performance and turnover. Meta-analyses resulted in estimated true criterion-related validities of .15 (k = 36, N = 5,377) for P-O fit as a predictor of job performance and .24 (k = 8, N = 2,476) as a predictor of turnover, compared with a stronger effect of .31 (k = 109, N = 108,328) for the more commonly studied relation between P-O fit and work attitudes. In contrast to the relations between P-O fit and work attitudes, the lower 95% credibility values for the job performance and turnover relations included zero. In addition, P-O fit's relations with job performance and turnover were partially mediated by work attitudes. Potential concerns pertaining to the use of P-O fit in employment decision making are discussed in light of these results.     

An examination of the predictive criterion-related validity of professional certification tests

This study examines the predictive criterion-related validity of a series of professional certification tests for water and wastewater management operators. Certification test data were obtained on 164 operators holding one of three jobs in water or wastewater management facilities. The certification test scores were broken down into four component scores and a total score. Criterion data consisted of performance evaluations obtained from the operator's supervisor, a self-rating from the operator, or both. Test scores were correlated with the job performance evaluations. The results indicated that scores on the certification tests were not related to rated job performance by job type, job level, source of performance evaluation, or component of job performance. The findings are discussed in the context of establishing an appropriate criterion against which to validate certification tests, and the practical problems of doing so in an applied setting.     

Risk factors for institutional misconduct among incarcerated women: an examination of the criterion-related validity of the Personality Assessment Inventory

Although widely researched in male offender samples, relatively little is known about the clinical utility of the Personality Assessment Inventory (PAI; Morey, 1991) among female prisoners. In this study, we examined the utility of various theoretically relevant PAI scales to predict 3 types of institutional misconduct (general, aggressive/defiant, and covert infractions) in a sample of 113 female inmates incarcerated for at least 1 year. The Antisocial Features (ANT) scale was the most consistent and effective predictor of misbehavior, with limited evidence to suggest that other PAI scales could demonstrate any incremental validity beyond this measure. More important, ANT continued to be associated with institutional misconduct even after controlling for criminal background variables such as prior convictions and a history of violence.     

Confirmatory factor analysis of the antisocial process screening device with a clinical inpatient population

Research linking psychopathic tendencies to childhood behavior problems is predicated on the assumption that the findings are based on using psychometrically sound measures of psychopathic tendencies. Although evaluated with community samples, the psychometric properties of such measures have not been evaluated with inpatient clinical populations thereby limiting conclusions. Hence, the goal of the present study was to evaluate the factor structure of a widely used measure-the Antisocial Process Screening Device (APSD)-with an inpatient population. Parents of 328 children (M age = 8.90 years) completed the APSD at the time of their child's admission for treatment. Although confirmatory factor analyses supported the three- and two-factor solutions proposed by the authors of the measure, the two-factor structure was more parsimonious. Factors of the two-factor model were also related to symptoms of childhood behavior disorders. The findings extend current research on the APSD's factor structure to an inpatient population.     

The construct validity of burnout

Considerable confusion exists in the burnout literature about the similarities and separateness among the constructs of burnout, depression and job satisfaction. This study examined the construct validity of burnout using Campbell & Fiske's multitrait-multimethod approach. The constructs of burnout, depression and orderliness (chosen as a discriminant construct) were assessed by three types of self-report instruments. Data were analysed for 181 females and 139 male faculty members who spent at least 50 per cent of their time in teaching. Results indicated that measures of burnout largely met Campbell & Fiske's criteria for construct validity: burnout measures possessed adequate to good internal reliability and convergent validity. However, burnout measures also correlated substantially with measures of depression. Results are discussed in terms of possible explanations for the substantial overlap that has been demonstrated among measures of burnout, depression and job satisfaction.     

The criterion-related validity of conscientiousness in personnel selection: A meta-analytic reality check

A key finding in personnel selection is the positive correlation between conscientiousness and job performance. Evidence predominantly stems from concurrent validation studies with incumbent samples but is readily generalized to predictive settings with job applicants. This is problematic because the extent to which faking and changes in personality affect the measurement likely vary across samples and study designs. Therefore, we meta-analytically investigated the relation between conscientiousness and job performance, examining the moderating effects of sample type (incumbent vs. applicant) and validation design (concurrent vs. predictive). The overall correlation of conscientiousness and job performance was in line with previous meta-analyses ($\overline{r}=.17,k=102,n=23,305$). In our analyses, the correlation did not differ across validation designs (concurrent: $\overline{r}=.18,k=78,n=19,132$; predictive: $\overline{r}=.15,k=24,n=4173$), sample types (incumbents: $\overline{r}=.18,k=92,n=20,808$; applicants: $\overline{r}=.14,k=10,n=2497$), or their interaction. Critically, however, our review revealed that only a small minority of studies (~12%) were conducted with real applicants in predictive designs. Thus, barely a fraction of research is conducted under realistic conditions. Therefore, it remains an open question if self-report measures of conscientiousness retain their predictive validity in applied settings that entail faked responses. We conclude with a call for more multivariate research on the validity of selection procedures in predictive settings with actual applicants.     

Brazilian-Portuguese empathy quotient: evidences of its construct validity and reliability

Empathy is the ability to read other's mind and understand their intentions. This paper examines the psychometric properties of the Brazilian-Portuguese version of a short form of the empathy quotient (EQ, 15 items). The EQ-15 was administered to 237 participants from the general population from Jo?o Pessoa, Brazil. Confirmatory factor analyses supported the tripartite model with cognitive, emotional and social empathy factors. In line with other studies, gender differences were only observed for the first two factors, with female participants scoring higher on both, which suggests that the social factor might not constitute a truly dimension of empathy. Strong evidence for convergent and discriminant validity was only observed for the cognitive factor. The poor psychometric parameters of the emotional and social factors are argued to reflect the complexity and contrasting ideas of their items. The possibility of elaborating specific items for the emotional and social factors is also discussed.     

The content validity of juvenile psychopathy: an empirical examination

This study examined the content validity of a juvenile psychopathy measure, the Childhood Psychopathy Scale (CPS; D. R. Lynam, 1997), based on a downward translation of an adult instrument, the Hare Psychopathy Checklist-Revised (PCL-R; R. D. Hare, 1991). The CPS was compared with two other indices of juvenile psychopathy: (a) an index derived from expert ratings and (b) an empirical index based on correlations with adult psychopathy. The 100 items of the Common Language Q-Sort (CLQ; A. Caspi et al., 1992) provided a common metric for the comparison. Psychopathy and personality were assessed at age 13 years with the mother-reported CPS and the CLQ. Psychopathy was assessed at age 24 years with the interviewer-rated Psychopathy Checklist: Screening Version (PCL:SV; S. D. Hart, D. N. Cox, & R. D. Hare, 1995). Data from over 250 participants of the middle sample of the Pittsburgh Youth Study were used to examine these relations. Item content analyses demonstrated considerable overlap among the three indices, indicating that the downward translation utilizes criteria similar to those of experts and the empirically-derived measure. In addition, these indices, even after removing overlapping items, demonstrated considerable convergence, also supporting the content validity of the downward translation. These results suggest that the downward translation method is adequate for understanding the juvenile psychopathy construct.     

The Kuwait University Anxiety Scale: reliability and criterion-related validity in Syrian college students

The Kuwait University Anxiety Scale was administered to 51 male and 56 female undergraduates in Syria. Respectively, the coefficients alpha for men, women, and the whole group were .91, .92, and .92; the test-retest rs after 14 to 21 days .88, .81, and .84; and the criterion-related validities .64, .62, and .64 against scores on Spielberger, et al.'s Trait Anxiety scale.     

An examination of the construct validity of the Social Anhedonia Scale

Two studies bearing on the construct validity of the Social Anhedonia Scale (SA) are reported. Study 1 reported convergent validity coefficients between the SA scale and numerous measures of traits reflecting a generalized attitude toward others (e.g., social interest and empathy) as well as several traits reflecting maladjustment without a social component as a test of discriminant validity (e.g., pessimism and rigidity). Study 1 found impressive convergent validity for the SA scale and mixed discriminant validity. Study 2 found that the SA scale was related to two dimensions of striving for security, one of which is strongly influenced by negative interpersonal tendencies, suggesting support for the validity of the SA scale. Implications for the measurement and concept of social anhedonia are presented.     

An examination of an appropriateness index and its effect on validity coefficients

Differences in validity coefficients were examined for both an appropriately responding group and an inappropriately responding group on four computer adaptive tests. Appropriate responding occurs when an examinee follows a clear pattern in answering test items relative to item difficulties. An Appropriateness Index score is computed based on Item Response Theory item parameters. The subjects were 155 college students from a medium sized midwestern university. The procedure required the subjects to complete four computer adaptive tests, as well as four criterion measures. High and low appropriateness groups were then formed by splitting the groups based on median values. The differences in validity coefficients were then examined for each of the four tests. All validity coefficients were smaller for the inappropriate groups, however only two of the four differences were significant. The appropriateness index was found to be an important factor in the consideration of validity coefficients.Portions of this paper were presented at the 34th Annual meeting of the Southeastern Psychological Association, March 31, 1988, New Orleans, Louisiana.Preparation of this article was supported in part by London House, Inc. We gratefully acknowledge the assistance of Yolanda Levy for the collection of data, and Debra Frost and Dr. John Jones for comments on a draft of this article.     

The Mosher Guilt Scales: a construct validity extension

The three subscales of the Mosher Guilt Scales (sex guilt, hostility guilt, and morality conscience guilt) and several personality and attitude measures were ad ministered to college subjects in an effort to examine the construct validity of the Mosher Guilt Scales. The variables that were compared with the guilt subscales were hostility, anxiety, religious orthodoxy, self-esteem, and the importance of religious and economic values in one's life. The results indicated that both the construct validity of the Mosher Guilt Scales and the need for the continued use of its three subscales were supported. Among the most crucial findings of the study were that both of the religiosity measures were positively correlated with all three guilt subscales, and that economic values were negatively correlated with both sex guilt and morality-conscience guilt.     

Trying to understand the different pieces of the construct validity puzzle of assessment centers: an examination of assessor and assessee effects

This study examined the effects of assessor-related factors (i.e., type of assessor) and assessee-related factors (i.e., type of assessee profile) on the construct validity of assessment center ratings. In particular, 3 types of assessors (26 industrial/organizational [I/O] psychologists, 20 managers, and 27 students), rated assessee performances that varied according to cross-exercise consistency (i.e., relatively inconsistent vs. relatively consistent) and dimension differentiation (relatively undifferentiated vs. relatively differentiated). Construct validity evidence was established for only one assessee profile and only in the I/O psychologist and managerial samples. More generally, these results indicate that 3 factors (poor design, assessor unreliability, and especially cross-situational inconsistent assessee performances) may explain why construct validity evidence is often not established in operational assessment centers.     

The construct validity of depressive personality disorder

This study examined the construct validity of depressive personality disorder (DPD: American Psychiatric Association, 1994). Adult psychiatric outpatients (N = 900) underwent comprehensive Axis I and II evaluations and provided data on 4,768 of their 1st-degree relatives. Despite modest overlap, DPD was not redundant with any Axis I or II disorder. Participants with DPD exhibited more Axis I and Axis II comorbidity, and greater psychosocial dysfunction, than participants without DPD. Relatives of participants with DPD had higher rates of mood disorders, alcohol abuse, and antisocial personality. Results are consistent with findings of several other similar investigations. The authors argue that DPD is a valid construct and should be conceptualized as a personality disorder as opposed to a mood disorder.     

The criterion-related validity of the Childhood Autism Rating Scale and the Autism Behavior checklist

The present study sought to examine the relationship between two popular instruments for screening autistic children: the Childhood Autism Rating Scale (CARS) and the Autism Behavior Checklist (ABC). The sample included 77 individuals. The records of 48 subjects contained firm diagnoses of autism, while the suggestion of autism was found in the remaining records. Correlations between the two scales ranged from –.16 to .73 (median=.39). The validity coefficient between the two total scores was .67. Using cutoff scores recommended by the authors, an analysis of the sensitivity of the two instruments was conducted. The CARS correctly identified 98% of the autistic subjects; it identified 69% of the possibly autistic as autistic. The ABC correctly identified 88% of the autistic subjects, while it identified 48% of those subjects considered possibly autistic as autistic. A phi coefficient was computed to estimate the degree of relationship between the nominal classifications produced by the two instruments. A moderate relationship was found, (r = .54). Implications of the results are discussed.     

The multidimensional nature of ageism: construct validity and group differences

The authors investigated the factor structure and construct validity of the Fraboni Scale of Ageism and the age and gender differences in ageism scores. Confirmatory factor analyses supported the multidimensional nature of FSA scores and generally corroborated the initial factor structure reported by M. Fraboni, with some notable exceptions. Essentially, the present findings were aligned with theoretical models of ageism that emphasize both cognitive facets and affective facets. That is, on the basis of their factor analytic findings, the authors redefined Fraboni's original factors of Antilocution, Avoidance, and Discrimination as Stereotypes, Separation, and Affective Attitudes, respectively, because of the clustering of items within factors. The revised 3-factor structure accounted for 36.4% of the variance in FSA scores. FSA factor scores significantly related to other scores from other measures of age-related attitudes, with higher correlations among factors that were similar in terms of their cognitive nature versus their affective nature. Finally, younger individuals and men had significantly higher ageism scores on the FSA than older individuals and women. The authors discussed the importance of adequately assessing ageism, with particular emphasis devoted to the understanding of age bias.     

The construct and criterion validity of emotional intelligence and its potential utility for management studies

In this study, the authors reviewed the definition of emotional intelligence (EI) and argued that El is conceptually distinct from personality. In Study 1, the authors showed that EI was related to yet distinct from personality dimensions and that it had incremental predictive power on life satisfaction. The authors examined the construct validity of self-reports and others' ratings of EI using two samples in Study 2. In a student sample, parents' ratings explained additional variance in the students' life satisfaction and feelings of powerlessness after controlling for the Big Five personality dimensions. In the work sample, peer ratings were found to be significant predictors of job performance ratings provided by supervisors after controlling for the Big Five personality dimensions. Other implications for future research on EI are discussed.