Implications of direct and indirect range restriction for meta-analysis methods and findings

Range restriction in most data sets is indirect, but the meta-analysis methods used to date have applied the correction for direct range restriction to data in which range restriction is indirect. The authors show that this results in substantial undercorrections for the effects of range restriction, and they present meta-analysis methods for making accurate corrections when range restriction is indirect. Applying these methods to a well-known large-sample empirical database, the authors estimate that previous meta-analyses have underestimated the correlation between general mental ability and job performance by about 25%, indicating that this is potentially an important methodological issue in meta-analysis in general.     

INCREASED ACCURACY FOR RANGE RESTRICTION CORRECTIONS: IMPLICATIONS FOR THE ROLE OF PERSONALITY AND GENERAL MENTAL ABILITY IN JOB AND TRAINING PERFORMANCE

A recently developed procedure produces substantial improvements in the accuracy of corrections for range restriction and reveals that predictive validities of employment selection methods are higher than previously thought. This study applied this procedure to meta‐analytic validities of 2 personality measures (Conscientiousness and Emotional Stability) and general cognitive ability measures. Results show that the increases in validity estimates are greater for cognitive ability than for personality, reducing the relative validity of personality. In addition, the incremental validity of these personality measures over that provided by cognitive ability is smaller than previously estimated. Path analyses based on the more accurate data reveal relatively smaller path coefficients from these personality traits to job and training performance. These findings have implications for theories of job performance and for the practical value in selection of personality measures relative to cognitive ability measures.     

Cohen's d Corrected for Case IV Range Restriction: A More Accurate Procedure for Evaluating Subgroup Differences in Organizational Research

Organizational and staffing researchers are often interested in evaluating whether subgroup differences exist (e.g., between Caucasian and African‐American individuals) on predictors of job performance. To investigate subgroup differences, researchers often will collect data from current employees to make inferences about subgroup differences among job applicants. However, the magnitude of subgroup differences (i.e., Cohen's d) within incumbent samples may be different (i.e., smaller) than the magnitude of subgroup differences in applicant samples because selection of applicants typically reduces the variance of scores on the predictors (i.e., because lower scoring applicants are not selected). If researchers seek to generalize a d value in an incumbent sample to the applicant population, they may use Bobko, Roth, and Bobko's (correcting the effect size of d for range restriction and unreliability, 2001) Case II or III correction. By extension, Hunter, Schmidt, and Le (implications of direct and indirect range restriction for meta‐analysis methods and findings, 2006) have proposed a Case IV correction, which is more realistic than Bobko et al.'s approach. Therefore, this paper develops a Case IV correction for d (i.e., d_c4). The simulation results showed that the d_c4 was generally accurate across 6,000 simulation conditions. Moreover, 2 published datasets were reanalyzed to show the influence of the Case IV correction on d. In addition, implications and future directions of the d_c4 are discussed.     

THE EFFECTS OF RANGE RESTRICTION ON ESTIMATES OF CRITERION INTERRATER RELIABILITY: IMPLICATIONS FOR VALIDATION RESEARCH

This paper explores the implications of the fact that range restriction affects reliability coefficients as well as validity coefficients. Of concern was the possibility that, if range restriction produced an underestimate of criterion reliability, the use of such an estimate in correcting observed validity coefficients for attenuation could result in overestimating population validity, lb address this issue, 3 scenarios were considered: truncation on the predictor, truncation on an unobserved variable, and truncation on the criterion. The 3 scenarios were found to have substantial differences in their consequences for reliability and validity estimation. Although range restriction did result in an underestimate of criterion reliability, it is generally not the case that these underestimates will result in overestimates of validity if researchers make effective use of available information about the presence and degree of range restriction.     

Correcting correlation coefficients for group heterogeneity when one variable is a dichotomy

A method is illustrated for estimating correlation coefficients and mean criterion scores in a full-range population from bivariate distributions available in a selected sample when the criterion is a dichotomy. The proposed method requires only the assumptions needed for use of formulas for correcting correlation coefficients for restriction of range when both variables are continuous and is suitable for use when restriction is due to either direct or indirect selection.The research reported in this paper was sponsored by the 6570th Personnel Research Laboratory, AMD, under AFSC Project 7717.     

Correcting for indirect range restriction in meta-analysis: testing a new meta-analytic procedure

Using computer simulation, the authors assessed the accuracy of J. E. Hunter, F. L. Schmidt, and H. Le's (2006) procedure for correcting for indirect range restriction, the most common type of range restriction, in comparison with the conventional practice of applying the Thorndike Case II correction for direct range restriction. Hunter et al.'s procedure produced more accurate estimates of both the mean and standard deviation in meta-analysis than the conventional procedure. Even when its key assumption that the effect of selection on a 3rd variable is fully mediated by the independent variable was violated, Hunter et al.'s procedure was still relatively more accurate than the conventional procedure. When applied to data from a previously published meta-analysis, the new procedure yielded results that led to different substantive conclusions.     

Using individual differences to predict job performance: correcting for direct and indirect restriction of range

The present study investigates the relationship between individual differences, indicated by personality (FFM) and general mental ability (GMA), and job performance applying two different methods of correction for range restriction. The results, derived by analyzing meta-analytic correlations, show that the more accurate method of correcting for indirect range restriction increased the operational validity of individual differences in predicting job performance and that this increase primarily was due to general mental ability being a stronger predictor than any of the personality traits. The estimates for single traits can be applied in practice to maximize prediction of job performance. Further, differences in the relative importance of general mental ability in relation to overall personality assessment methods was substantive and the estimates provided enables practitioners to perform a correct utility analysis of their overall selection procedure.     

Alternative Approaches for Measuring Values: Direct and Indirect Assessments in Performance Prediction

Two general approaches have been used to measure human values. In one approach, a direct approach, people are asked to endorse value statements. In the other approach, an indirect approach, people are asked to make choices indicative of their values. The intent of the current study was to compare these two general approaches to the measurement of values. Initially, 195 undergraduates were asked to complete both a direct measure and an indirect measure of their values before starting work on three performance tasks: an entrepreneurial task, a consulting task, and a marketing task. When scores on these performance tasks were regressed on the value measures, it was found that both types of measures yielded effective prediction. The indirect measures, however, appeared to yield better prediction and better discrimination of cross-task performance differences than the direct measures. The implications of these findings with respect to the identification and assessment of values are discussed.     

An Improved Correction for Range Restricted Correlations Under Extreme,Monotonic Quadratic Nonlinearity and Heteroscedasticity

Standardized tests are frequently used for selection decisions, and the validation of test scores remains an important area of research. This paper builds upon prior literature about the effect of nonlinearity and heteroscedasticity on the accuracy of standard formulas for correcting correlations in restricted samples. Existing formulas for direct range restriction require three assumptions: (1) the criterion variable is missing at random; (2) a linear relationship between independent and dependent variables; and (3) constant error variance or homoscedasticity. The results in this paper demonstrate that the standard approach for correcting restricted correlations is severely biased in cases of extreme monotone quadratic nonlinearity and heteroscedasticity. This paper offers at least three significant contributions to the existing literature. First, a method from the econometrics literature is adapted to provide more accurate estimates of unrestricted correlations. Second, derivations establish bounds on the degree of bias attributed to quadratic functions under the assumption of a monotonic relationship between test scores and criterion measurements. New results are presented on the bias associated with using the standard range restriction correction formula, and the results show that the standard correction formula yields estimates of unrestricted correlations that deviate by as much as 0.2 for high to moderate selectivity. Third, Monte Carlo simulation results demonstrate that the new procedure for correcting restricted correlations provides more accurate estimates in the presence of quadratic and heteroscedastic test score and criterion relationships.     

Methods of correcting correlations for indirect restriction of range with non-interval data

A derivation is presented for correcting correlations for indirect restriction of range when correlations between one selection variable and non-interval measures are involved. Following a systematic investigation of special cases, formulas are given for practical application in validation work. Procedures for the case with several selection variables are briefly discussed.This study was initiated when the author was with the Institute of Military Psychology, Sweden.     

Effects of initial success or failure in a stressful or relaxed environment on subsequent task performance

Forty male and 40 female undergraduate subjects worked on anagram problems in either a stressful or a relaxed environment after having experienced either success or failure on a similar task. Internal arousal, which was expected to be minimal following success in a relaxed setting and maximal following failure in the stressful environment, was assumed to bear an inverted U-shaped relation to performance. Consistent with the implications of these assumptions, achievement in a stressful setting was lower following initial failure than following initial success, while initial failure in a relaxed environment facilitated subsequent performance. The interpretation of these results was supported by the effects of the experimental variables on indirect measures of arousal as well as by the relationship between direct measures of arousal and task performance. In addition, the achievement of female subjects tended to be facilitated by failure while the performance of males was consistently debilitated by it.     

Direct versus Indirect Well-Being Measures: Using Partially Structured Stimuli to Evaluate Well-Being

Given the important limitations of direct (explicit) measures, the study of indirect (implicit) measures of psychological constructs has received increased attention of social researchers. However, in the field of well-being the measures that have traditionally been used have tried to capture people’s well-being in an explicit and direct way, with some exceptions that are based mainly on adaptations of the Implicit Association Test (IAT). Given the critics that the IAT has received, the first aim of the present research was to develop a new life satisfaction measure using partially structured stimuli (PSWBS). The second goal was to examine the relation between direct and indirect well-being measures. In the first study, PSWBS showed good internal consistency, test–retest reliability and construct validity. Factorial analyses indicated that this measure acted as an indirect measure. In addition, direct and indirect well-being showed a significant but weak relation. To explain this result, in a second study we analyzed the moderating role of social comparison orientation on the relation between direct and indirect well-being measures. As expected, more social comparison tendency implied a stronger relation. In a third study we analyzed the criterion validity of both kind of measures. The PSWBS, the Personal Wellbeing Index-Adults and the Satisfaction With Life Scale significantly correlated with a criterion validity coefficient in terms of a semi-structured interview made by an external experimenter. Finally, in a fourth experiment we analyzed the influence of social desirability on direct and indirect measures of well-being. In contexts of high desirability, partially structured stimuli seemed to be more resistant to response factors (e.g., control of impressions) than traditional direct measures.

     

Moving Forward Indirectly: Reanalyzing the validity of employment interviews with indirect range restriction methodology

This study provides updated estimates of the criterion‐related validity of employment interviews, incorporating indirect range restriction methodology. Using a final dataset of 92 coefficients (N = 7,389), we found corrected estimates by structural level of .20 (Level 1), .46 (Level 2), .71 (Level 3), and .70 (Level 4). The latter values are noticeably higher than in previous interview meta‐analyses where the assumption was made that all restriction was direct. These results highlight the importance of considering indirect range restriction in selection. However, we found a number of studies involving both indirect and direct restriction, which calls into question the viability of assuming all restriction is now indirect. We found preliminary empirical support for correction of one of these multiple restriction patterns, indirect then direct.     

Recall versus recognition: a methodological note

Some current investigators have made direct comparisons of recognition performance with cued recall. In other instances, indirect comparisons of recognition and recall have been made by observing the difference in recognition performance with different types of target material, for example, high- and low-frequency words; this difference has been then compared with the difference obtained when recall is used to measure retention of the same type of target materials. Finally, some experimenters have examined how recognition performance is related to the type of target material that is used, for example, words, pictures, and so on. The results obtained from all of these studies have been used to make inferences about the nature of the memorial process. We have attempted to call attention to the generally accepted but neglected finding that performance on any recognition test can be manipulated by changing the characteristics of the distractors, thus casting doubt on the generality of the experimental findings as well as their theoretical implications.     

The relation between oral reading miscue patterns and comprehension: A test of the relative explanatory power of psycholinguistic and interactive views of reading

The use of oral reading to assess silent reading ability has gained wide acceptance despite the lack of definitive evidence that the two processes are similar. Moreover, there are a number of theoretical positions that run counter to assumptions underlying oral reading error analysis as a measure of silent reading competency. This study examined the relation between multiple oral reading miscue measures and postreading comprehension performance. Correlations between the oral reading and comprehension measures indicated a negative relation between oral reading and recall following the reading of familiar passages and a positive relation between oral reading and an inferential comprehension measure following the reading of unfamiliar passages. In addition, when the subjects read disrupted passages, their attempts at correcting incongruous errors and their self-report interviews indicated that their oral reading was not related to their performance on comprehension product measures. The results, which are discrepant with previous research relating oral reading to comprehension, are discussed in terms of a comparison between psycholinguistic and interactive models of reading comprehension.     

More than g: Evidence for the Incremental Validity of Performance-Based Assessments for Predicting Training Performance

Past research has consistently shown that tests measuring specific cognitive abilities provide little if any incremental validity over tests of general mental ability when predicting performance on the job. In this study, we suggest that the seeming lack of incremental validity may have been due to the type of content that has traditionally been assessed. Therefore, we hypothesised that incremental validity can be obtained using specific cognitive abilities that are less highly correlated with g and are matched to the tasks performed on the job. To test this, we examined a recently developed performance-based measure that assesses a number of cognitive abilities related to training performance. In a sample of 310 US Navy student pilots, results indicated that performance-based scores added sizeable incremental validity to a measure of g. The significant increases in R² ranged from .08 to .10 across criteria. Similar results were obtained after correcting correlations for range restriction, though the magnitude of incremental validity was slightly smaller (ΔR² ranged from .05 to .07).     

Estimating Coefficients of Equivalence and Stability for Job Performance Ratings: The importance of controlling for transient error on criterion measurement

Previous research on measurement error in job performance ratings estimated reliability using coefficients: alpha, test–retest, and interrater correlation. None of these three coefficients control for the four main sources of error in performance ratings. For this reason, coefficient of equivalence and stability (CES) has been suggested as the ideal estimate of reliability. This article presents the estimates of CES for a time interval of 1, 2, and 3 years. The values obtained for a single rater were .51, .48, and .44, respectively. For two raters, the values were .59, .55, and .51. The findings suggest that previous reliability estimates based on alpha, test–retest, and interrater coefficients overestimated the reliability of job performance ratings. In the present study, the interrater coefficient overestimates reliability by 13.6–25.4% for an interval time of 1–3 years, as it does not control for transient error. Results also showed that the importance of transient error increases as the length of the interval between the measures increases. Based on the results, it is suggested that corrected validities based on interrater reliability underestimate the magnitude of the validity. The implications of these findings for future efforts to estimate criterion reliability and predictor validity are discussed.     

Item Variances and Median Splits: Some Encouraging and Reassuring Findings

ABSTRACT The article by Bissonnette, Ickes, Bernstein, and Knowles (1990) presented in this issue contains a series of simulations which vary widely in the degree to which they resemble the parameters that have been found in actual data. Their conclusions feature warnings based on simulations containing large confounds between trait extremity and item variance. In real data, however, the corresponding confounds typically have been small. When Bissonnette et al. used appropriately small confounds in their simulations, their results showed that median split techniques failed to inflate Type I error rates. The constructive implications of Bissonnette et al.'s simulations indicate that moderator-variable researchers should always perform a simple check for range restriction and for variance-extremity confounds, and that there is no danger of spurious conclusions as long as these checks conform to what has typically been found with real data.     

THE RELATIONSHIP BETWEEN WORK EXPERIENCE AND JOB PERFORMANCE: A CONCEPTUAL AND META-ANALYTIC REVIEW

A gap in the conceptual development of the work experience construct was addressed by creating a framework specifying two dimensions along which work experience measures can vary. The dimensions of measurement mode (amount, time, and type) and level of specificity (task, job, organizational) formed nine separate categories of measures of work experience. The usefulness of the conceptual framework was tested by conducting a meta-analytic review of the existing literature on work experience using the dimensions in the framework as potential moderators of the experience-performance relationship. Results of the meta-analysis ( N = 25,911; K = 44) revealed an estimated population correlation of 27 between experience and performance after correcting for sampling error and criterion unreliability. In addition, the results showed that amount ( M = .43) and task-level ( M = .41) measures of work experience had the highest correlations with measures of job performance. In addition, work experience had the highest correlations with hard (e.g. work samples) as opposed to soft (e.g. ratings) measures of job performance ( M = .39 vs. M = .24). Implications and directions for future research are discussed.