APPLYING RANGE RESTRICTION CORRECTIONS USING PUBLISHED NORMS: THREE CASE STUDIES

This study evaluated the accuracy of range restriction corrections. Initial range restriction corrections used published norms as a proxy for unrestricted applicant data. The accuracy of the initial correction was evaluated using two large independent follow-up samples ( n = 983 and n = 1,389). The author found the initial norm-based correction to be very consistent with empirical findings. The application of these findings to range restriction corrections with customized (i.e., non-published) selection tests is also discussed.     

A NOTE ON DIRECT RANGE RESTRICTION CORRECTIONS IN UTILITY ANALYSIS

Brogden's equations for employment test utility analysis have been used in conjunction with direct range restriction correction formulas, because the data used typically come from selected applicant (incumbent) samples. We show that the use of direct range restriction corrections in utility analysis is not needed. Identical results can be obtained using a simpler equation that employs a regression coefficient based on the employee (i.e., restricted or job incumbent) population. This utility estimation procedure, which does not employ range restriction corrections, is derived and discussed. We recommend that this new equation be used in the practice of selection utility analysis because it not only provides identical utility estimates to those obtained using current equations that employ direct range restriction correction formulas, but also is computationally simpler.     

Cohen's d Corrected for Case IV Range Restriction: A More Accurate Procedure for Evaluating Subgroup Differences in Organizational Research

Organizational and staffing researchers are often interested in evaluating whether subgroup differences exist (e.g., between Caucasian and African‐American individuals) on predictors of job performance. To investigate subgroup differences, researchers often will collect data from current employees to make inferences about subgroup differences among job applicants. However, the magnitude of subgroup differences (i.e., Cohen's d) within incumbent samples may be different (i.e., smaller) than the magnitude of subgroup differences in applicant samples because selection of applicants typically reduces the variance of scores on the predictors (i.e., because lower scoring applicants are not selected). If researchers seek to generalize a d value in an incumbent sample to the applicant population, they may use Bobko, Roth, and Bobko's (correcting the effect size of d for range restriction and unreliability, 2001) Case II or III correction. By extension, Hunter, Schmidt, and Le (implications of direct and indirect range restriction for meta‐analysis methods and findings, 2006) have proposed a Case IV correction, which is more realistic than Bobko et al.'s approach. Therefore, this paper develops a Case IV correction for d (i.e., d_c4). The simulation results showed that the d_c4 was generally accurate across 6,000 simulation conditions. Moreover, 2 published datasets were reanalyzed to show the influence of the Case IV correction on d. In addition, implications and future directions of the d_c4 are discussed.     

Validity of Graduate Record Examinations (GRE) general test scores for admissions to colleges of veterinary medicine

From the standpoint of test validation, veterinary medicine provides both a unique context in which to study the validity of Graduate Record Examinations (GRE) test scores and a singular opportunity to address the shortcomings typical of many GRE validity studies. This article documents a study of the validity of the GRE General Test for predicting 1st-year grade averages in a comprehensive sample of veterinary medical colleges. For each of 16 veterinary medical colleges, statistical corrections were applied to correct for the effects of range restriction in the predictors and unreliability of the criterion. When fully corrected for both range restriction and unreliability, the resulting validity coefficients were, on average,.53 for the combination of all 3 GRE General Test scores,.59 for undergraduate grade point average, and.71 for GRE scores and undergraduate grade point average together.     

A theoretical expectancy table calculator

A program for calculating theoretical expectancy tables is described. These tables are useful for communicating the meaning of correlation coefficients, since they display the relationship in a simple way. The program allows users to go beyond empirical expectancy tables, which are routinely calculated, by allowing the table to reflect corrections for effects such as unreliability and restriction of range.     

The criterion-related validity of integrity tests: an updated meta-analysis

Integrity tests have become a prominent predictor within the selection literature over the past few decades. However, some researchers have expressed concerns about the criterion-related validity evidence for such tests because of a perceived lack of methodological rigor within this literature, as well as a heavy reliance on unpublished data from test publishers. In response to these concerns, we meta-analyzed 104 studies (representing 134 independent samples), which were authored by a similar proportion of test publishers and non-publishers, whose conduct was consistent with professional standards for test validation, and whose results were relevant to the validity of integrity-specific scales for predicting individual work behavior. Overall mean observed validity estimates and validity estimates corrected for unreliability in the criterion (respectively) were .12 and .15 for job performance, .13 and .16 for training performance, .26 and .32 for counterproductive work behavior, and .07 and .09 for turnover. Although data on restriction of range were sparse, illustrative corrections for indirect range restriction did increase validities slightly (e.g., from .15 to .18 for job performance). Several variables appeared to moderate relations between integrity tests and the criteria. For example, corrected validities for job performance criteria were larger when based on studies authored by integrity test publishers (.27) than when based on studies from non-publishers (.12). In addition, corrected validities for counterproductive work behavior criteria were larger when based on self-reports (.42) than when based on other-reports (.11) or employee records (.15).     

FACTORS AFFECTING VALIDITY OF A REGIONALLY ADMINISTERED ASSESSMENT CENTER

Validity coefficients for a single assessment center implemented in multiple locations are presented. Correction for range restriction, criterion unreliability, and sampling error did not account for a large portion of the variability in these validity coefficients. The type of assessor used, the center's administrative arrangement, and prior assessor-assessee contact moderated these validities. It is suggested that the quality of the implementation of a selection procedure when there is local latitude in its implementation is an important factor in determining the procedure's effectiveness.     

How many parameters can a model have and still be testable?

A standard rule of thumb states that a model has too many parameters to be testable if and only if it has at least as many parameters as empirically observable quantities. We argue that when one asks whether a model has too many parameters to be testable, one implicitly refers to a particular type of testability, which we call quantitative testability. A model is defined to be quantitatively testable if the model's predictions have zero probability of being correct by chance. Next, we propose a new rule of thumb, based on the rank of the Jacobian matrix of a model (i.e., the matrix of partial derivatives of the function that maps the model's parameter values onto predicted experimental outcomes). According to this rule, a model is quantitatively testable if and only if the rank of the Jacobian matrix is less than the number of observables. (The rank of his matrix can be found with standard computer algorithms.) Using Sard's theorem, we prove that the proposed new rule of thumb is correct provided that certain “smoothness” conditions are satisfied. We also discuss the relation between quantitative testability and reparameterization, identifiability, and goodness-of-fit testing.     

Pro- and antisaccades: dissociating stimulus and response influences the online control of saccade trajectories

The authors examined whether the diminished online control of antisaccades is related to a trade-off between movement planning and control or the remapping of target properties to a mirror-symmetrical location (i.e., vector inversion). Pro- and antisaccades were examined in a standard no-delay schedule wherein target onset served as the movement imperative and a delay cuing schedule wherein responses were initiated 2,000 ms following target onset. Importantly, the delay cuing schedule was employed to equate pro- and antisaccade reaction times. Online control was evaluated by indexing the strength of trajectory amendments at normalized increments of movement time. Antisaccades exhibited fewer online corrections than prosaccades, and this result was consistent across cuing schedules. Thus, the diminished online control of antisaccades cannot be tied to a trade-off between movement planning and control. Rather, the authors propose that the intentional nature of dissociating stimulus and response (i.e., vector inversion) engenders a slow mode of cognitive control that is not optimized for fast oculomotor corrections.     

Pro- and Antisaccades: Dissociating Stimulus and Response Influences the Online Control of Saccade Trajectories

The authors examined whether the diminished online control of antisaccades is related to a trade-off between movement planning and control or the remapping of target properties to a mirror-symmetrical location (i.e., vector inversion). Pro- and antisaccades were examined in a standard no-delay schedule wherein target onset served as the movement imperative and a delay cuing schedule wherein responses were initiated 2,000 ms following target onset. Importantly, the delay cuing schedule was employed to equate pro- and antisaccade reaction times. Online control was evaluated by indexing the strength of trajectory amendments at normalized increments of movement time. Antisaccades exhibited fewer online corrections than prosaccades, and this result was consistent across cuing schedules. Thus, the diminished online control of antisaccades cannot be tied to a trade-off between movement planning and control. Rather, the authors propose that the intentional nature of dissociating stimulus and response (i.e., vector inversion) engenders a slow mode of cognitive control that is not optimized for fast oculomotor corrections.     

Predictive Validity of the Air Force Officer Qualifying Test for Non-Rated Officer Specialties

The Air Force Officer Qualifying Test (AFOQT) is used to qualify applicants for officer commissioning and aircrew training programs. The current study examined its predictive validity for 14 officer technical training courses for which there are no additional AFOQT minimum qualification requirements beyond those for officer commissioning. Sample sizes ranged from 16 to 2,190 with a mean and median size of 753 and 319 officers. Ninety percent (63 of 70) of the observed correlations between the AFOQT composites and average technical training grades were statistically significant. Meta-analyses were conducted to determine whether the AFOQT validities were generalizable across training specialties. Analyses were done on the observed data, after correction for range restriction, and after correction for both range restriction and criterion unreliability. The pattern of validities was similar for all three meta-analyses. The lower bound of the 95% confidence interval and the 95% credibility interval around the weighted mean validities were greater than zero for all five AFOQT composites supporting its utility for making personnel selection decisions for these jobs. The Verbal composite had the lowest and the Academic Aptitude composite had the highest weighted mean validity. The weighted mean of the validity coefficients across training specialties ranged from .261 to .326 for the observed data, .322 to .387 for the range-restriction corrected data, and .360 to .433 for the fully corrected data. Additional efforts are required to examine the generalizability of the results for a broader range of occupational specialties and to set minimum qualifying scores.     

Implications of direct and indirect range restriction for meta-analysis methods and findings

Range restriction in most data sets is indirect, but the meta-analysis methods used to date have applied the correction for direct range restriction to data in which range restriction is indirect. The authors show that this results in substantial undercorrections for the effects of range restriction, and they present meta-analysis methods for making accurate corrections when range restriction is indirect. Applying these methods to a well-known large-sample empirical database, the authors estimate that previous meta-analyses have underestimated the correlation between general mental ability and job performance by about 25%, indicating that this is potentially an important methodological issue in meta-analysis in general.     

Primacy of information about means selection over outcome selection in goal attribution by infants

It has been shown that, when observing an action, infants can rely on either outcome selection information (i.e., actions that express a choice between potential outcomes) or means selection information (i.e., actions that are causally efficient toward the outcome) in their goal attribution. However, no research has investigated the relationship between these two types of information when they are present simultaneously. In an experiment that addressed this question directly, we found that when outcome selection information could disambiguate the goal of the action (e.g., the action is directed toward one of two potential targets), but means selection information could not (i.e., the action is not efficiently adjusted to the situational constraints), 7- and 9-month-old infants did not attribute a goal to an observed action. This finding suggests that means selection information takes primacy over outcome selection information. The early presence of this bias sheds light on the nature of the notion of goal in action understanding.     

The elusive matching bias effect in the disjunctive selection task

When reasoning with conditional statements (i.e., if [not] p then [not] q), for example when solving Wason's selection task, subjects tend to display matching bias: Options which match the entities named in the rule tend to be selected irrespective of whether this is logically appropriate. Recently, there have been suggestions that the underlying causes of matching bias reflect a general phenomenon that applies to many types of logical rule, not just conditionals. A study is reported in which performance is investigated for selection tasks with categorical or disjunctive rules. Although matching bias was clearly present for categorical rules, inverted matching bias was identified for disjunctive rules, calling into question the generality of the phenomenon and its explanations. In addition, performance at one task was not correlated with performance at the other, calling into question recent cognitive capacity accounts of selection task performance.     

Reliability of the Go/No Go Association Task

The Go/No Go Association Task (GNAT; Nosek & Banaji, 2001) is an implicit measure with broad application in social psychology. It has several conceptual strengths to recommend it over other implicit methods, but the belief that it has poor reliability coupled with the absence of a method for calculating this important psychometric property has hindered its wider acceptance and use. Using data obtained from six GNAT studies covering a wide range of content areas, Study 1 compares the properties of different methods for estimating reliability of the GNAT. Study 2 demonstrates a resampling procedure to investigate how reliability varies as a function of block length. Study 1 shows that with appropriately chosen stimuli the GNAT can be a very reliable measure, while Study 2 indicates that as an empirical rule of thumb 50 to 80 trials per block should yield adequate to very good reliability. However, researchers are urged to calculate their own reliability coefficients, to this end we discuss GNAT design issues and provide procedures for calculating GNAT reliability which we hope will enhance the utility of the GNAT as a measure and promote its use in studying implicit cognition.     

INCREASING THE ACCURACY OF CORRECTIONS FOR RANGE RESTRICTION: IMPLICATIONS FOR SELECTION PROCEDURE VALIDITIES AND OTHER RESEARCH RESULTS

The common practice in meta-analyses and in individual studies of correcting for direct range restriction even though range restriction is actually indirect has long been known to lead to undercorrection, but this error has been assumed to be small. Using validity generalization data sets for 4 jobs, this study calibrated this error by comparing meta-analysis results based on corrections for direct range restriction with the more accurate results from a recently developed method of correcting for indirect range restriction. It was found that, on average, correction for direct range restriction resulted in substantial underestimation of operational validities for both job performance measures (21%) and training performance measures (28%). In addition, 90% credibility values were on average underestimated by 38%–40%. In addition to the implications for personnel selection, these findings suggest that similar underestimation of important relationships has occurred in other areas of research, with potential implications for theory development.     

How to Assess a Model's Testability and Identifiability

Formal definitions are given of the following intuitive concepts: (a) A model is quantitatively testable if its predictions are highly precise and narrow. (b) A model is identifiable if the values of its parameters can be ascertained from empirical observations. (c) A model is redundant if the values of some parameters can be deduced from others or if the values of some observables can be deduced from others. Various rules of thumb for nonredundant models are examined. The Counting Rule states that a model is quantitatively testable if and only if it has fewer parameters than observables. This rule can be safely applied only to identifiable models. If a model is unidentifiable, one must apply a generalization of the Counting Rule known as the Jacobian Rule. This rule states that a model is quantitatively testable if and only if the maximum rank (i.e., the number of linearly independent columns) of its Jacobian matrix (i.e., the matrix of partial derivatives of the function that maps parameter values to the predicted values of observables) is smaller than the number of observables. The Identifiability Rule states that a model is identifiable if and only if the maximum rank of its Jacobian matrix equals the number of parameters. The conclusions provided by these rules are only presumptive. To reach definitive conclusions, additional analyses must be performed. To illustrate the foregoing, the quantitative testability and identifiability of linear models and of discrete-state models are analyzed. Copyright 2000 Academic Press.     

Trying to understand the different pieces of the construct validity puzzle of assessment centers: an examination of assessor and assessee effects

This study examined the effects of assessor-related factors (i.e., type of assessor) and assessee-related factors (i.e., type of assessee profile) on the construct validity of assessment center ratings. In particular, 3 types of assessors (26 industrial/organizational [I/O] psychologists, 20 managers, and 27 students), rated assessee performances that varied according to cross-exercise consistency (i.e., relatively inconsistent vs. relatively consistent) and dimension differentiation (relatively undifferentiated vs. relatively differentiated). Construct validity evidence was established for only one assessee profile and only in the I/O psychologist and managerial samples. More generally, these results indicate that 3 factors (poor design, assessor unreliability, and especially cross-situational inconsistent assessee performances) may explain why construct validity evidence is often not established in operational assessment centers.     

LinkedIn as a new selection method: Psychometric properties and assessment approach

Various surveys suggest LinkedIn is used as a screening and selection tool by many hiring managers. Despite this widespread use, fairly little is known about whether LinkedIn meets established selection criteria, such as reliability, validity, and legality (i.e., no adverse impact). We examine the properties of LinkedIn-based assessments in two studies. Study 1 shows that raters reach acceptable levels of consistency in their assessments of applicant skills, personality, and cognitive ability. Initial ratings also correlate with subsequent ratings done 1-year later (i.e., demonstrating temporal stability), with slightly higher correlations when profile updates are taken into account. Initial LinkedIn-based ratings correlate with self-reports for more visible skills (leadership, communication, and planning) and personality traits (Extraversion), and for cognitive ability. LinkedIn-based hiring recommendations are positively associated with indicators of career success. Potential adverse impact is also limited. Profiles that are longer, include a picture, and have more connections are rated more positively. Some of those features are valid cues to applicants’ characteristics (e.g., applicants high on Conscientiousness have longer profiles). In Study 2, we show that an itemized LinkedIn assessment is more effective than a global assessment. Implications of these findings for selection and future research are discussed.     

“Be careful what you wish for…”: A quantitative and qualitative investigation of “fata1 attractions”

This study uses questionnaire data from 301 terminated romantic relationships to investigate hypotheses concerning the conditions under which a quality that is initially appealing in a partner is later disliked (i.e., a “fatal attraction”). Individuals are expected to be prone to fatal attractions when they are attracted to a quality in a romantic partner that is “different” in any of three ways: (1) different from the individual's own qualities (i.e., dissimilar); (2) different from average (i.e., extreme, or unique); or (3) different from normative expectations (i.e., gender atypical). Such attractions are less likely, however, when individuals are drawn to similarity in another. Findings from a logistic regression analysis provide support for four of the five hypotheses. Fatal attractions are significantly more likely when an individual is drawn to partner characteristics seen either as “dissimilar” from her or his own, as “unique,” or as “extreme” in nature. When these characteristics are viewed as “similar,” disenchantment is less likely. Fatal attractions are neither more, nor less, probable, however, when partner qualities contradict gender stereotypes. A qualitative analysis of open-ended questionnaire responses provides further support for these attraction patterns. The findings have implications for diverse theories, including those of mate selection and dialectics.