Scoring Biodata: Is it rational to be quasi‐rational?

Five quasi‐rational biodata scales were developed by empirically keying biodata items to predict scores on a measure of the Big Five dimensions of personality. The criterion‐related validities of the quasi‐rational scales were compared to empirical and rational biodata keying methods using supervisory ratings of job performance as the criterion. Empirical keying outperformed the quasi‐rational and rational methods (which had similar validities).     

Construct-oriented Biodata Development for Selection to a Differentiated Performance Domain

This study describes the development of a multidimensional biodata form which used explicit constructs to guide item generation and rational scale development, construct validation, criterion measurement and empirical keying. These constructs were goal-orientation, teamwork, customer service, resourcefulness, learning ability and leadership. Exploratory and confirmatory factor analyses in both applicant and incumbent samples were used to identify and test the model which included the thirteen, more differentiated rational scales relating to these six, broader constructs. Empirical keying of the rationally developed scales was conducted against criterion construct scales conceptually related to each predictor construct. Empirical keying at the item level was found to result in higher validities and cross-validities than either empirical keying at the scale level, or rational keying. The item keyed instrument also demonstrated incremental validity over a test of cognitive ability for specific work performance domains as well as overall work performance.     

Empirical Keying of Biographical Data: Cross-Validity as a Function of Scaling Procedure and Sample Size

In recent years, the military has devoted considerable effort to the develop- ment of empirically keyed biodata instruments for use in selection. Although studies using empirical keying procedures are common in the personnel selection literature, relatively few studies have compared these procedures. Using data collected from Naval Academy midshipmen, we compared nine empirical keying procedures: vertical percent (five strategies), horizontal percent, mean criterion, phi coefficient, and rare response. For each keying procedure, five different sample sizes were used to determine the minimum sample size needed to obtain stable results. For the three largest samples, all of the criterion-based methods yielded scales with significant cross-validities. Among these methods, two vertical percent strategies generally produced the most valid scales for the four largest samples. Without exception, the cross-validities for the only noncriterion-based method (rare response) failed to reach significance. The effects of unit versus differential weighting and scale length versus item-alternative validity are discussed.     

Specification of Scales in Biodata Form Development: Rational vs. Empirical and Global vs. Specific

Although empirical keying has been the most popular scoring strategy with biodata, researchers have increasingly argued that rational approaches are better for advancing theory. Higher validities and less faking with empirical keys, however, have made many reluctant to abandon them. Research in the personality field provided support for the notion that many rational biodata scales may be multi-faceted. Development of more specific scales was suggested as a technique for creating rational scales with validities that more closely approximate that of empirical keys. Three different strategies (rational, internal, and external/empirical) were used in the scoring of a biodata inventory for use with mechanical equipment franchise owners. The rational and internal approaches were investigated within two samples and at two levels - very specific constructs and global constructs. In addition, two types of criteria were used, including an objective measure of sales and supervisory performance ratings. The specific rational scales were as predictive as the empirical item key. Strategies for developing and scoring a biodata form using a rational approach are discussed.     

INDIVIDUAL DIFFERENCES IN INTERVIEWER RATINGS: THE IMPACT OF STANDARDIZATION, CONSENSUS DISCUSSION, AND SAMPLING ERROR ON THE VALIDITY OF A STRUCTURED INTERVIEW

This research investigated empirical issues regarding the validity of individual interviewers'( N = 62) ratings collected after a structured interview. Each interviewer rated an average of 25 interviewees. One hypothesis examined but not supported was that systematic interviewer errors will attenuate interview validity when data are aggregated across interviewers. Also investigated was the validity of ratings averaged across interviewers compared to consensus ratings; consensus ratings were shown to have significantly but probably not practically higher validities. Third, a meta-analysis of individual interviewer validities revealed that all of the variance in validities could be attributed to sampling error. Results and implications are discussed.     

Item selection counts: a comparison of empirical key and rational scale validities in theory-based and non-theory-based item pools

Little explicit attention has been given to the impact of item pools on the validities and cross-validities of different background data scoring approaches. This study tests the idea that pools of items theoretically related to the performance of interest will outperform pools of items with no hypothesized relationship with the criterion. Validities and cross-validities of rational scales and empirical keys created from theory- and non-theory-based item pools were compared for 3 criteria. When size of the item pools was held constant, theory-based empirical keys (correlational and vertical percent) and rational scales showed larger validities and cross-validities than non-theory-based empirical keys (correlational and vertical percent) and showed minimal shrinkage in cross-validities. Even when item pool for the non-theory-based keys was expanded to include all items in the instrument, the theory-based keys showed comparable or slightly better validities and cross-validities for 2 of the 3 criteria, including college GPA, which was separated from the predictors by 4 years.     

《中国大学生人格量表》的编制*

在建立中国人人格大七结构模型以及成熟的人格量表基础上,本研究通过实证研究建立了适用于大学生群体的人格量表。统计结果表明,由68个项目、7个维度构成的中国大学生人格量表(CCSPS)与中国人人格的7因素模型有着良好的拟合,而且信度和效度检验均符合心理测量学要求;此外,在55098名被试上建立了CC-SPS的常模。研究结果表明CCSPS针对中国大学生群体可靠的人格测量工具。     

Empirical correlates of the MMPI-2 Restructured Clinical (RC) Scales in mental health, forensic, and nonclinical settings: an introduction

We introduce 3 studies of the empirical correlates of the MMPI-2 Restructured Clinical (RC) Scales: a study by Arbisi, Sellbom, and Ben-Porath (2008/this issue) that expands the range of correlates of the RC scale in mental health settings; an investigation by Sellbom, Ben-Porath, Baum, Erez, and Gregory (2008/this issue) of the empirical correlates of the scales in a forensic setting; and a study by Forbey and Ben-Porath (2008/this issue) examining convergent and discriminant validities of the RC scale in a nonclinical setting. To serve as a general introduction to the 3 new empirical studies presented in this special section, we begin with a review of the existing literature on the RC scales and briefly describe these studies.     

Unmanned Aircraft Systems Selection Practices: Current Research and Future Directions

This review examines evolving personnel selection practices for unmanned aircraft systems/remotely piloted aircraft (UAS/RPA) across the U.S. Air Force, U.S. Naval Services, and U.S. Army. Findings across services revealed several themes, including strengths (e.g., strong predictive validities, consistency in operator profiles), weaknesses (e.g., small number of predictive validation studies, small sample sizes), and gaps (e.g., need for further investigation of noncognitive predictors). Ongoing and proposed research, including development and implementation of new instruments and methodologies, are discussed, followed by suggestions to facilitate enhanced UAS/RPA selection practices across the services.     

Diagnostic validity statistics and the MCMI-III

This article (a) describes and illustrates the nonredundant and clinically important information that may be obtained from 5 diagnostic validity statistics (DVSs): incremental validities of positive and negative test diagnoses, Cohen's kappas, Cohen's effect sizes, and areas under receiver operating characteristic (ROC) curves and (b) determines values of these DVSs for 24 Million Multiaxial Clinical Inventory III scales from results reported in 1994 and 1997 validity studies. The DVSs for the 1997 study (T. Millon, R. Davis, & C. Millon, 1997) were often more than 3 times larger than corresponding DVSs for the 1994 study (T. Millon, 1994). The author suggests these large differences could reflect not only effects of factors that caused underestimation of validities by DVSs of the 1994 study, but also effects of factors that may have caused overestimation of validities by DVSs of the 1997 study.     

PREDICTION OF TRADES APPRENTICES'PERFORMANCE ON JOB SAMPLE CRITERIA

A criterion-related validation was conducted to assess the validity of four aptitude tests and five tests of content taken directly from job tasks in predicting job sample performance of apprentices in eight skilled trades. Observed validities were above .40 (corrected for range restriction, validities averaged .52). Though there were large subgroup mean differences on both predictor and criterion measures, there was no evidence of significant differential prediction.     

A Matter of Context: A Meta‐Analytic Investigation of the Relative Validity of Contextualized and Noncontextualized Personality Measures

The empirical evidence that has accumulated in support of the notion that personality is a valid predictor of employee performance is vast, yet debate on the matter continues. This study investigates frame‐of‐reference effects as they relate to the validity of self‐report measures of personality. Specifically, we compare the validities of general, noncontextualized personality measures and work‐specific, contextualized measures. The findings suggest that personality measures are a more valid predictor of performance when the scale items or instructions are framed specifically so as to reference work‐specific behaviors. We found that the validities for noncontextualized measures of personality ranged from .02 to .22, with a mean validity of .11. The validities for contextualized measures ranged from .14 to .30, with a mean of .24 . Additional moderator analyses were conducted in an effort to examine several alternate explanations for these validity differences. Specifically, we examined differences between the developmental purpose (general use vs. workplace use) and reliabilities of each type of personality measure. We also compared the validities from published studies to those from unpublished studies. Results suggest that these moderators did not have an impact on the validity differences between noncontextualized and contextualized measures.     

Combining the judgments of experts: How many and which ones?

R. M. Hogarth (1978, Organizational Behavior and Human Performance, 21, 40–46) presents an analytical model which, under certain conditions, may be useful for estimating both the number of experts to include in a staticized group and which experts. This paper reports a study of an important real-world judgment task used to evaluate (1) the extent to which the specified conditions hold and (2) the effect of violations of the conditions on the ability of the model to approximate the actual empirical validities that result from forming such groups. The results indicate that, although the specified conditions hold only moderately well, the model suggested by Hogarth provides a remarkably good approximation to the actual empirical validities of the staticized groups.     

Units of analysis and item statistics for environmental assessment scales

Choice of units of analysis is a critical problem in evaluating environmental assessment items. The “Ecological Fallacy” involves interpreting results based on ecological entities, such as environmental settings, as applying to individuals. A less familiar error, the “Individual Differences Fallacy,” involves interpreting results based on individuals as applying to settings. Although this second error has been ubiquitous, little is known about the empirical consequences of using different units of analysis in item analysis. This study examined this issue when the units were (1) individuals, (2) observed settings, or (3) “artificial” or random settings. Neither item validities based on individuals nor item validities based on settings yielded unequivocal results. However, a measure developed specifically for environmental research, the Split-Sample Correlation, appeared useful for identifying environmental items. Researchers constructing environmental assessment scales should select items with both high Split-Sample Correlations and high setting-level item validities.     

THE IMPLICATIONS FOR PERSONNEL SELECTION OF APPARENT DECLINES IN PREDICTIVE VALIDITIES OVER TIME: A CRITIQUE OF HULIN, HENRY, AND NOON

Hulin, Henry, and Noon (1990) reviewed evidence from a number of studies which supported, in their view, the position that predictive validities decreased over time. If correct, their results would have significant implications for personnel selection practice and research. However, further analysis of their evidence suggested that their results may have only limited generalizability. More specifically, few of the studies they used to support their claim of decreasing predictive validities were field studies of prediction-criterion pairs. Furthermore^ reported data on lagged intercorrelations were of limited relevance to the question of decreasing validities. Finally, a large body of data relevant to the issue of time-lagged validities in a personnel selection context were omitted because the data did not meet Hulin et al.'s restrictive criteria.     

DESIGN, VALIDITY, AND USE OF STRATEGICALLY FOCUSED EMPLOYEE ATTITUDE SURVEYS

This paper presents logic, procedures, validity, and use of employee attitude surveys targeted on strategic objectives of the firm. The logic is that employees at the front line are in an optimal position to report on the degree to which strategic initiatives are being carried out. The procedure is to design survey questions that focus on the initiative rather than on employees' personal feelings or satisfaction. To validate employee reports, the relationships between those reports and customer satisfaction over four quarters of the use of the survey are shown. Evidence shows that the logical and empirical keying of employee surveys to strategic initiatives and objectives of the firm provide data of immediate use to management, in the present case to both marketing and human resources management.     

Units of analysis and item statistics for environmental assessment scales

Choice of units of analysis is a critical problem in evaluating environmental assessment items. The “Ecological Fallacy” involves interpreting results based on ecological entities, such as environmental settings, as applying to individuals. A less familiar error, the “Individual Differences Fallacy,” involves interpreting results based on individuals as applying to settings. Although this second error has been ubiquitous, little is known about the empirical consequences of using different units of analysis in item analysis. This study examined this issue when the units were (1) individuals, (2) observed settings, or (3) “artificial” or random settings. Neither item validities based on individuals nor item validities based on settings yielded unequivocal results. However, a measure developed specifically for environmental research, the Split-Sample Correlation, appeared useful for identifying environmental items. Researchers constructing environmental assessment scales should select items with both high Split-Sample Correlations and high setting-level item validities.     

ESTIMATION OF EMPLOYMENT VALIDITIES BY LESS EXPERIENCED JUDGES

In a previous study, Schmidt, Hunter, Croll and McKenzie (1983) demonstrated that estimates of the validity of cognitive tests made by highly trained and experienced judges are more accurate than empirical estimates obtained from small-sample validity studies. The present study examined whether less experienced judges could also produce accurate estimates. Twenty-eight recent Ph.D.'s in I/O Psychology estimated observed validities for the same 54 job-test combinations used by Schmidt et al. (1983). The estimates of these judges contained about twice as much random error as the experts' estimates. Systematic error of the less experienced judges was also greater than that of the experts (.0732 vs .019). The systematic errors of the two sets of judges were in opposite directions: less experienced judges overestimated validities, on average, while experts underestimated them. The results show that the estimates of less experienced judges contain less information than those of experts, but also that averages of estimates of several less experienced judges are as accurate as those obtained from small-sample empirical studies.     

The Effects of Social Desirability and Faking on Personality and Integrity Assessment for Personnel Selection

A review of the extant literature and new empirical research suggests that social desirability is not much of a concern in personality and integrity testing for personnel selection. In particular, based on meta-analytically derived evidence, it appears that social desirability influences do not destroy the convergent and discriminant validity of the Big Five dimensions of personality (Emotional Stability, Extraversion, Openness to Experience, Agreeableness, and Conscientiousness). We also present new empirical evidence regarding gender and age differences in socially desirable re- sponding. Although social desirability predicts a number of important work variables such as job satisfaction, organizational commitment, and supervisor ratings of training success, social desirability does not seem to be a predictor of overall job performance and is only very weakly related to specific dimensions of job performance such as technical proficiency (r = -.07) and personal discipline ( r = .05). Large sample investigations of the moderating influences of social desirability in actual work settings indicate that social desirability does not moderate the criterion-related validities of personality variables or integrity tests. The criterion-related validity of integrity tests for overall job performance with applicant samples in predictive studies is .41. Controlling for social desirability in integrity or personality test scores leaves the operational validities intact, thereby suggesting that social desirability functions neither as a mediator nor as a suppressor variable in personality-performance.     

The hazards of underspecified models: the case of symmetry in everyday predictions

Should one be more confident when predicting the whole (or an event based on a larger sample) from the part (or an event based on a smaller sample) than when predicting the reverse? The relevant literature on judgment under uncertainty argues that such predictions are symmetrical but that, as an empirical matter, people often fail to appreciate this symmetry. The authors show that symmetry in prediction does not necessarily hold. In addition to an empirical study involving predictions about soccer games, they develop a theoretical model showing that, at least for the ranges of numerical values usually found in everyday judgment problems, symmetry in predictions is uncommon when 2 different sample sizes are involved. The complexity of the theoretical model used in this analysis raises questions about model specification in judgmental research.