Validation of intelligence and mechanical aptitude tests in selecting employees for manufacturing jobs

One hundred ninety-three manufacturing employees who produce electro-mechanical components participated in a concurrent criterion-related validity study. The employees were administered three tests: The Bennett Mechanical Comprehension Test (Form S); The Flanagan Aptitude Classification Test-Mechanics; and the Thurstone Test of Mental Alertness (Form A). Job performance was measured by a supervisor rating of fifteen job dimensions, assessed at two points in time separated by 60 days. Correlational and multiple regression analyses were used to assess the relationship between test scores and job performance ratings. The results revealed that the Bennett Mechanical Comprehension test was the best single predictor of job performance (uncorrectedr =.38), and the incremental gain in predictability from additional tests was not significant. The results were discussed in the context of the changing nature of manufacturing jobs and the inadequacy of conventional mechanical aptitude tests to be sensitive to these changes.     

Applicant Reactions to Personality Tests: Effects of Item Invasiveness and Face Validity

In a simulated employee selection exercise, two hundred and ten participants responded to a personality test that varied in terms of item invasiveness and item face validity. A third factor of empirical job-relatedness was manipulated via test instructions. Reactions to the test, the organization, and behavioral intentions about accepting a job offer were measured. Results indicated that item invasiveness and face validity interacted in the prediction of all dependent variables. Specifically, item invasiveness affected the dependent variables only when face validity was low. Implications for the use of personality tests and integrity tests were discussed.     

Revisiting predictive bias of cognitive ability tests against Hispanic American job applicants

Previous studies have concluded that cognitive ability tests are not predictively biased against Hispanic American job applicants because test scores generally overpredict, rather than underpredict, their job performance. However, we highlight two important shortcomings of these past studies and use meta-analytic and computation modeling techniques to address these two shortcomings. In Study 1, an updated meta-analysis of the Hispanic–White mean difference (d-value) on job performance was carried out. In Study 2, computation modeling was used to correct the Study 1 d-values for indirect range restriction and combine them with other meta-analytic parameters relevant to predictive bias to determine how often cognitive ability test scores underpredict Hispanic applicants’ job performance. Hispanic applicants’ job performance was underpredicted by a small to moderate amount in most conditions of the computation model. In contrast to previous studies, this suggests cognitive ability tests can be expected to exhibit predictive bias against Hispanic applicants much of the time. However, some conditions did not exhibit underprediction, highlighting that predictive bias depends on various selection system parameters, such as the criterion-related validity of cognitive ability tests and other predictors used in selection. Regardless, our results challenge “lack of predictive bias” as a rationale for supporting test use.     

A Concurrent Validity Study Applied to a Secretarial Position in a State Universities Civil Service System

In a concurrent validity study, a comprehensive job analysis of a mid-level secretarial position resulted in the development of highly valid employment selection instruments. Six hundred fifty-nine supervisors and 883 incumbents in 20 locations participated in the job analysis. Scores from the selection test correlated with composite ratings from a research performance appraisal (RPA) .41 (p<.001). Two forms of the test (A and B) were developed. Form A and Form B test scores and RPA composite ratings correlated .55 and .48 (p<.001) respectively. The unbiased estimate of equivalence reliability of Form A and Form B was .94. The two tests correlated .89 (p<.001).     

WORK SAMPLE TESTS IN PERSONNEL SELECTION: A META‐ANALYSIS OF BLACK–WHITE DIFFERENCES IN OVERALL AND EXERCISE SCORES

Work sample exams are generally thought to have either low or comparatively low levels of ethnic group differences when used for personnel selection. Such exams are sometimes called “simulation exercises” and involve having applicants perform a set of tasks that are similar to those performed on the job. The nearly ubiquitous meta‐analytic value of Black–White subgroup differences in the literature is d= .38. Unfortunately, this estimate is plagued by a variety of problems (e.g., range restriction, inclusion of nonwork sample tests). Further, there are virtually no analyses that examine how the saturation of different constructs influence work sample tests. We gathered available data for Black–White ethnic group differences and found that overall work sample differences were markedly larger for samples of job applicants (d= .73) than previously thought. We also examined how different exercises and saturation of different constructs influenced work sample ds. For example, work sample test ratings of cognitive and job knowledge skills were associated with a mean observed d= .80, whereas ratings of various social skills were associated with mean observed ds that varied from .21 to .27. We urge scientists and practitioners to consider both the method and the constructs that are targeted when forecasting predictor ds.     

The Validity of Assessment Centres for the Prediction of Supervisory Performance Ratings: A meta‐analysis

The current meta‐analysis of the selection validity of assessment centres aims to update an earlier meta‐analysis of assessment centre validity. To this end, we retrieved 26 studies and 27 validity coefficients (N=5850) relating the Overall Assessment Rating (OAR) to supervisory performance ratings. The current study obtained a corrected correlation of .28 between the OAR and supervisory job performance ratings (95% confidence interval .24≤ρ≤.32). It is further suggested that this validity estimate is likely to be conservative given that assessment centre validities tend to be affected by indirect range restriction.     

Integrity Testing and Job Performance Within High Performance Work Teams: A Short Note

Integrity tests have exhibited robust validity for predicting performance among traditional jobs. However, some organizations have shifted to self-managed work teams and the relationship between integrity and performance within these jobs is untested. One hundred fourteen high performance team members within a single department completed the Hogan Personality Inventory (HPI). The HPI Reliability scale was used as the integrity measure. The subjects' job performance was evaluated by their team leaders. Performance ratings were transformed to equalize ratings across the eight teams. Results indicate that integrity was related to transformed performance ratings (r = .25) within a high performance team environment. Limitations and implications are discussed.     

Aural and Visual Sources of Validity in Structured Employment Interviews

This study tests effects of aural and visual cues on the validity of structured interviews. We recorded simulated job interviews with 40 managers from three utility companies on videotape. Supervisors provided performance ratings. Undergraduate research participants (N = 194) saw and heard the videotapes, heard them without the picture, or saw them without the sound and rated interviewees accordingly. Approximately 6 participants rated each interviewee in each condition of cue availability. Performance ratings and participants′ pooled ratings correlated .36 when they could hear and see the interviews, .33 when they could only hear the interviews, and .32 when they could only see the interviews. Interview ratings based on sight alone correlated .53 with interview ratings based on sound alone. One explanation is that aural cues, visual cues, and supervisory performance ratings all reflect true differences between interviewees in traits related to management effectiveness. Another is that interviewers and supervisors make the same mistakes when interpreting implications of aural cues and visual cues for management effectiveness.     

Creativity,Originality, and Appropriateness: What do Explicit Instructions Tell Us About Their Relationships?

Originality is a necessary part of creativity, but creative things are more than just original. They also solve a problem, or more generally are somehow fitting or appropriate. Yet previous research found an inverse relationship between ratings of originality and ratings of appropriateness. The present investigation employed a different methodology — it focused on the generation of ideas instead of judgments about them — to reexamine the relationships between originality, appropriateness, and creativity. Undergraduate students (N = 170) from two large universities received either realistic or unrealistic divergent thinking tasks. These were given with one of four types of instructions. These asked them to give (a) as many ideas as possible (which is the standard type of instruction for these kinds of tests), (b) only original ideas, (c) only appropriate ideas, or (d) only creative ideas. Brief definitions of originality, appropriateness, or creativity were also provided. Comparisons of the four groups indicated that there were significant differences between the different kinds of tasks, with the realistic set eliciting more appropriate ideas than the unrealistic, but the unrealistic tasks eliciting more original and varied (flexible) ideas. There was an interaction indicating that the magnitude of the impact of the instructions varied across tasks. Correlational analyses indicated that the correlation between the originality and appropriateness scores was the lowest among all possible inter‐index relationships (only 7% shared variance). Future research and practical implications are explored.     

The Relationship Between Applicant Reactions, the Likelihood of Complaints, and Organization Attractiveness

This study examined the effect of applicants' reactions to selection measures on complaint intentions and perceptions of organizational attractiveness. Despite recent challenges to selection tests for invasion of privacy, limited research has examined the link between specific reactions and potential legal challenges. As predicted, perceived invasiveness was significantly related to perceived likelihood of complaints for three selection measures. Further, perceived job relatedness was negatively related to likelihood of complaints for a test battery and a math test, but not for an integrity test. On the other hand, neither job relatedness nor invasiveness was significantly related to organizational attractiveness for any of the tests. Limitations and implications for future research are discussed.     

Developing Measures of Basic Job-Relevant English Proficiency for the Prediction of Job Performance and Promotability

The present study is an attempt to devise reliable and valid measures of job-relevant basic English proficiency to be applied in the prediction of job performance and promotability in an entry-level manual labor job dominated by workers whose native language is not English. An audio test and a reading test were developed to assess the ability to understand spoken English and written English respectively using job relevant material. Results showed that both tests demonstrated high predictive validity for the criteria of supervisory assessments of incumbents' English proficiency on the job, overall job performance, and promotability. In predicting ratings of overall job performance and promotability, each test provided incremental validity over the predictability provided by the other test. In addition, the tests also provided incremental validity in predicting promotability over the predictability provided by overall job performance.     

Self-presentation processes in job analysis: a field experiment investigating inflation in abilities, tasks, and competencies

Although job analysis is a widely used organizational data collection technique, little research has investigated the extent to which job analysis information is affected by self-presentation processes. This study represents the first direct test of the propositions offered by F. P. Morgeson and M. A. Campion (1997) concerning self-presentation in job analysis measurement. Using an experimental design, the authors examined job incumbent response differences across ability, task, and competency statements. Results indicated that ability statements were more subject to inflation than were task statements across all rating scales. Greater endorsement of nonessential ability statements was responsible for the differences. This produced higher endorsement of ability items but lower mean ratings. Finally, frequency and importance ratings of global competency statements were generally higher than decomposed ability and task scales, but required-at-entry judgments demonstrated the opposite relationship.     

Testing the Rules of Justice: The Effects of Frame‐of‐Reference and Pre‐Test Validity Information on Personality Test Responses and Test Perceptions

The resurgence of personality tests in selection has sparked interest in factors that may increase the utility and acceptability of these tests. Following a justice framework, the present study explores two possible methods for improving the psychometric properties and test‐taker perceptions of a widely used measure of personality, the NEO‐Five Factor Inventory. The first manipulation altered respondents' frame‐of‐reference (FOR) by adding “at‐work” tags to the personality test. The second provided information about the validity and appropriateness of the personality test for selection. Under the controlled setting of a laboratory experiment, participants (n=345) were randomly assigned to one of the conditions in the following between‐subjects design: 2 (FOR: work specific vs. generic) × 2 (information: validity vs. control). The FOR manipulation produced consistent effects on the personality test responses, but in contrast to recent claims, produced no effect on test perceptions. Alternatively, the information manipulation primarily influenced job‐relatedness perceptions, but had a modest negative effect on the psychometric properties of the personality test. These results show some possibilities, and difficulties, for enhancing perceptions of personality tests. They also have important implications for justice theory because they suggest that interactions among the procedural justice rules may yield unexpected and contradictory effects.     

Better Late than Never? The Relationship between Ratings of Attendance,Punctuality, and Overall Job Performance among Nursing Home Employees

This study examined the relationship between lateness, absenteeism, and overall job performance among nursing home employees. We addressed four issues: (a) agreement between subordinate and superior in their ratings, (b) the stability of rated lateness and rated absenteeism over 6 months, (c) the correlation between ratings of lateness and absenteeism assigned by the worker to himself or herself and the corresponding ratings from supervisors, and (d) the relationship of rated lateness and rated absenteeism to impressions of overall job performance. Neither attendance nor punctuality showed inter‐rater agreement between the worker and his/her supervisor on either of the two occasions. Ratings of attendance were not stable across time for either the subordinate (r=.16) or the supervisor (r=.04), but punctuality ratings showed moderate temporal stability over 6 months for both subordinate (r=.59) and supervisor (r=. 41) impressions. On the first administration of the rating form, attendance and punctuality were not significantly correlated to each other in either the workers' self‐ratings (r=.17) nor the supervisors' ratings of the workers (r=.08). On the second administration, however, these two facets were moderately correlated with each other in both workers' (r=.54) and supervisors' (r=. 39) impressions of employee behavior. Compared with attendance, punctuality was somewhat more highly correlated with impressions of overall job performance from the perspective of the worker herself/himself as well as the supervisor.     

Generalizability of Job Performance Measurements: Marine Corps Rifleman

The purpose of this research was to identify personality constructs to be assessed in the selection of officers in the Turkish Armed Forces by using a personality-oriented job analysis approach. Personality-oriented job analytic interviews were conducted with currently employed and former officers (N = 78). Content analysis of the interviews led to the identification of a list of attributes presumed to be relevant. The attributes were then rated by a group of officers (N = 447) for relevance and importance. Principal component analysis of the weighted ratings (i.e., Relevance × Importance) resulted in 5 personality dimensions considered to be relevant to the job of military officer: Conscientiousness–Self-Discipline, Military Factor, Self-Confidence, Agreeableness–Extraversion, and Leadership. Finally, using a confirmatory approach, we tested the 5-factor structure against 1-factor and 2-factor solutions. Results suggest that the 5-factor solution has significantly better fit than the alternative models, providing some initial evidence for construct validity.     

Selecting trainee conference interpreters: A preliminary study

This study developed and assessed a set of psychometric tests to aid selection of candidate simultaneous and consecutive interpreter trainees. Three kinds of test were used based either on text materials, linguistic subskills or speed stress Twelve tests in all were administered to 29 trainees beginning an intensive course. The test scores were correlated with judges' ratings in the final interpreting examination. Students who passed the examination had higher mean scores on all tests than those who failed (significant for seven tests). Test scores were positively inter-correlated. Of the two factors characterizing the pattern of correlations, the first loaded highly on completion/detection tests and simultaneous examination ratings; the second loaded highly on recall tests and consecutive examination ratings. Text-based tests were more predictive than subskills or speed stress tests. It is concluded that tests would aid candidate selection, and estimates of improved selection are given. The relationship between test type and interpreting is discussed together with possible reasons underlying differential performance on tests of the same kind.     

The Relationship Between SME Job Experience and Job Analysis Ratings: Findings with and without Statistical Control

This investigation examined the relationship between SME job experience and job analysis ratings. Managerial job incumbents (N = 209) provided job analysis ratings which measured 9 task frequency and 6 skill importance components. Job experience was operationalized as both job tenure and residualized job tenure (i.e., statistically-controlled for SME characteristics correlated with job tenure). Respondents were categorized into low, medium, and high groups for each tenure measure. MANOVA results indicated that task ratings differed across residualized tenure groups, while skill ratings did not. Statistical control of the job tenure measure made a substantial difference in results. For the non-residualized tenure groups, findings were reversed: skill ratings differed across non-residualized tenure groups, while task ratings did not. The results of this research are integrated with the existing literature and implications for future research and practice are offered.