Reliability (Generalizability) of Job Performance Measurements: Navy Machinist Mates

The Armed Services have undertaken a Joint Services Project to develop prototype methods for measuring job performance and, if feasible, to link enlistment standards to on-the-job performance. As one step toward meeting the first purpose of the project, our study assessed the reliability of scores on four measures of job performance of machinist mates in the Navy: a hands-on performance test, a paper-and-pencil job knowledge test, job task perform- ance ratings, and global ratings. Generalizability theory was used to estimate the reliability of the measures and to improve the design of each. The generalizability analyses investigated the consistency of performance scores over observers, tasks (equipment operation, casualty control, preventative maintenance), and types of raters (supervisor, peer, and self). Recommenda- tions about the optimal designs of studies for decision making (e.g., pre- dicting hands-on performance scores from the other measures of job perform- ance) were based on the findings.     

Predicting negotiation skills

The purpose of this study was to validate a selection process that was designed to predict negotiation skills for a telephone collections job. First, a job analysis was performed (n=343) to identify the major functions and behavioral requirements of the job. Next, the test, a situational judgment inventory, was developed, and a concurrent criterion related validation study was completed (n=249). The criterion measures were supervisory ratings of performance. Validity coefficients ranged from r=.41 (p<.001) to r=.45 (p<.001). In conclusion, the situational judgment inventory was found an effective vehicle for measuring telephone negotiation skills.     

Predicting sales skills

The purpose of this study was to validate a selection process designed to predict sales skills for a telephone sales job. First, a job analysis was performed (n=213) to identify the major functions and behavioral requirements of the job. Next, the test, a situational judgment inventory, was developed, and a concurrent criterion related validation study was completed (n=236). The criterion measures were supervisory ratings of performance. Validity coefficients ranged from r=.16 (p<.01) to r=.24 (p<.001). In conclusion, the situational judgment inventory was found an effective vehicle for measuring telephone sales skills.Sincere thanks to Roz Lobus for her assistance in the earlier phases of this study.     

Replication and Extension of Models of Supervisory Job Performance Ratings

This study tested causal models of job knowledge, job proficiency, and supervisor ratings in a sample (N = 838) of U.S. Air Force enlisted personnel. Results indicated that (a) effects of ability and experience were linear, not interactive; (b) different conceptualizations of "experience" play somewhat different causal roles in the determination of job knowledge, job proficiency, and supervisory ratings; (c) general support for the mediational roles of job knowledge and job proficiency; and (d) supervisory ratings reflect both technical (task-based) and motivational (contextual) aspects of performance.     

人事决策信息利用和效能预测模型

本研究以 1 2 2名人事管理人员为被试 ,采用问卷测量法 ,分析了企业人事决策信息利用的现状 ,信息利用指标和效能的关系。结果发现 :1企业人事决策中比较重视能力要求信息、组织环境信息和个人档案材料 ,忽略心理特点信息的利用 ;2非程序信息、人职匹配信息对效能指标的直接预测力较强 ,其他指标的预测力较弱 ;3能力要求、考核成绩、个人档案材料等信息通过人职匹配信息的利用对效能指标产生影响。文章还构建了人事决策的效能预测模型 ,并讨论了研究结果对企业人事决策诊断和发展的理论和实际意义。     

PREDICTING NAVY EFFECTIVENESS: EXPECTATIONS, MOTIVATION, PERSONALITY, APTITUDE, AND BACKGROUND VARIABLES

The purpose of this study was to identify predictors of performance within seven Navy occupational groups. Life history, expectations, motivation, personality, and aptitude variables were used as predictors of a 2-year effectiveness criterion for 7,923 enlisted Navy men and women. Results of multiple regression analyses showed that the most powerful predictors included: years of schooling, school expulsions and suspensions, the two Comrey Personality Scales of Social Conformity and Orderliness, arrests, age, General Classification Test (aptitude), and Peer Cohesion (expectations). Comparisons across groups indicated that the development of separate equations for each occupation was not supported. Recommendations were made to improve selection procedures and to change several aspects of the organization, suggestions which would be expected to increase rates of effective performance.     

Expressed vs inventoried interests as predictors of paramedical effectiveness

Two interest measures, a composite of six items which provide a direct assessment of an individual's preferences for Navy health care work and a Navy Vocational Interest Inventory (NVII) scale which assessed the degree of similarity between an individual's activity preferences and those of hospital corpsmen (HMs), were considered along with aptitude test scores as potential predictors of effectiveness among 2429 Navy enlisted personnel assigned to training for HM jobs. Effectiveness was defined as completion of HM training and remaining on the job for at least 2 years. The six-item composite and NVII HM scale scores were correlated .51 with each other and were correlated .27 and .28, respectively, with the effectiveness criterion. When these measures were added to the aptitude measure by using multiple regression procedures, the validity for predicting effectiveness was increased significantly from .36 to .42 with a cross-validity of .42 for the three-variable equation. These findings provide support for the contention that interest measures can be useful predictors of occupational success as well as occupational entry. Furthermore, since most of the interest-related attrition that was found in this study occurred during HM training, these findings suggest that job withdrawal attributable to mismatches in vocational interests may tend to occur very soon after the individual enters the occupational field.     

A Review and Extension of Current Models of Dynamic Criteria

An important issue in personnel selection and test validation has been the nature of performance criteria and more specifically the existence of dynamic criteria. There is a continuing debate regarding the extent to which performance and validity coefficients remain stable over time. We examine research within work, laboratory, and academic settings and evaluate existing models of dynamic criteria. Building on previous models, we propose an integrative model of dynamic criteria that identifies important issues for ability and performance constructs and discusses how variables related to the task, job, and organization can affect the temporal stability of criterion performance and the ability-performance relationship.     

The Validity of Job Knowledge Measures

The validity of written job knowledge tests as predictors of job performance and training success was examined. Based on a sample of 363,528 persons and 502 coefficients, meta-analyses were performed to investigate the extent of validity generalization and the effects of two hypothesized moderator variables: content similarity and job complexity. Corrected mean validities were 0.45 for studies predicting job performance and 0.47 for studies of training success. Support was found for both moderators since validities were higher for high complexity jobs and when job-test content similarity was high. The implications of this study for personnel selection and research in the area of work performance are discussed.     

Prediction of job satisfactoriness for workers with severe handicaps from aptitudes,personality, and training ratings

The predictive validity of General Aptitude Test Battery and Sixteen Personality Factor Questionnaire scores were compared to standard training ratings made by vocational instructors against the criterion of work performance measured by the Minnesota Satisfactoriness Scales for a sample of 106 employees with severe handicaps. The psychometric test variables were not correlated with the criterion; however, the training ratings were consistently predictive of the job satisfactoriness scores. These results suggest that the employment potential of job applicants with disabilities can be assessed more accurately using situational training ratings, as opposed to standardized psychometric test scores.     

Sexual harassment experiences and equal opportunity perceptions of Navy women

In 1989, the Navy began tracking the sexual harassment rates among its officer and enlisted personnel by administering the Navy Equal Opportunity Sexual Harassment (NEOSH) Survey on a biennial basis. While previous reports on the results of the NEOSH Survey have discussed the occurrence, frequency, and effects of sexual harassment in the Navy [A. L. Culbertson, P. Rosenfeld, and C. E. Newell (1993) Sexual Harassment in the Active-Duty Navy: Findings from the Navy-Wide Survey (TR-94-2), San Diego, CA: Navy Personnel Research and Development Center], this study examines whether victims of sexual harassment differ in their perceptions of equal opportunity (EO) climate from those who have not been harassed. Data from the 1991 NEOSH Survey were reanalyzed. Sixty-seven percent of the women officers were White, 25% African-American, and 18% Hispanic. Of the women enlisted, 52% were White, 31% African-American, and 18% Hispanic. The results indicated that women who were sexually harassed perceived that there was less EO in the Navy than women who had not been harassed. This was true for both officer and enlisted personnel. In addition, specific organizational consequences of sexual harassment were identified.The opinions expressed in this paper are those of the authors, are not official and do not necessarily represent the views of the Navy Department.     

DEVELOPMENT OF MULTIPLE JOB PERFORMANCE MEASURES IN A REPRESENTATIVE SAMPLE OF JOBS

The goal of criterion development in Project A was to construct multiple measures of the major components of job performance such that the total performance domain for a representative sample of the population of entry-level enlisted positions in the U.S. Army was covered. These measures were to be used as criteria against which to validate both experimental and existing predictors of job performance. The initial model specified that performance is multidimensional within two major categories of dimensions designated as organization-wide and job specific. The development strategy involved describing the total domain of job content via extensive task analyses and critical incident analyses, generating the critical performance dimensions that constitute it, constructing measures for each dimension, and evaluating each measure using expert judgment and field test data. The specific measures developed consisted of rating scales, tests of job knowledge, hands-on job samples, and archival records. The major steps in the job analyses, content sampling, instrument construction, and instrument evaluation are described, and the final array of criterion measures is presented.     

Test–Retest Reliability of Ratings of Job Performance Dimensions in Managers

This article examines the test–retest reliability of supervisory ratings for several dimensions of job performance and for overall job performance. We found that the test–retest reliability of overall job performance is .79 (SD = .08), a value very close to the one found by Viswesvaran, Ones and Schmidt (1996), and that the average test–retest reliability for specific dimensions of job performance is .57 (SD = .07). We also found that some dimensions of job performance appear to be easier to rate than others. We suggest some implications of these findings for research and practice of personnel selection.     

PREDICTING LONG-TERM FIREFIGHTER PERFORMANCE FROM COGNITIVE AND PHYSICAL ABILITY MEASURES

Firefighters from 1 academy training class were observed for 23 years, beginning with their selection test consisting of a g-saturated written exam (GCA) and firefighting simulations loaded on a strength/endurance (SE) factor. Operational validity coefficients for both GCA and SE were high for training success and remained consistently high for job performance ratings throughout the study. The operational validity for combined GCA and SE predictors was .86 for a composite job rating measure covering 21 years of service. A structural model produced similar results for more broadly defined GCA and SE latent variables. Both analyses suggested approximately equal weighting for GCA and SE for a fire service selection test. Results indicate considerable latitude in choosing cognitive and physical predictors for firefighter screening if the predictors are highly loaded on GCA and SE.     

INTERNATIONAL VALIDITY GENERALIZATION OF GMA AND COGNITIVE ABILITIES: A EUROPEAN COMMUNITY META-ANALYSIS

This article reports on a series of meta-analyses into the criterion validity of general menatl ability (GMA) and specific cognitive ability tests for predicting job performance ratings and training success in the European Community (EC). Meta-analyses were computed on a large EC database examining the operational validity of GMA and other specific cognitive abilities, including verbal, numerical, spatial-mechanical, perceptual and memory ( N ranged from 946 to 16,065) across 10 EC member countries. The results showed that tests of GMA and specific cognitive ability are very good predictors of job performance and training success across the EC. Evidence for the international validity generalization of GMA and specific cognitive abilities was presented. The results for the EC meta-analyses showed a larger operational validity than previous meta-analyses in the U.S. for predicting job performance. For training success, the European and American results are very similar. Implications for the international generalizability of GMA test validities, practical use of cognitive ability tests for personnel selection, and directions for future research are discussed.     

Are observer ratings of applicants’ personality also faked? Yes,but less than self‐reports

Although self‐report personality tests are a comparatively cheap and easy‐to‐administer personnel selection tool, researchers have criticized them for not predicting enough criterion‐related variance. Researchers have suggested using observer‐ratings of personality (e.g., as part of a reference check from a supervisor) because observer‐ratings have been reported to be more predictive. However, it is theoretically and empirically unclear whether supervisors also engage in faking (the intentional distortion of responses). Study 1 explored faking among managers who were first asked to imagine that a subordinate had to leave his/her job for private reasons and then to rate the personality of the subordinate. A week later, managers rated their subordinates honestly. A repeated‐measures MANOVA indicated that managers did fake. Study 2 (among supervisors of working students) replicated the above findings but also showed that there is less faking in supervisor‐ratings than in self‐ratings. Furthermore, we found no evidence that the validity of personality scales for predicting academic performance depends on self‐ versus observer‐ratings or on an applicant versus an honest condition. These two studies thus show that practitioners should not equate personality ratings obtained from observers in a selection context with honest personality ratings.     

Influences of cognitive complexity on the dimensions underlying perceptions of the work environment

The literature on cognitive processes suggests that individuals differ in the dimensions they use to describe the work environment and thus the behavioral implications they attach to characteristics of that environment. Using the technique described by Bem and Allen (1974), navy enlisted personnel were separated into subgroups with high, medium, and low cognitive complexity scores. Responses about perceived characteristics of the work environment produced six components for the high complexity group versus four for the low group. Questions about supervisor behaviors, role characteristics, and work group interactions yielded similar dimensions for all groups, although differences were found for responses about macroorganizational characteristics. Significant but low relationships were found between complexity scores and global indices of job performance.This article is Report Number 79-2, supported by Naval Medical Research and Development Command, Department of the Navy, under Research Work Unit MF58.524.022-0011. The views presented are those of the authors. No endorsement by the Department of the Navy has been given or should be inferred.     

Estimating Coefficients of Equivalence and Stability for Job Performance Ratings: The importance of controlling for transient error on criterion measurement

Previous research on measurement error in job performance ratings estimated reliability using coefficients: alpha, test–retest, and interrater correlation. None of these three coefficients control for the four main sources of error in performance ratings. For this reason, coefficient of equivalence and stability (CES) has been suggested as the ideal estimate of reliability. This article presents the estimates of CES for a time interval of 1, 2, and 3 years. The values obtained for a single rater were .51, .48, and .44, respectively. For two raters, the values were .59, .55, and .51. The findings suggest that previous reliability estimates based on alpha, test–retest, and interrater coefficients overestimated the reliability of job performance ratings. In the present study, the interrater coefficient overestimates reliability by 13.6–25.4% for an interval time of 1–3 years, as it does not control for transient error. Results also showed that the importance of transient error increases as the length of the interval between the measures increases. Based on the results, it is suggested that corrected validities based on interrater reliability underestimate the magnitude of the validity. The implications of these findings for future efforts to estimate criterion reliability and predictor validity are discussed.     

PERFORMANCE APPRAISAL BEHAVIORS: SUPERVISOR PERCEPTIONS AND SUBORDINATE REACTIONS

This study examined supervisor perceptions and subordinate reactions to formal performance-appraisal reviews. The performance-appraisal behaviors of supervisors and the reactions of their subordinates were studied in a sample of university employees. A factor analysis revealed that there were three dimensions of formal performance appraisals: two developmental dimensions (being supportive; emphasizing performance improvement) and one administrative dimension (discussing pay and advancement). Regression analyses suggested that supervisors supported highly rated individuals and stressed improvement efforts for poor performers. After controlling for the level of previous performance ratings, results indicated that support in the appraisal review was associated with higher levels of employee motivation, while discussing pay and advancement was associated with higher levels of employee satisfaction. Unfortunately, improvement efforts by the supervisors did not influence job performance one year later.