The impact of job complexity and performance measurement on the temporal consistency, stability, and test-retest reliability of employee job performance ratings

Although research has shown that individual job performance changes over time, the extent of such changes is unknown. In this article, the authors define and distinguish between the concepts of temporal consistency, stability, and test-retest reliability when considering individual job performance ratings over time. Furthermore, the authors examine measurement type (i.e., subjective and objective measures) and job complexity in relation to temporal consistency, stability, and test-retest reliability. On the basis of meta-analytic results, the authors found that the test-retest reliability of these ratings ranged from .83 for subjective measures in low-complexity jobs to .50 for objective measures in high-complexity jobs. The stability of these ratings over a 1-year time lag ranged from .85 to .67. The analyses also reveal that correlations between performance measures decreased as the time interval between performance measurements increased, but the estimates approached values greater than zero.     

Estimating Coefficients of Equivalence and Stability for Job Performance Ratings: The importance of controlling for transient error on criterion measurement

Previous research on measurement error in job performance ratings estimated reliability using coefficients: alpha, test–retest, and interrater correlation. None of these three coefficients control for the four main sources of error in performance ratings. For this reason, coefficient of equivalence and stability (CES) has been suggested as the ideal estimate of reliability. This article presents the estimates of CES for a time interval of 1, 2, and 3 years. The values obtained for a single rater were .51, .48, and .44, respectively. For two raters, the values were .59, .55, and .51. The findings suggest that previous reliability estimates based on alpha, test–retest, and interrater coefficients overestimated the reliability of job performance ratings. In the present study, the interrater coefficient overestimates reliability by 13.6–25.4% for an interval time of 1–3 years, as it does not control for transient error. Results also showed that the importance of transient error increases as the length of the interval between the measures increases. Based on the results, it is suggested that corrected validities based on interrater reliability underestimate the magnitude of the validity. The implications of these findings for future efforts to estimate criterion reliability and predictor validity are discussed.     

Test–Retest Reliability of Ratings of Job Performance Dimensions in Managers

This article examines the test–retest reliability of supervisory ratings for several dimensions of job performance and for overall job performance. We found that the test–retest reliability of overall job performance is .79 (SD = .08), a value very close to the one found by Viswesvaran, Ones and Schmidt (1996), and that the average test–retest reliability for specific dimensions of job performance is .57 (SD = .07). We also found that some dimensions of job performance appear to be easier to rate than others. We suggest some implications of these findings for research and practice of personnel selection.     

Self-Ideal Congruence Among Engineering Managers

62 engineering managers described their actual and ideal job selves using Q sorts. Self-ideal congruence scores (Z scores) from the Q sorts were correlated with personality test scores, measures of job performance, and other variables. A significant positive relationship was found between self-ideal congruence and management effectiveness. The concept of self-ideal congruence, utilized by client-centered counselors to measure progress of counseling, also has meaning in an industrial context with “normal” persons. Since managers with high self-ideal congruence received higher job performance ratings, the self-ideal congruence score can possibly be considered as an index of job adjustment or measure of self-in-job actualization.     

Construct Validity of Multi‐Source Performance Ratings: An Examination of the Relationship of Self‐, Supervisor‐, and Peer‐Ratings with Cognitive and Personality Measures

Although more and more organizations prefer using multi‐source performance ratings or 360° feedback over traditional performance appraisals, researchers have been rather skeptical regarding the reliability and validity of such ratings. The present study examined the validity of self‐, supervisor‐, and peer‐ratings of 195 employees in a Dutch public organization, using scores on an In‐Basket exercise, an intelligence test, and a personality questionnaire as external criterion measures. Interrater agreement ranged from .28 to .38. Variance in the ratings was explained by both method and content factors. Support for the external construct validity was rather weak. Supervisor‐ratings were not found to be superior to self‐ and peer‐ratings in predicting the scores on the external measures.     

Prediction of job satisfactoriness for workers with severe handicaps from aptitudes,personality, and training ratings

The predictive validity of General Aptitude Test Battery and Sixteen Personality Factor Questionnaire scores were compared to standard training ratings made by vocational instructors against the criterion of work performance measured by the Minnesota Satisfactoriness Scales for a sample of 106 employees with severe handicaps. The psychometric test variables were not correlated with the criterion; however, the training ratings were consistently predictive of the job satisfactoriness scores. These results suggest that the employment potential of job applicants with disabilities can be assessed more accurately using situational training ratings, as opposed to standardized psychometric test scores.     

A Theory of Individual Differences in Task and Contextual Performance

This article describes a theory of job performance that assumes that job performance is behavioral, episodic, evaluative, and multidimensional. It defines job performance as the aggregated value to the organization of the discrete behavioral episodes that an individual performs over a standard interval of time. It uses the distinction between task and contextual performance to begin to identify and define underlying dimen- sions of the behavioral episodes that make up the performance domain. The theory predicts that individual differences in personality and cognitive ability variables, in combination with learning experiences, lead to variability in knowledge, skills, and work habits that mediate effects of personality and cognitive ability on job perform- ance. An especially important aspect of this theory is that it predicts that the kinds of knowledge, skills, work habits, and traits that are associated with task performance are different from the kinds that are associated with contextual performance.     

A Concurrent Validity Study Applied to a Secretarial Position in a State Universities Civil Service System

In a concurrent validity study, a comprehensive job analysis of a mid-level secretarial position resulted in the development of highly valid employment selection instruments. Six hundred fifty-nine supervisors and 883 incumbents in 20 locations participated in the job analysis. Scores from the selection test correlated with composite ratings from a research performance appraisal (RPA) .41 (p<.001). Two forms of the test (A and B) were developed. Form A and Form B test scores and RPA composite ratings correlated .55 and .48 (p<.001) respectively. The unbiased estimate of equivalence reliability of Form A and Form B was .94. The two tests correlated .89 (p<.001).     

Validity of Selection/ Classification Variables for Performance in Three Navy Ratings

This study examined speed of advancement and score on a job knowledge test as job performance criteria in the validation of selection and classification variables for three Navy ratings. Longitudinal samples spanning a 4-year period were drawn for men in radioman, machinist's mate, and hospital corpsman ratings who, within their first enlistment, advanced to E-4 by means of the Navy competitive advancement system. Validity coefficients for Navy enlisted selection and classification variables (Armed Services Vocational Aptitude Battery [ASVAB] and education) were computed for personnel in each rating, subgrouped by entry pay grade. Validity coefficients for the job knowledge test were high by historical standards, and the effects were highest for the subgroups with the highest selection standards. Validity coefficients for the advancement criterion were much lower. Advancement, the criterion that appeared to have the greater face validity, was not as good a measure of job performance as the job knowledge test. The criteria were predicted much better by mental ability than by level of education variables. Personnel who enlisted at E-2 and E-3 pay grades spent substantially more time working at the journeyman performance level during their first enlistment than did personnel who enlisted at E-1.     

PROGRESS IN PSYCHOMETRICS: CAN INDUSTRIAL AND ORGANIZATIONAL PSYCHOLOGY CATCH UP?

Murphy and DeShon (2000) show that interrater correlations do not provide reasonable estimates of the reliability of job performance ratings, and suggest that better estimates can be obtained by applying the methods of generalizability theory. Schmidt, Viswesvaran, and Ones (2000) criticize our suggestions as radical, and argue that: (a) the reliability of ratings should be evaluated using the parallel test model rather than the more general and more realistic generalizability model, (b) reliability and validity are distinct concepts that should not be confused, and (c) measurement models have little to do with substantive models of the processes that generate scores on a test or measure. All three of these ideas were once part of the psychometric mainstream, but progress in psychometrics over the last 3 decades has moved the field well beyond these assumptions and approaches. Modern psychometric theory calls for close linkages between measurement models and substantive models of the phenomena being measured.     

DEVELOPMENT OF MULTIPLE JOB PERFORMANCE MEASURES IN A REPRESENTATIVE SAMPLE OF JOBS

The goal of criterion development in Project A was to construct multiple measures of the major components of job performance such that the total performance domain for a representative sample of the population of entry-level enlisted positions in the U.S. Army was covered. These measures were to be used as criteria against which to validate both experimental and existing predictors of job performance. The initial model specified that performance is multidimensional within two major categories of dimensions designated as organization-wide and job specific. The development strategy involved describing the total domain of job content via extensive task analyses and critical incident analyses, generating the critical performance dimensions that constitute it, constructing measures for each dimension, and evaluating each measure using expert judgment and field test data. The specific measures developed consisted of rating scales, tests of job knowledge, hands-on job samples, and archival records. The major steps in the job analyses, content sampling, instrument construction, and instrument evaluation are described, and the final array of criterion measures is presented.     

The Utility of Utility Analysis

Recent research on utility analysis that focuses on estimating the dollar value of performance gains was critically reviewed and a case was made that other metrics for expressing utility may be desirable. The need to validate the accu- racy of utility predictions was emphasized, and factors that could adversely affect their accuracy were described in a framework of criteria types and job types. A set of research propositions was offered and implications for future work on utility were discussed. In addition to its traditional role as an aid to decision making, utility analysis as a means to understanding the perform- ance construct was emphasized.     

INTERRATER CORRELATIONS DO NOT ESTIMATE THE RELIABILITY OF JOB PERFORMANCE RATINGS

Interrater correlations are widely interpreted as estimates of the reliability of supervisory performance ratings, and are frequently used to correct the correlations between ratings and other measures (e.g., test scores) for attenuation. These interrater correlations do provide some useful information, but they are not reliability coefficients. There is clear evidence of systematic rater effects in performance appraisal, and variance associated with raters is not a source of random measurement error. We use generalizability theory to show why rater variance is not properly interpreted as measurement error, and show how such systematic rater effects can influence both reliability estimates and validity coefficients. We show conditions under which interrater correlations can either overestimate or underestimate reliability coefficients, and discuss reasons other than random measurement error for low interrater correlations.     

Clarifying Construct Validity: Definitions,Processes, and Models

Despite definitions in standard sources, personnel managers, psychologists, and psychometricians persistently encounter problems that are best referred to as the ambiguous nature of validity. The purpose of this article is to pro- vide an overview of construct validity and personnel testing, to demonstrate its practical utility, and to clarify with concrete examples certain theories and models, as well as to illustrate the meaning of the terminology used by com- mentators on the topic. A brief historical overview of testing and validation is presented; the progress of construct validity and its acceptance by various sec- tors of society is discussed parsimoniously in the section The Seven Wonders of Personnel Psychology. In the past, personnel psychologists have not done a very good job of understanding the constructs that underlie test perform- ance. Some new approaches can help to correct this. A process should be rou- tinely used on all tests in order to develop an understanding of the constructs that underlie performance on an employment test; only by knowing the cor- rect criterion and method of measuring it can we ascertain the intrinsic valid- ity of our measures.     

ANTECEDENTS OF INVOLUNTARY TURNOVER DUE TO A REDUCTION IN FORCE

This study examined antecedents to involuntary turnover due to a reduction in force. The authors used structural equation modeling and logit regression analysis with a sample of 194 salespeople to test an exploratory process model of involuntary turnover. Results showed that general mental ability and conscientiousness were indirectly correlated with involuntary turnover through job performance, whereas the relationship of tenure, gender, job involvement, sales volume, and supervisory ratings of job performance to turnover was direct. The results demonstrate that involuntary turnover decisions are significantly related to performance; however, other variables also influenced these decisions. Implications and future research needs are discussed.     

Validation of personality constructs for the selection of insurance industry employees

One hundred and nine employees working in clerical jobs in an insurance company participated in a concurrent criterion-related validity study. The employees were administered the Hogan Personality Inventory which consists of six basic scales and six occupational scales. Five criterion measures were collected: a within-department forced quintile distribution by the manager, and managerial ratings of the job factors of job knowledge, interpersonal skills, organization, and overall performance. The twelve test scores were correlated with each of the five criterion measures. The results revealed the Adjustment, Clerical Aptitude and Managerial Potential scales correlated significantly (p<.05) with each of the five criteria, while the Service Orientation scale correlated significantly with three of the five criteria. The results were discussed in the context of using personality scales to forecast job performance in the insurance industry.     

Construction and Validation of a Biodata Selection Instrument

Abstract

Biodata instruments can be used in selection for several purposes (including screening, prediction, analysis of decision strategies). This article is concerned with the construction and validation of a biodata instrument for the selection of intermediaries by a profit-making employment agency in the Netherlands. The instrument was constructed according to the sample approach and measures work experiences with regard to relevant tasks and job dimensions. A field study gathered biodata scores from applicants (n £ 300) and present employees (n £ 70), interview ratings from the applicants, and ratings of job performance for the employees. The results obtained indicated that the instrument was a potentially useful selection device. The homogeneity of scale scores was sufficient and the scores show little overlap with other selection devices (interview ratings). In addition, particular scores correlate at a statistically significant level (P < 0.01) with ratings of job performance of present employees (r £ 0.30 to 0.40). Estimates of the utility (financial gains) of using the instrument in selection revealed that the instrument may yield considerable pay-offs. It is concluded that the construction and validation of the instrument yielded several positive results, and that the sample approach in constructing biodata selection instruments is an option worth considering.     

AN EVALUATION OF ALTERNATE SCORING METHODS FOR THE MIXED STANDARD SCALE

This study investigated the psychometric properties of three methods of scoring a Mixed Standard Scale (MSS) performance evaluation: the patterned procedure as corrected by Saal (1979); a simple nonpatterned scoring procedure suggested by Prien, Jones, and Miller (1977), which gives equal weights to the performance statements; and a procedure that assigned differential weights to each statement on the basis of scale values provided by a panel of subject matter experts. Interrater reliabilities, scale variances for averaged ratings, and a convergent/discriminant validity analysis, which included an alternate method of job skill ratings, indicated no difference in the score distribution variance, interrater reliability, or validity of different method scores.     

Developing Measures of Basic Job-Relevant English Proficiency for the Prediction of Job Performance and Promotability

The present study is an attempt to devise reliable and valid measures of job-relevant basic English proficiency to be applied in the prediction of job performance and promotability in an entry-level manual labor job dominated by workers whose native language is not English. An audio test and a reading test were developed to assess the ability to understand spoken English and written English respectively using job relevant material. Results showed that both tests demonstrated high predictive validity for the criteria of supervisory assessments of incumbents' English proficiency on the job, overall job performance, and promotability. In predicting ratings of overall job performance and promotability, each test provided incremental validity over the predictability provided by the other test. In addition, the tests also provided incremental validity in predicting promotability over the predictability provided by overall job performance.     

RELIABILITY IS NOT VALIDITY AND VALIDITY IS NOT RELIABILITY

Interrater correlations do provide an index of reliability of job performance ratings. We show that the arguments presented by Murphy and DeShon (2000) lead to the radical conclusion that traditional measurement models–both classical theory and generalizability theory models–can be used neither with job performance ratings nor with other measures used in I-O and other areas of psychology and the social sciences. We show that this untenable conclusion is based on confusion of validity issues and questions with reliability issues and questions. It is also based on the incorrect belief that classical measurement models are capable of addressing only random response measurement error and cannot address other forms of measurement error. We also show that the solution Murphy and DeShon offer to the problem of measurement error in ratings, as they define this problem, cannot work. Properly understood, the position taken by Murphy and DeShon leaves us with the nihilistic conclusion that no appropriate measurement models are possible in psychological research, thus making meaningful research impossible.