IDENTIFYING OPTIMAL PREDICTOR COMPOSITES AND TESTING FOR GENERALIZABILITY ACROSS JOBS AND PERFORMANCE FACTORS

The initial examination of validity generalization in the Army Selection and Classification Project used data from a concurrent validation sample of 4,039 job incumbents drawn from a representative sample of nine jobs. The available data consisted of 24 predictor scores and five job performance factor scores on each individual. The major objectives were to determine (a) the degree of validity generalization across the major components of performance, with the job held constant, and (b) the degree of validity generalization across jobs within each major performance factor. After reducing the predictor set by eliminating variables that added no information, a modified confirmatory analysis was used to test the hypotheses that one equation would fit the data from all performance components and that one equation would fit the data from all jobs, given a particular performance component. The major findings were that different predictor equations were needed for each of the five criterion factors. For generalization across jobs, within each criterion factor, one equation fit the data for four of the five performance components. Different prediction equations were required for the component that reflects proficiency on the technical tasks specific to each job.     

A COMPARISON OF HOLISTIC VERSUS DECOMPOSED RATING OF POSITION ANALYSIS QUESTIONNAIRE WORK DIMENSIONS

Recent studies have attempted to reduce the cost and intrusiveness of the Position Analysis Questionnaire (PAQ) by limiting the amount of information provided to the analyst, with consistently negative results. We examined an alternative technique for improving the cost-effectiveness of the PAQ that avoids the need to rate the hundreds of items that constitute the instrument. Three groups of raters (professional job analysts, graduate students in industrial psychology who were familiar with the PAQ, and PAQ-unfamiliar undergraduates) made direct holistic ratings of the PAQ dimensions for four familiar jobs. The holistic ratings were compared with decomposed PAQ dimension profiles obtained from the item-level ratings of the professional analysts. Cronbach accuracy analyses indicated near-zero convergence between the holistic and decomposed dimension ratings, even for the professional PAQ job analysts. We conclude that holistic rating of dimensions is not an effective means of reducing the cost of a PAQ job analyses and that it is likely to be similarly ineffective with task- or ability-based instruments.     

职务评价的方差分析法在制定职务等级中的应用

职务评价的方差分析法的技术核心是通过方差分析确定职务可比价值各成分的权重。此方法的有效性和实用性已在协助国有企业内部分配改革的制定工资标准中得到验证,并显示其普遍性的方法论意义：职务评价值是职务价值的线性映射：凡涉及职务价值差异的问题都可用此方法解决。本研究应用方差分析法为国有金融系统行员制改革中制定统一的职务等级标准提供技术支持和科学依据。评价程序中,职务分析、职务分类、计算职务评价值等前三个步骤与制定工资标准的作法相同。然后,推算各类职务评价值的变异范围（以９５％置信区间代表）,再将各类职务按管理层次合并为大类。最后,寻求一个合理划分各管理层次大类变异范围的约数,划分整个评价值变异范围。对某银行系统４５２种职务的８４１个样本评价的结果,所确定的职务等级数、各职务大类的等级跨度、位次,都符合该银行系统的人事管理经验及改革设想     

A COMPARISON OF GLOBAL JUDGMENT VS. TASK ORIENTED APPROACHES TO JOB CLASSIFICATION

The purpose of this study was to determine the extent to which direct judgments of similarity by supervisors and incumbents could provide the same job classification results as a more elaborate job analysis procedure involving measures of task overlap among jobs. To accomplish this, 8 foreman jobs in a chemical processing plant were analyzed and compared on 237 task statements. In addition, 15 foremen incumbents and 17 supervisors evaluated the similarities among the same 8 foremen jobs in a paired comparisons rating task. The task-oriented job analysis required hundreds of man-hours to complete; the rating task took 15 minutes. Results using hierarchical cluster analysis and multidimensional scaling analysis revealed that the global judgments and the task-oriented data led to identical conclusions. Also, it was found that incumbent ratings produced the same results as ratings from supervisors. Uses, advantages, and disadvantages of the procedure are outlined.     

VALIDATING SYNTHETIC VALIDATION: COMPARING TRADITIONAL AND SYNTHETIC VALIDITY COEFFICIENTS

We describe a unique application of a synthetic validation technique to a selection system development project in a large organization. Job analysis data were collected from 4,725 job incumbents and 619 supervisors, and were used to identify 11 job families and 27 job components. We developed 12 tests to predict performance on these job components and conducted a concurrent validation study to collect test and job component data for 1,926 incumbents. We created a test composite for each job component and then chose a test battery for each job family based on its relevant job components. Synthetic validity coefficients were computed for each test battery and compared to traditional validity coefficients that were computed within job families with large sample sizes. The synthetic validity coefficient was very close to the within-family validity coefficient for most job families and was within the bounds of sampling error for all job families. Validities tended to be highest when predictors were weighted according to the number of job components to which they were relevant and job component criterion measures were unit weighted.     

EXPERT AND NAIVE RATERS USING THE PAQ: DOES IT MATTER?

A finding by Smith and Hakel (1979) is that job expert ratings on the PAQ correlate highly with ratings obtained from college students who are given no information about jobs other than their titles. One possible explanation for this finding is that the PAQ measures only trivial or common knowledge about work that both experts and naive observers possess. This particular view, of course, has serious implications for the use of the PAQ. In this paper we point out a problem in the way Smith and Hakel calculated the convergent validity between expert raters and naive raters. We also present the results from a replication study that indicate convergent validities are much lower than those reported by Smith and Hakel. Additional points are presented in order to caution against the interpretation that the PAQ measures only common knowledge about jobs.     

GENERALIZING PHYSICAL ABILITY TEST VALIDITY: A CASE STUDY USING TEST TRANSPORTABILITY, VALIDITY GENERALIZATION, AND CONSTRUCT-RELATED VALIDATION EVIDENCE

This paper describes a field study using Position Analysis Questionnaire (PAQ; McCormick, Jeanneret, & Mecham, 1972) data to support inferences of physical ability test (PAT) validity at a large utility company. The project developed a test transportability procedure using worker-oriented job analysis, and was based on synthesizing the findings of an earlier consortium project (American Gas Association, 1990), recent validity generalization evidence, and findings from a variety of physical ability construct-related research projects published or presented since the completion of the original consortium study. Selected PAQ divisional dimension scores for 95 jobs were analyzed via cluster analysis to develop five physical demand job families. Statistically derived job families were modified very slightly based on reviews by subject matter experts who found PAQ-derived job family structures quite reasonable (agreement of 95% between statistical and rational procedures). PAQ strength attributes were used to estimate the physical demands underlying job families. The statistical and analytic procedures developed and reported here could be applied in other organizational settings, and allow broader applicability of the PAT than would a task-based transportability model.     

Effects of analyst training and amount of available job related information on job analysis ratings

The present study investigated the effects of job analysis training and the amount of information available to the job analyst on job analysis ratings. Forty subjects were asked to complete the Position Analysis Questionnaire (PAQ) and the Job Components Inventory (JCI) for the position of College Resident Assistant. Results indicated that when using the PAQ as the job analysis instrument, there was no significant effect for either training or the amount of information received by the job analyst. For the JCI, the results indicated that trained analysts were more reliable and accurate than untrained analysts. Also, it was found that the more information received by the job analysts, the more reliable and accurate were their JCI ratings.     

JOB CLASSIFICATION APPROACHES AND THE IMPLEMENTATION OF VALIDITY GENERALIZATION RESULTS

This paper compares two job classification methods for showing the appropriateness of cognitive tests in settings that were not involved in supplying data for a validity generalization analysis. One approach was an elaborate quantitative procedure that involved a lengthy job inventory and a multivariate item analysis. This approach was shown to be highly successful when applied to the responses from 1179 job inventories collected in 54 petroleum-petrochemical plants from 30 different companies. The other procedure involved simple job classification judgments by supervisors and incumbents. This latter approach was shown to be as effective, but was much less time consuming and costly. Professional and legal implications of these findings are discussed.     

What Is Important on the Job? Differences Across Gender,Perspective, and Job Level1

The importance ratings of job competency dimensions on a 360‐degree feedback instrument were examined. We hypothesized that men (incumbents and bosses) would rate agentic behaviors higher in importance than would women, and that women (incumbents and bosses) would rate communal behaviors higher in importance than would men. Differences were found for men and women across rating sources and across job level (organizational position). The present findings suggest that men and women incumbents view different factors as important for their jobs and that people in higher level jobs place more importance on agentic behaviors than do those in lower level positions, and people in lower level jobs place more importance on communal behaviors than do those in higher level positions.     

Applications of Job Component/Synthetic Validity to Construct Validity

The origin and history of job component validity (JCV) and its relationship to construct validity are presented, followed by a summarization of how the methodology has been accomplished in research using the Position Analysis Questionnaire (PAQ). Data are presented that document the capability of both PAQ-based attribute profiles and predictions of aptitude data from PAQ job dimensions to establish job requirements. Finally, two practical ap- plications of JCV are described that illustrate the unique capability of the methodology to develop selection systems to screen individuals for jobs un- dergoing design or occurring in difficult-to-study environments.     

PAY PREFERENCES AND JOB SEARCH DECISIONS: A PERSON-ORGANIZATION FIT PERSPECTIVE

The present study investigated the degree to which pay preferences influenced job search decisions in both hypothetical and actual organizations, and the degree to which preferences for particular compensation attributes depended on job seekers' dispositional characteristics. Based on prior theory and research, we hypothesized that certain pay systems generally would be preferred by job seekers, that these pay systems would affect applicant attraction to organizations, and that different types of job seekers would be attracted to different types of pay systems. The sample comprised 171 college students who were seeking jobs during the study, and who represented six majors, three degree types, and two degree levels. Experimental policy-capturing results and results obtained about actual companies with which the job seekers would potentially interview supported hypotheses that organizations perceived to offer high pay levels, flexible benefits, individual-based pay, and fixed pay policies were more attractive to job seekers. Results further suggested that the attractiveness of these pay policies may be heightened by greater levels of fit between individual personality traits and compensation system characteristics.     

AN EMPIRICAL TEST OF A TRAIT-ORIENTED JOB ANALYSIS TECHNIQUE

A trait-oriented job analysis technique based on a checklist of 33 a priori carefully defined traits that encompass elements of the physical, mental, learned, motivational and social domains of the work world is described. The analysis identifies the relevant traits, their levels and weights, in relation to overall job performance. Results of discriminability tests were supportive of the job analysis technique and indicated that incumbents of jobs requiring a particular trait scored higher on measures (predictors) of that trait than incumbents of jobs not requiring that trait. Implications of the results for personnel selection and placement are discussed.     

EMPIRICALLY DERIVED JOB FAMILIES AS A FOUNDATION FOR THE STUDY OF VALIDITY GENERALIZATION

The purpose and background of a series of three studies dealing with validity generalization are discussed, an overview of the series is provided, and the initial study is described.
In the initial study a total of 76 insurance company jobs was analyzed by 203 raters in an effort to assess the potential usefulness of the Position Analysis Questionnaire (PAQ) as a job analysis device to be employed in a more extensive, companywide research program. Examination of rate-rerate and interrater reliability suggested that rate-rerate reliability was quite good, while interrater reliability estimates were somewhat lower. Cluster analysis techniques were applied to the five Overall and 17 Component dimensions of the PAQ yielding six job families in both analyses. The job families were described in terms of the various PAQ dimensions and were judged to be organizationally meaningful. Although the job clustering results were generally acceptable, it was conjectured that job families formed on the basis of company specific dimensions might prove even more meaningful.     

SYNTHETIC VALIDITY: PAST, PRESENT, AND FUTURE

Practitioners in the area of personnel selection can be confronted with situations in which the use of traditional validation strategies is not feasible. This situation can result from small numbers of incumbents, rapidly changing jobs, insufficient resources, or the lack of high-quality criterion data. Although traditional strategies are not possible, a set of techniques falling under the rubric of synthetic validity can be used to estimate predictor validity in these situations. Synthetic validity is a logical process of inferring validity on the basis of the relationships between components of a job and tests of the attributes that are needed to perform the job components. Unfortunately, synthetic validity is infrequently used and not well understood. This article provides an overview and conceptualization of synthetic validity, reviews the historical approaches to synthetic validity, describes the recent developments and trends, and suggests future directions for synthetic validity application and research. The relationship of synthetic validity with other validity concepts, the advantages, the limitations, and the legal issues are considered throughout.     

Measures of job perceptions: Gender and age of current incumbents,suitability, and job attributes

Two ways of examining the gender and age stereotypes of jobs, characteristics of current incumbents and potential suitability, were compared. Female (n = 70) and male (n = 66) college students, predominantly Caucasian ranging in age from 18 to 57 years, provided their gender and age perceptions for 58 jobs. Although the two concepts have not been clearly distinguished in the literature, they are conceptually and (as found here) empirically distinct. The important roles of current incumbents, suitability, and job attribute perceptions for discrimination research are presented.     

The Validity of Task Coverage Ratings by Incumbents and Supervisors: Bad News

In this study incumbents from three different jobs were asked to rate lists of their job tasks on various constructs (e.g., time, importance) and then estimate the percentage of their job tasks (task coverage) included in the task list. Incumbents made these ratings under one of two conditions. In two instances, two months after making an initial task coverage rating for the entire list of tasks, the same incumbents were asked to estimate the task coverage of a reduced list of tasks (i.e., half to two-thirds of the tasks were removed from the list presented for rating). In a third instance one group of incumbents completed an entire inventory while a second group completed a reduced inventory. The average task coverage rating for the entire inventories were high (percent estimates ranging in the 80's–90's) and the average of the reduced inventories was much higher than expected (percent estimates in the 70's). It was concluded that incumbents and supervisors were not able to accurately estimate task coverage.     

DETERMINING JOB GROUPS: APPLICATION OF HIERARCHICAL AGGLOMERATIVE CLUSTER ANALYSIS IN DIFFERENT JOB ANALYSIS SITUATIONS

The present study showed that researchers must consider underlying data structure when using hierarchical agglomerative cluster analysis to group jobs. Five cluster procedures were applied to four simulated data sets constructed to reflect common job analysis situations. The structures contained jobs varying in degree of task overlap, number of tasks performed, and relative number of people doing the jobs. Average linkage/distance was the most accurate procedure when jobs had highly positively correlated task profiles, a situation characteristic of jobs within a career family over a restricted range of levels. Average linkage/correlation was the most accurate for three other structures containing jobs whose profiles were not highly positively correlated. Such are characteristically found when analyzing (a) jobs in different functional units, (b) jobs over a wide range of hierarchical levels such as entry to advanced, and (c) jobs differing markedly in the number of incumbents.