The value of relative-time-spent ratings in task-oriented job analysis

Recent studies have called for the abandonment of the relative-time-spent scale in task inventories. This recommendation is based on findings that the job profile created with the scale data is highly correlated with the profile created from a much simpler “Do you perform this task?” checklist. We examined this issue using 3 inventories and 42 jobs (N=2252). Profile correlations were computed on only the tasks actually performed by incumbents to avoid possibly inflated rs due to including irrelevant tasks. The specificity of task inventory items was proposed as an explanation for the high correlation between the two job profiles. Specificity of items was examined by looking at both the type (job duties versus tasks) and the amount (number of items in job profile and average number of items relevant to each job) of items used in the inventory. Correlations between time spent and checklist profiles were in the .80's and .90's regardless of the number of irrelevant tasks or the specificity of tasks. We agree with previous military research and conclude that the relative-time-spent scale has limited incremental utility beyond a dichotomous checklist.     

JOB ANALYSIS: THE COMPOSITION OF SME SAMPLES

It is common for job analysts to solicit information from incumbents and supervisors (Subject Matter Experts or SMEs) when conducting a job analysis. These SMEs are asked to provide ratings on salient dimensions (e.g., frequency and importance of tasks that comprise the job). In constructing samples of SMEs for this purpose, it is reasonable to consider any possible influences that might bias or systematically influence the task ratings. The present paper considers the possible influence of SME demographic characteristics on task ratings of frequency. The tasks comprising the job of patrol officer in a large city were rated for frequency by approximately 700 incumbents. These ratings were gathered in two different years, 1982 and 1984. The total data set was used to conduct a components analysis of the 444-item task inventory. The first eight principal components were considered the dependent variables and four demographic characteristics the independent variables in an analysis of the 1982 data set. Analysis of variance and follow-up tests indicated that incumbent experience had a substantial influence on task ratings. Educational level and race had minimal effects on ratings. The sex factor was confounded by the experience factor, making interpretation of the sex effect equivocal. Mechanisms that might account for the experience effect are discussed. It was concluded that incumbent experience is a salient issue in job analysis using SME groups.     

Does Acquisitive Self‐Presentation in Personality Self‐Ratings Enhance Validity? Evidence from two experimental field studies

The aim of the present research was to demonstrate that acquisitive self‐presentation in personality scales is not a barrier to their criterion‐related validities in human resource contexts, but rather a means to improve them. A pilot study (Study 1) with 96 job incumbents provided preliminary positive evidence. In Study 2, in the experimental group (n=99), the instructions asked job incumbents to work on a Big‐Five personality inventory (BFI‐K) as if they took part in a personnel selection procedure for a personally very attractive position. In the control group (n=93) of Study 2, job incumbents were asked to answer the inventory items honestly. As expected, the correlation between the self‐ratings of the motive to get along (i.e., which comprises emotional stability, conscientiousness, and agreeableness) and contextual performance assessments was significantly higher in the experimental group than in the control group in Study 2. Additionally, the correlation between the self‐ratings of the motive to get ahead (i.e., which comprises extraversion and openness to experience) and task performance and leadership assessments was significantly higher in the experimental group than in the control group in Study 2. It is proposed that responding to a personality inventory in a human resource decision situation should be conceptualized as a workplace simulation.     

Validation of personality and cognitive measures for insurance claims examiners

Claims Examiners are critical for the profitability of large insurance companies. Mistakes in hiring these people can result in improperly processed claims and/or turnover, and this yields negative financial consequences for the company. This paper describes research evaluating the validity of selection procedures for the position of claims examiner at Blue Cross and Blue Shield of Oklahoma. A job analysis indicated that both cognitive skills and personal abilities are necessary for successful performance as a claims examiner. The job analysis also indicated that both subjective and objective criterion measures reflect important aspects of job performance. The criterion data included nine supervisory ratings, employee nominations, average percent of performance, absences, disciplinary actions, and sick leave. In a concurrent validation strategy, we tested a sample of experienced incumbents (N=50) using four cognitive ability tests (Basic Skills Tests: BST) and two measures of personality (Hogan Personality Inventory and PROFILE). Scales from two personality inventories were combined to form the Claims Examiner Inventory (CEI). Scores from the CEI and three of the four cognitive tests correlated significantly with the average percent of performance and six of the supervisory ratings, including the average supervisory rating. In addition to the cognitive measures, the personality measure contributed significantly to the prediction of percent of performance achieved with R=.64. These results are consistent with other research findings that support the use of both personality and cognitive measures as predictors of job performance.     

JOB CLASSIFICATION APPROACHES AND THE IMPLEMENTATION OF VALIDITY GENERALIZATION RESULTS

This paper compares two job classification methods for showing the appropriateness of cognitive tests in settings that were not involved in supplying data for a validity generalization analysis. One approach was an elaborate quantitative procedure that involved a lengthy job inventory and a multivariate item analysis. This approach was shown to be highly successful when applied to the responses from 1179 job inventories collected in 54 petroleum-petrochemical plants from 30 different companies. The other procedure involved simple job classification judgments by supervisors and incumbents. This latter approach was shown to be as effective, but was much less time consuming and costly. Professional and legal implications of these findings are discussed.     

Is the Political Skill Inventory Fit for Personnel Selection? An Experimental Field Study

The political skill inventory (PSI) assesses social effectiveness in organizations by self‐reports and has demonstrated strong evidence of validity. It was the purpose of this experimental field study to investigate construct and criterion‐related validity of the PSI when used under conditions of personnel selection. In the experimental group (n=102), the instructions asked job incumbents to work on the PSI, a social desirability scale, and a Big‐Five personality inventory as if they took part in a personnel selection procedure for a personally very attractive position. Additionally, they were asked to report yearly income. In the control group (n=110), job incumbents were asked to answer the items honestly. As expected, in both conditions, the PSI did not correlate with social desirability, but it correlated positively with extraversion, conscientiousness, and income, and negatively with neuroticism, thus demonstrating construct and incremental criterion‐related validity under both conditions. Implications and limitations are discussed.     

A COMPARISON OF GLOBAL JUDGMENT VS. TASK ORIENTED APPROACHES TO JOB CLASSIFICATION

The purpose of this study was to determine the extent to which direct judgments of similarity by supervisors and incumbents could provide the same job classification results as a more elaborate job analysis procedure involving measures of task overlap among jobs. To accomplish this, 8 foreman jobs in a chemical processing plant were analyzed and compared on 237 task statements. In addition, 15 foremen incumbents and 17 supervisors evaluated the similarities among the same 8 foremen jobs in a paired comparisons rating task. The task-oriented job analysis required hundreds of man-hours to complete; the rating task took 15 minutes. Results using hierarchical cluster analysis and multidimensional scaling analysis revealed that the global judgments and the task-oriented data led to identical conclusions. Also, it was found that incumbent ratings produced the same results as ratings from supervisors. Uses, advantages, and disadvantages of the procedure are outlined.     

Evaluation of an index to detect inaccurate respondents to a task analysis inventory

An infrequency index assesses the tendency of respondents on job analysis questionnaires to rate job-irrelevant tasks as important. The purpose of the study was to evaluate whether this index could detect respondents who made rating errors on job-relevant tasks. The respondents were incumbents of three jobs: mental health workers (N=254), clerical workers (N=102), and police corporals (N=118). Differences in interrater reliability between low and high scorers on the infrequency index were found for the mental health workers on one of two response scales. For all three samples, individuals who scored higher on the infrequency index rated job-relevant tasks significantly higher. It was concluded that the index can detect respondents making constant error due to response sets and distortions. However, the usefulness of the index to detect individuals making random errors may be limited. On the basis of these results and on conceptual grounds, it was argued that job analysts should incorporate the infrequency index as part of their task analysis questionnaires.     

The impact of raters' cognition on judgment accuracy: An extension to the job analysis domain

Eighty-six incumbents of three different jobs produced job-analytic ratings using either a decomposed (task-based) or a holistic (job-based) rating strategy. Approximately half of them received rater training in making inferential decisions. When tasks were less complex than the job as a whole, rating decomposition generally had positive effects on ratings' quality. Similarly, when the number of tasks rated was low to moderate, rater training was effective. A contingency approach, where limitations concerning the use of rating decomposition and inferential training were outlined, should serve to inform future uses and theories of rating aids in job analysis.We would like to acknowledge David Dorsey for his significant contribution to the rater training program. This article is a summary of the doctoral dissertation of Juan Sanchez, which was conducted under the supervision of Edward Levine.     

Introduction to the Special Issue: Personality and the Prediction of Job Performance: More Than the Big Five

The effects of motivated distortion on forced-choice (FC) and normative inventories were examined in three studies. Study 1 examined the effects of distortion on the construct validity of the two item formats in terms of convergent and discriminant validity. The results showed that both types of measures were susceptible to motivated distortion, however the FC items were better indicators of personality and less related to socially desirable responding when participants were asked to respond as if applying for a job. Study 2 considered the criterion-related validity of the inventories in terms of predicting supervisors' ratings of job performance, finding that distortion had a more deleterious effect on the validity of the normative inventory with some enhancement of the validity of the FC inventory being observed. Study 3 investigated whether additional constructs are introduced into the measurement process when motivated respondents attempt to increase scores on FC items. Results of Study 3 indicated that individuals higher in cognitive ability tend to have more accurate theories about which traits are job-related and therefore are more successful at improving scores on FC inventories. Implications for using personality inventories in personnel selection are discussed.     

Structuring knowledge retrieval: An analysis of decomposed quantitative judgments

Subjects were asked to estimate the answers to 16 questions concerning uncertain quantities like, “How many people are employed by hospitals in the U.S.?” under five different aiding conditions. The most-aided group (Full Algorithm) was given a complete algorithm and asked to make estimates for all the parts of the algorithm and to combine the parts as indicated to arrive at an estimate of the desired quantity. The second group (Partial Algorithm) was given the same algorithm without indications of how to combine the parts. After making estimates of the parts, these subjects then estimated the desired quantity. The third group (List & Estimate) was asked to list components or factors they thought were relevant, make an estimate of each item on their list, and then estimate the desired quantity. The fourth group (List) was asked to make such a list, but they were not asked to make estimates of each item before making an estimate of the desired quantity. The fifth group received no aid. The results generally showed improved performance in terms of both accuracy and consistency across subjects with increasing structure of the aid. Generalization of these results to practical estimation situations is possible but limited by the need, in real situations, for the estimator to develop the algorithm, a task that was done here by the experimenters.     

Analyzing Job Analysis Data Using Mixture Rasch Models

An important piece of validity evidence to support the use of credentialing exams comes from performing a job analysis of the profession. One common job analysis method is the task inventory method, where people working in the field are surveyed using rating scales about the tasks thought necessary to safely and competently perform the job. This article describes how mixture Rasch models can be used to analyze these data, and how results from these analyses can help to identify whether different groups of people may be responding to job tasks differently. Three examples from different credentialing programs illustrate scenarios that can be found when applying mixture Rasch models to job analysis data. Discussion of what these results may imply for the development of credentialing exams and other analyses of job analysis data is provided.     

Projection of Own on Others' Job Characteristics: Evidence for the False Consensus Effect in Job Characteristics Information

The present study is an examination of the influence of rater characteristics as a source of bias in job characteristics information, as gleaned from the Job Diagnostic Survey (JDS). Participants (N=133) completed an initial JDS for own job characteristics. Non‐incumbents (n=104) then completed a secondary JDS in order to rate incumbents' job characteristics. Non‐incumbents' personality dimensions, own job characteristics, job satisfaction level, and incumbent satisfaction level were hypothesized as predictors of ratings of job characteristics of incumbents. Results indicate that two characteristics, job autonomy and dealing with others, were influenced by non‐incumbents standing on these characteristics, and demonstrated evidence of the false consensus effect. Limitations and implications for interpreting job information are discussed.     

Predicting Performance with Contextualized Inventories,No Frame‐of‐reference Effect?

A recent meta‐analysis showed that contextualized personality inventories have incremental predictive validity over generic personality inventories when predicting job performance. This study aimed to investigate the differences between two types of contextualization of items: Adding an ‘at work’ tag versus completely modifying items. One hundred thirty‐nine pharmacy assistants from 29 pharmacies filled out a generic, a tagged, and a completely modified personality inventory. The assistants also provided participant reactions for each of the personality inventories. Performance ratings were collected from the supervising pharmacists. We expected to find incremental criterion validity for both the tagged inventory and the completely modified inventory for predicting job performance. However, the results showed an unexpected decrease in predictive validity for the contextualized inventories. Contextualized inventories were liked less than the generic inventory, but were considered somewhat more face valid and predictive by the participants.     

The Practical Effects of Bias on Compensable Factor Weights in Job Evaluation: A Policy-Capturing Experiment

Subjects (n=64) were shown videotapes of actors portraying incumbents of three jobs, then they completed job evaluation forms for two of them and indicated what they felt these jobs should be paid relative to the third. Job evaluations using the compensable factor weights determined in policy-capturing analyses were then conducted. Results of the job evaluations depended on whether effects of job type and gender bias on compensable factor weights were controlled. Failure to control for these biases produced a pronounced effect on job classification decisions. The estimate of this effect was three to four pay grades. Implications for the determination of weights in job evaluation plans are provided.     

Five Factor Biodata Inventory: resistance to faking

Faking on a biographical inventory compared to a traditional personality inventory was assessed in measuring the Five Factor Model of Personality. 705 subjects were randomly assigned to either an Answer Honestly or Faking condition. All subjects were recruited from psychology classes at two New Jersey State colleges. Women comprised 68.6% of the participants. The average age of the subjects was 24 yr. 383 subjects took part in the Answer Honestly condition. 322 participated in the Faking condition. In the Faking condition, subjects responded as if applying for the position of librarian. All subjects completed a biodata inventory, the NEO-Five Factor Inventory, a social desirability scale, a letter-cancellation task, and self-reported their grade point average. Criterion-related validity was assessed for both test scores across samples. Comparisons between samples indicated that subjects inflated scores on both inventories in socially desirable directions. Biodata Inventory scores were less elevated under the Faking conditions than the NEO-Five Factor Inventory scores.     

The effects of cognitive style and training on performance ratings' validity

This study investigated how personal cognitive style and training effect rating validity with two different rating tasks. Male undergraduate volunteers (n = 53) served as raters and rated videotaped lecturers. Using the Embedded Figures Test to measure cognitive style, two groups of raters were formed: those who tend to structure information presented (articulated) and those who do not (global). Half of each cognitive style received observational training designed to be congruent with the behavioral rating task. All raters completed two rating tasks: one requiring an evaluative judgment and one requiring a judgment of behavior frequency. It was hypothesized that with the evaluative rating task, cognitive style would be and training would not be a significant predictor of validity, because the training was not relevant to the task. It was also hypothesized that with the observational task training would improve rating validity (overcoming cognitive style), because the training was relevant to the rating task. Both hypotheses were supported.I wish to thank Dr. Kevin Murphy for the use of the videotapes.     

任职者任务绩效水平对其工作分析结果的影响

在控制了个体人口统计学变量和排除组织水平变量的影响下,探讨了电厂设计和编辑两个职位任职者的任务绩效水平对于他们Occupational Information Network (以下缩写为O*NET)工作分析问卷评价结果的影响。层次回归结果发现,控制了相关因素后,发电厂设计人员的任务绩效水平能显著影响其对技术性技能的水平评价,编辑的任务绩效水平能显著影响信息处理的重要性评价和水平评价,在一定程度上证实了“工作分析结果差异来源于真实差异”的理论