Development of Standard Anchors for Scoring Generic Past-Behaviour Questions in Structured Interviews

A structured interviewing system was developed which included an item bank of pre-written past-behaviour questions. Scoring anchors for each of 168 questions in the item bank were created from analysis of protocols from individual assessments. A concurrent validation study was conducted, focusing on the job of highway patrol sergeant. Job knowledge experts used a job analysis tool to select a subset of twelve questions from the item bank. Panel interviews were conducted with 32 highway patrol sergeants as subjects. Average inter-rater reliability was 0.65 and interview validity was 0.42 for group consensus scores and 0.44 for panel average scores. Discussion focused on practical issues affecting the interviewing system.     

ITEM GENERATION PROCEDURES AND BACKGROUND DATA SCALES: IMPLICATIONS FOR CONSTRUCT AND CRITERION-RELATED VALIDITY

Background data measures are one of the best predictors of job performance. Nonetheless, questions have been raised about their content and construct validity. The present effort describes a set of procedures for developing construct and content valid background data items. Data gathered in seven field studies and six laboratory studies are presented bearing on the reliability and validity of the measures constructed using these item generation procedures. Findings in these studies indicate that construct-based item generation procedures yield reliable scales evidencing both content and construct validity. Furthermore, these scales are capable of predicting performance in a variety of settings. Theoretical and practical implications of these findings for the development and validation of background data measures are discussed.     

PERFORMANCE APPRAISAL, PROMOTION AND THE COURTS: A CRITICAL REVIEW

Twenty-three Title VII court cases were reviewed in order to determine the standards set by the courts in their assessment of performance appraisal systems when used as the basis for promotion decisions. The topics covered were adverse impact determination, the courts' adjudication strategy, and the evidence needed to justify the performance appraisal procedures. Among the major findings was the courts': (1) failure to adhere to the "applicant flow technique" of adverse impact determination, (2) interest in assessing performance appraisal systems regardless of their adverse impact, (3) ignorance regarding acceptable validation procedures, and (4) focus on objectivity in lieu of validity. The discussion offered suggestions to employers for developing a professionally sound and legally defensible appraisal system.     

QUANTITATIVE APPROACHES TO JOB CLASSIFICATION: A REVIEW AND CRITIQUE

Job classification decisions underlie nearly all personnel functions (e.g., grouping for test validity generalization, compensation, performance appraisal, training needs analysis, career path and succession planning). Pearlman (1980) performed a comprehensive review of the job-family literature, paying special attention to the problem of forming job families when generalizing aptitude test validities. This review addresses a more specific topic: selecting among and using the numerous quantitative job-classification procedures, with a focus on the decision-making tasks and practical difficulties that confront users of each.     

EVALUATION OF TASK AND JOB SKILL LINKAGE JUDGMENTS USED TO DEVELOP TEST SPECIFICATIONS

The research on job analysis judgments, such as "time spent," has been relatively limited, particularly with reference to external criteria remote from the job analysis operation. The more complex job analysis judgment linking a job skill to specific tasks or duties has not been systematically examined. While it would appear that a simple scaling of importance of a skill for a task or duty or a retranslation judgment would suffice, the fact is that a single job skill may be a prerequisite for performance in a variety of tasks, and any one task may require multiple skills of varying levels for effective performance. With a multiple assignment of tasks to job skills, the evaluation becomes considerably more difficult. In the present study, a sequence of statistical evaluations was conducted to examine, first, the reliability of the subject matter expert (SME) panel's association of tasks and job skills and, second, the factor structure of the task by job skill relationship. The results are discussed with reference to developing selection test specifications and test budgets.     

Development and validation of the job crafting scale

We developed and validated a scale to measure job crafting behavior in three separate studies conducted in The Netherlands (total N = 1181). Job crafting is defined as the self-initiated changes that employees make in their own job demands and job resources to attain and/or optimize their personal (work) goals. In Study 1 and 2 the Dutch job crafting scale (JCS) was developed and tested for its factor structure, reliability, and convergent validity. The criterion validity of the JCS was examined in Study 3. The results indicated that there are four independent job crafting dimensions, namely increasing social job resources, increasing structural job resources, increasing challenging job demands, and decreasing hindering job demands. These dimensions could be reliably measured with 21 items. The JCS shows convergent validity when correlated with the active constructs proactive personality (+), personal initiative (+), and the inactive construct cynicism (?). In addition, results indicated that self-reports of job crafting correlated positively with colleague-ratings of work engagement, employability, and performance — thus supporting the criterion validity of the JCS. Finally, self-rated job crafting behaviors correlated positively with peer-rated job crafting behaviors.     

OPERATIONALIZING THE "BEHAVIORAL CONSISTENCY" APPROACH: SELECTION TEST DEVELOPMENT BASED ON A CONTENT-ORIENTED STRATEGY

This paper delineates a systematic procedure for operationalizing the "behavioral consistency" notion, proposed by Wernimont and Campbell (1968), for a Civil Service job. The steps used in developing selection tests from a content-oriented strategy are illustrated, and the transformation of specific job behaviors into tests related to job content but modified so that inexperienced applicants can complete them is demonstrated. Test reliability and judged content validity of the examinations are presented.     

An openness scale for the California Psychological Inventory

We developed a 36-item scale to measure Openness, using items on the California Psychological Inventory (CPI; Gough, 1957, 1987, 1996), Form 434. Items were initially chosen on the basis of content validity. Five samples (N = 2,375) were used to establish reliability, validity, and norms; 4 samples consisted of university undergraduate students, and 1 comprised applicants for nonmanagement call centerjobs. Internal consistency estimates obtained in each sample averaged approximately .75, and test-retest stability, assessed in 1 sample, was estimated at .84. Cross-correlations with related scales, for example, the NEO Personality Inventory-Revised Openness scale (Costa & McCrae, 1992) and other CPI-based scales, provided evidence of construct validity. Statistically significant predictive validities were obtained in 2 call centerjob-incumbent samples, with range-corrected true validities of .20 to .36 for a number of job performance criteria. Construct and predictive validity were found to be higher than for other scales consisting of CPI items designed to measure Openness or a related construct. Finally, norms were prepared for university undergraduate students (n = 1,847) and nonmanagement service-sector job applicants (n = 528).     

The Development of an Integrated Performance Category System for Supervisory Jobs in the U.S. Navy

When a single job is the target, established job analysis approaches provide relatively straightforward methods for identifying performance requirements. However, when multiple jobs are involved, the identification of a single set of relevant performance dimensions can be more daunting. In the application here, there was also a compelling requirement to develop a dimension set that conformed with U.S. Navy fleet personnel perceptions of the performance domain. Accordingly, the behavioral performance constructs that experienced Navy officers believed differentiate effective from ineffective supervisory performance were gathered using a personal construct theory protocol. We then used a methodology suggested by Borman and Brush (1993) to integrate and summarize these personal work constructs, to provide—through a series of qualitative and quantitative strategies—a dimension set targeted toward supervisors in all Navy communities. The resulting dimensions revealed certain themes that might not have emerged if traditional job analysis strategies had been used. The dimension set and the resulting behaviorally anchored rating scales appear widely relevant to Navy supervisor jobs, and the performance appraisal system that employs the scales is nearing implementation. In addition, performance feedback and development tools were developed to complement the performance appraisal system.     

A counterintuitive hypothesis about employment interview validity and some supporting evidence

This study found mixed support for the hypothesis that the difference in criterion-related validity between unstructured and structured employment interviews is due solely to the greater reliability of structured interviews. Using data from prior meta-analyses, this hypothesis was tested in 4 data sets by using standard psychometric procedures to remove the effects of measurement error in interview scores from correlations with rated job performance and training performance. In the 1st data set. support was found for this hypothesis. However, in a 2nd data set structured interviews had higher true score correlations with performance ratings, and in 2 other data sets unstructured interviews had higher true score correlations. We also found that averaging across 3 to 4 independent unstructured interviews provides the same level of validity for predicting job performance as a structured interview administered by a single interviewer. Practical and theoretical implications are discussed.     

SEX BIAS IN JOB EVALUATION PROCEDURES

Issues pertaining to possible sex bias in job evaluation procedures are outlined and relevant research pertaining to these issues reviewed. Attention is given to possible sex bias in job analysis procedures, choice and weighting of factors, reliability and validity issues, and other concerns. Future research needs are discussed, with particular attention being paid to the reliability and validity aspects of job evaluation instruments.     

The development of a measure to assess vocational maturity in adults with delayed career development

The development and validation of a measure of vocational maturity for educationally and economically “disadvantaged” adults who experience delayed career development is described. The Adult Vocational Maturity Assessment Interview (AVMAI) measures the attitudinal, cognitive, and behavior aspects involved in coping with exploratory and early entrance stage vocational development tasks. The measure employs a structured interview format accompanied by a detailed content scoring manual. It includes 120 items which yield scores for eight scales: Orientation to Education; Orientation to Work; Concern with Choice; Self-Appraisal: Interests and Abilities; Self-Appraisal: Personality Characteristics; Self-Appraisal: Values; Exploring Occupations; and Using Resources. Evidence for the measure's reliability, content, and construct validity is presented by examining the internal consistency of the scales, the reliability of its scoring procedures, its interscale correlations, and its performance in a pretest-post-test experimental situation. Research and counseling uses of the measure are discussed in the context of the need for vocational psychologists to develop and use measures that are appropriate for the specific career development problems of different groups in society.     

A METHODOLOGICAL APPROACH TO JOB CLASSIFICATION FOR PERFORMANCE APPRAISAL PURPOSES

This paper illustrates a methodological approach to classifying jobs for the purpose of developing performance evaluation instruments. The approach was to administer a worker-oriented job inventory to 2,023 incumbents across several jobs and several levels of responsibility (rank) in the U. S. Coast Guard. The data from the job inventory were then analyzed using Tucker's Three-Mode Factor Analysis. Output from the three-mode factor analysis was used to identify combinations of jobs and ranks for which separate appraisal instruments could be developed. In addition, output from the three-mode factor analysis was used to suggest the content of the various appraisal instruments. Advantages, applications, and limitations of this approach are discussed.     

GENERALIZING PHYSICAL ABILITY TEST VALIDITY: A CASE STUDY USING TEST TRANSPORTABILITY, VALIDITY GENERALIZATION, AND CONSTRUCT-RELATED VALIDATION EVIDENCE

This paper describes a field study using Position Analysis Questionnaire (PAQ; McCormick, Jeanneret, & Mecham, 1972) data to support inferences of physical ability test (PAT) validity at a large utility company. The project developed a test transportability procedure using worker-oriented job analysis, and was based on synthesizing the findings of an earlier consortium project (American Gas Association, 1990), recent validity generalization evidence, and findings from a variety of physical ability construct-related research projects published or presented since the completion of the original consortium study. Selected PAQ divisional dimension scores for 95 jobs were analyzed via cluster analysis to develop five physical demand job families. Statistically derived job families were modified very slightly based on reviews by subject matter experts who found PAQ-derived job family structures quite reasonable (agreement of 95% between statistical and rational procedures). PAQ strength attributes were used to estimate the physical demands underlying job families. The statistical and analytic procedures developed and reported here could be applied in other organizational settings, and allow broader applicability of the PAT than would a task-based transportability model.     

A Concurrent Validity Study Applied to a Secretarial Position in a State Universities Civil Service System

In a concurrent validity study, a comprehensive job analysis of a mid-level secretarial position resulted in the development of highly valid employment selection instruments. Six hundred fifty-nine supervisors and 883 incumbents in 20 locations participated in the job analysis. Scores from the selection test correlated with composite ratings from a research performance appraisal (RPA) .41 (p<.001). Two forms of the test (A and B) were developed. Form A and Form B test scores and RPA composite ratings correlated .55 and .48 (p<.001) respectively. The unbiased estimate of equivalence reliability of Form A and Form B was .94. The two tests correlated .89 (p<.001).     

A TECHNOLOGY FOR PROGRAM MAINTENANCE: PROGRAMMING KEY RESEARCHER BEHAVIORS IN A STUDENT HOUSING COOPERATIVE

Behavioral researchers play critical, but often unanalyzed, roles in the programs they develop. Unless they replace their key activities with standardized procedures, their continued participation may be essential to program success—a situation that is often not only impractical but may be prohibitively expensive and disliked by local staff. This study was conducted in a student housing cooperative that is dependent on close researcher supervision for its continued health and survival. A key activity of the co-op researchers was to provide public recognition for good job performance by co-op members. The purposes of this study were (a) to replace that idiosyncratic recognition with systematic procedures so members, instead of the researchers, would provide public recognition to each other for good job performance; and (b) to evaluate those procedures by comparing job performance when member-delivered recognition was provided and when it was not. When the procedures were in place, job performance increased and fines for poor job performance and complaining at meetings decreased. This study suggests that procedures can be developed to reduce program reliance on the researcher that are effective, inexpensive, sustainable, and acceptable to the participants—a first step toward developing a technology of program maintenance.     

BEHAVIORAL OBSERVATION SCALES FOR PERFORMANCE APPRAISAL PURPOSES

Behavioral items ( N = 78) critical to the job success of logging supervisors were developed from 1204 critical incidents, the frequency with which a supervisor ( N = 300) engaged in each behavior was rated on a 5-point Likert type scale by two sets of observers. A factor analysis reduced the items to 38 and 33, respectively, for the two sets of observers which in turn constituted 10 and 11 factors or criteria for performance evaluation purposes. Multiple regression equations based on composite scores were used to predict cost-related measures of logging crew effectiveness. The shrinkage in Rs after double cross-validation was moderately small. Moreover, the behavioral observation scales (BOS) that were developed by factor analyzing the observation ratings had moderately high reliability and accounted for more variance in the cost-related measures than did the BOS developed by traditional judgmental clustering techniques. The similarities and differences between BOS and BES procedures are discussed.     

Validation of the Supervisory Analyst Exam (Series 16) by the New York Stock Exchange

The Supervisory Analyst (Series 16) Examination is a criterion-referenced licensure/certification examination used to qualify persons for supervisory positions in financial data analysis for New York Stock Exchange member firms. A job analysis was conducted to update the examination to ensure its content validity. The analysis revealed 4 major tasks performed by the supervisory analyst. KSAs were determined for each of the tasks. The KSAs were distributed between two parts: 1) Securities Analysis, and 2) Regulatory Administration. Multiple-choice items were written for each part, and the distribution of items was designed to reflect the relative weights of the KSAs as determined by the job analysis.     

初中生数学自我效能问卷的编制

根据国内外关于数学自我效能研究的文献,结合我国初中生数学学习的内容特点与实际背景情况,编制数学自我效能问卷。研究以初一与初二学生为对象,采用开放式问卷对188名学生和53名数学教师进行调查,首先编制出44个项目的初测问卷;采用封闭式问卷对479名学生的初测结果进行验证性因素分析与项目分析,形成了包含日常生活中数学任务的效能、数学相关课程的效能与数学学业问题解决效能三个维度的26个项目的正式问卷;最后对350名学生施测正式问卷。分析研究结果表明,该数学自我效能问卷的因素结构清晰,具有合理的信度和效度。