Predicting Training and Job Performance for Transit Operators

Although considerable meta‐analytic research has validated the use of cognitive ability tests, structured interviews, and personality tests with training and job performance criteria, few studies have investigated the validity of these measures with transit operators. There are the only two single studies of concurrent validation research specifically with transit operators. This article presents the results of a predictive validation study conducted with transit operator applicants for a large urban transit authority in Canada. Key knowledge, skills, and abilities were determined for the role and used as a basis for the design and choice of predictors and criteria. Four predictors were used in the study: education, cognitive ability, personality assessment, and structured interview. Criteria included training performance (formative and summative), probationary performance, preventable accidents, and lost time injuries. Validation results supported cognitive ability, structured interview, and several personality factors as predictors of training performance, but less so for job performance. The use of formative training ratings greatly augmented the evidence supporting the predictors beyond typical organizational criteria.     

Measuring Customer Service Orientation Using a Measure of Interpersonal Skills: A Preliminary Test in a Public Service Organization

Organizations are placing increased emphasis on identifying individuals with customer service orientation. In the present investigation we test whether interpersonal skills, as measured through Holland and Baird's (1968) Interpersonal Competence Scale, provides a narrow, yet valid, measure of customer service orientation. Data were collected from a sample of bus transit operators. Interpersonal skills was positively related to operator self-reported performance, but was not related to supervisor ratings or objective measures of performance. Implications for the study and use of broad versus narrowly defined personality constructs in organizational settings are discussed.     

Selection Test Anxiety: Investigating applicants' self‐ vs other‐referenced anxiety in a real selection setting

Test anxiety has received limited attention in personnel selection research, although it may impair the test performance of applicants. This paper describes the development and validation of a new two‐dimensional measure of applicants' test anxiety, namely the Self‐ versus Other‐Referenced Anxiety Questionnaire (SOAQ), that embeds worrisome cognitions of anxious applicants in the social evaluative context of ‘self’ (Self‐Referenced Anxiety) and ‘significant others’ (Other‐Referenced Anxiety). An exploratory factor analysis (calibration sample), followed by a confirmatory factor analysis (validation sample) and correlations with several proximal and distal theoretical constructs indicated satisfactory psychometric properties and construct validity for both SOAQ scales. Structural equation modeling further showed a differential impact of Self‐ and Other‐Referenced Anxiety on applicants' test performance within a real personnel selection context. The scientific and practical relevance of these findings are discussed.     

Skilled anticipation in real-world tasks: measurement of attentional demands in the domain of tennis

Anticipatory skills are characteristic of expert performance in a range of dynamic domains. However, models of the processes underlying such skilled anticipation remain limited. This study describes the construction and validation of a video-based test of anticipation in tennis and examined whether video test performance is less attentionally demanding for expert tennis players than for novices. Expert players performed the anticipation test more effectively than novices and were less vulnerable to dual-task decrement. This article discusses these findings in relation to models of skilled anticipation. The video test could be used for both selection and identification of training needs.     

Computerized Neurocognitive Testing in the Management of Sport-Related Concussion: An Update

Since the late nineties, computerized neurocognitive testing has become a central component of sport-related concussion (SRC) management at all levels of sport. In 2005, a review of the available evidence on the psychometric properties of four computerized neuropsychological test batteries concluded that the tests did not possess the necessary criteria to warrant clinical application. Since the publication of that review, several more computerized neurocognitive tests have entered the market place. The purpose of this review is to summarize the body of published studies on psychometric properties and clinical utility of computerized neurocognitive tests available for use in the assessment of SRC. A review of the literature from 2005 to 2013 was conducted to gather evidence of test-retest reliability and clinical validity of these instruments. Reviewed articles included both prospective and retrospective studies of primarily sport-based adult and pediatric samples. Summaries are provided regarding the available evidence of reliability and validity for the most commonly used computerized neurocognitive tests in sports settings.     

Video‐based Testing: A high‐fidelity job simulation that demonstrates reliability,validity, and utility

A conceptual rationale is presented for the use of a high‐fidelity simulation: a video‐based test (VBT), in which applicants view job‐related scenarios, respond orally, and are later scored on five dimensions by trained raters. Using applicant data, we empirically investigate the criterion‐related validity, reliability, and utility of the VBT. Overall, we found that the VBT predicted performance in training and on the job. In addition, different raters strongly agreed with one another on each applicant's scores. Despite somewhat higher development costs, the VBT demonstrated high utility. We suggest that organizations consider the use of VBTs as a selection instrument for jobs that require the use of applied social and interpersonal skills.     

Psychometric Properties of Parent–Child (0–5 years) Interaction Outcome Measures as Used in Randomized Controlled Trials of Parent Programs: A Systematic Review

This systematic review sought to identify observational measures of parent–child interactions commonly implemented in parenting program research, and to assess the level of psychometric evidence available for their use with this age group. Two separate searches of the same databases were conducted; firstly, to identify eligible instruments, and secondly to identify studies reporting on the psychometric properties of the identified measures. Five commercial platforms hosting 19 electronic databases were searched from their inception to conducted search dates. Fourteen measures were identified from Search 1; a systematic search of randomized controlled trial evaluations of parenting programs. For Search 2, inclusion/exclusion criteria were applied to 1327 retrieved papers that described the development and/or validation of the 14 measures identified in Search 1. Seventeen articles met the inclusion criteria, resulting in five observational measures for the final review. Data were extracted and synthesized using the COSMIN rating system to describe the methodological quality of each article alongside the overall quality rating of the psychometric property reported for each measure using the Terwee checklist. Measure reliability was categorized into four domains (internal consistency, test-re-test, inter-rater, and intra-rater). Measure validity was categorized into four domains (content, structural, convergent/divergent, and discriminant). Results indicated that the majority of psychometric evidence related to children aged from birth the three with internal consistency, inter-rater reliability, and structural validity the most commonly reported properties, although this evidence was often weak. The findings suggest further validation of the included measures is required to establish acceptability for the whole target age group.

     

Predicting sales skills

The purpose of this study was to validate a selection process designed to predict sales skills for a telephone sales job. First, a job analysis was performed (n=213) to identify the major functions and behavioral requirements of the job. Next, the test, a situational judgment inventory, was developed, and a concurrent criterion related validation study was completed (n=236). The criterion measures were supervisory ratings of performance. Validity coefficients ranged from r=.16 (p<.01) to r=.24 (p<.001). In conclusion, the situational judgment inventory was found an effective vehicle for measuring telephone sales skills.Sincere thanks to Roz Lobus for her assistance in the earlier phases of this study.     

Option weights should be determined empirically and not by experts when assessing knowledge with multiple‐choice items

Multiple‐choice tests are frequently used in personnel selection contexts to measure knowledge and abilities. Option weighting is an alternative multiple‐choice scoring procedure that awards partial credit for incomplete knowledge reflected in applicants’ distractor choices. We investigated whether option weights should be based on expert judgment or on empirical data when trying to outperform conventional number‐right scoring in terms of reliability and validity. To obtain generalizable results, we used repeated random sub‐sampling validation and found that empirical option weighting, but not expert option weighting, increased the reliability of a knowledge test. Neither option weighting procedure improved test validity. We recommend to improve the reliability of existing ability and knowledge tests used for personnel selection by computing and publishing empirical option weights.     

Combining personality and cognitive ability predictors for hiring service-oriented employees

Recent developments in personality theory and assessment have led to calls for inclusion of personality measures in personnel selection. A measure of service-oriented personality was used in conjunction with cognitive and perceptual ability tests in a concurrent validation study of 202 medical clerical positions. The ability tests explained 5% of criterion variance; an additional 5–8% was accounted for by the personality measures. The results were interpreted as tentative support for broadening use of personality measurement in personnel selection, although a need for greater attention to construct validation was also emphasized.We would like to acknowledge the financial assistance provided by the University of Colorado Graduate School of Business for data entry and analysis, as well as the cooperation of the National Computer Systems Corporation and of Kaiser Permanente and its employees. We are also indebted to Garth Coombs, Deborah Crown, and Chuck Hulin for their insightful comments.     

The Early Numeracy Test in Finnish: children's norms

This article concerns psychometric aspects of the Finnish Early Numeracy Test, which has been translated from the original Dutch Early Numeracy Test. The advantage of the test is that young children's numerical skills can be assessed systematically, which was not possible previously in Finland. A norm study was conducted in which the test was administered to a representative sample of 1,029 Finnish children. The reliability and validity results justify the use of the test as an early-childhood screening test and in research on number sense. The indicative effects of gender, parental professional education, domicile and number of siblings are discussed.     

Measurement Equivalence Between Applicant and Incumbent Groups: An IRT Analysis of Personality Scales

Incumbents are often used in the development and validation of a wide variety of personnel selection instruments, including noncognitive instruments such as personality tests. However, the degree to which assumed motivational factors impact the measurement equivalence and validity of tests developed using incumbents has not been adequately addressed. This study addressed this issue by examining the measurement equivalence of 6 personality scales between a group applying for jobs as sales managers in a large retail organization (N = 999) and a group of sales managers currently employed in that organization (N = 796). A graded item response theory model (Samejima, 1969) was fit to the personality scales in each group. Results indicated that moderately large differences existed in personality scale scores (approximately 1/2 standard deviation units) but only one of the six scales contained any items that evidenced differential item functioning and no scales evidenced differential test functioning. In addition, person-level analyses showed no apparent differences across groups in aberrant responding. The results suggest that personality measures used for selection retain similar psychometric properties to those used in incumbent validation studies.     

Psychometric and instructional validation approaches in the hierarchical sequencing of learning tasks

Two strategies used to test prerequisite relationships among intellectual skills were employed to examine the sequence of three types of subtraction tasks. A psychometric validation strategy yielded results similar to an instructional validation strategy. This finding provides support for the use of the more efficient psychometric approach for validating large numbers of learning hierarchies in relatively short periods of time.     

The masculine role inventory: Components and correlates

An investigation was conducted to develop and validate a self-report inventory, the Masculine Role Inventory (MRI), designed to measure the success preoccupation, restrictive emotionality and inhibited affection aspects of the masculine role. Restrictive emotionality was defined as the restricted expression of privately felt emotions. Inhibited affection refers to the inhibition of feelings of affection and tenderness toward others. Success preoccupation was defined as a persistent preoccupation with success and career development to the exclusion of interpersonal pursuits and devotion. The psychometric properties of the MRI were evaluated with reliability and factor analyses. Self-disclosure correlates were also investigated. The results demonstrated adequate reliability and validity for the MRI.This article is based in part on a doctoral dissertation conducted under the supervision of Toni Falbo. Gratitude is extended to Mark R. Leary, William L. Hays, Ricardo Ainslie, Robert L. Helmreich, and Kathline Waddell for their many helpful comments, and to Raymond C. Hawkins II for his invaluable assistance. Portions of this data were presented at the 1983 annual meeting of the Southwestern Psychological Association, San Antonio, Texas.     

Assessment of the behavioral inhibition system and the behavioral approach system: adaptation and validation of the Sensitivity to Punishment and Sensitivity to Reward Questionnaire (SPSRQ) in a Chilean sample

The goal of the present study is to estimate the psychometric properties of the Sensitivity to Punishment and Sensitivity to Reward Questionnaire (SPSRQ; Torrubia, Avila, Moltó, & Caseras, 2001) in a sample of Chilean college students. The main hypothesis is that the instrument would show appropriate levels of reliability and validity, in light of previous validation studies. A pilot study was conducted in order to generate the adapted version of the questionnaire, which was then applied to a student sample from different undergraduate careers (n = 434). The results show the expected levels of reliability (test-retest and internal consistency). The factorial validity does not comply with the expected model, suggesting a further consideration of the structure of the questionnaire. External validity is appropriate, as the questionnaire shows the expected correlations with other personality measures. It is concluded that the SPSRQ is adequate for the context of validation, and this study contributes to the generalization of the questionnaire, since the results are consistent with the expected psychometric properties that have been reported in the literature.     

Using Invariance to Examine Cheating in Unproctored Ability Tests

Despite their widespread use in personnel selection, there is concern that cheating could undermine the validity of unproctored Internet‐based tests. This study examined the presence of cheating in a speeded ability test used for personnel selection. The same test was administered to applicants in either proctored or unproctored conditions. Item response theory differential functioning analyses were used to evaluate the equivalence of the psychometric properties of test items across proctored and unproctored conditions. A few items displayed different psychometric properties, and the nature of these differences was not uniform. Theta scores were not reflective of widespread cheating among unproctored examinees. Thus, results were not consistent with what would be expected if cheating on unproctored tests was pervasive.     

PROJECT A: SPECIFICATION OF THE PREDICTOR DOMAIN AND DEVELOPMENT OF NEW SELECTION/ CLASSIFICATION TESTS

A major goal of the Army Selection and Classification Project was to develop an experimental predictor battery that would best supplement the Armed Forces Vocational Aptitude Battery for making selection and classification decisions for entry-level enlisted personnel. That is, what predictor measures would best serve the needs of all the jobs in an entire selection/classification system? This paper describes the characteristics of the new test battery and the procedures that were used to develop it. The major steps in the procedure were a structured literature search using a standard protocol, an extensive expert judgment study of expected true validities for a population of predictor variables against a population of performance components, fabrication of modularized software and a special response pedestal for computerized measurement of perceptual and psychomotor abilities, evaluations of experimental measures in three iterative pilot tests and one major field test, and a series of reviews by a panel of scientific advisers. The test battery that resulted from this 2 1/2-year development effort is described. The basic psychometric properties of each measure, as determined in a large concurrent validation sample, are also described.     

Applicant Selection Expectations: Validating a multidimensional measure in the military

Research has suggested the importance of applicants' expectations of forthcoming selection procedures in predicting how applicants react to selection procedures. Validated measures of selection expectations are still scarce, however. This study reports on the validation of the Applicant Expectation Survey (AES), intended to measure applicants' expectations of forthcoming selection procedures. The AES was validated using three military applicant samples and showed sound psychometric properties (i.e., reliability, measurement invariance, discriminant validity) for a five‐factorial oblique structure consisting of 26 items. The five factors (i.e., Warmth/respect, Chance to demonstrate potential, Difficulty of faking, Unbiased assessment, Feedback) were positively related to several organizational outcome measures and to applicants' perceptions of the selection procedure, providing evidence for the predictive validity of the AES.     

Análisis factorial confirmatorio de la Escala de habilidades sociales en universitarios chilenos

This paper presents a psychometric study performing a confirmatory factor analysis of Gismero's Social Skills Scale. A background review is conducted on evaluation tools generally used for social skills, and the general characteristics of the scale are presented for the tool studied. A review is also conducted on psychometric properties of this tool. The sample consisted of 1206 college students at the Universidad Católica de Temuco, Chile. Confirmatory factor analysis was conducted using a polychoric correlation matrix, and the structure proposed by Gismero was analyzed. Reliability assessment procedures, such as internal consistency (Ordinal Alpha) and discriminant validity analysis, were also conducted using AF5 factors. The results show good internal reliability of the tool, suitability of the model to the data, and good discriminant validity. In conclusion, the Gismero's Social Skills Scale is considered as a suitable tool for assessing social skills in Chilean university students. A slight modification of the model structure in the second order factor level is proposed.