Validities of Unit-Weighted Composites of the ASVAB and the ECAT Battery

Currently, assignment to training in each entry-level enlisted job in the U.S. military is influenced by examinees’ scores on the classification composite for that job from the Armed Services Vocational Aptitude Battery (ASVAB). For a cross-service sample of the 18 occupations in the Enhanced Computer-Administered Test (ECAT) project, enhancements of their respective operational ASVAB composites (9 unique ones) are evaluated here. Analyses are reported on the gains in validity from adding 1 test with unit weight to each composite, the addition coming either from the other ASVAB tests or from the 9 tests of the ECAT battery. In more than half the schools, the best ASVAB additions were Coding Speed or Mechanical Comprehension. In the ECAT battery, 1 test of psychomotor ability (Two-Hand Tracking) and 2 tests of working memory (Mental Counters and Sequential Memory) were the most promising additions. The utility of the new composites for person–job matching is discussed.     

Recruiting Effectiveness of the ASVAB Career Exploration Program

To assess the recruiting effectiveness of the Department of Defense's Armed Services Vocational Aptitude Battery (ASVAB) Career Exploration Program, a variety of sources, including historical databases, surveys, and interviews, were used. Despite downsizing and reductions in recruiting resources over the course of the evaluation period, there is evidence that a higher percentage of program participants subsequently enlist than the U.S. youth population as a whole. The overwhelming majority of a sample of recruiters rated the program as an important lead source as well as an effective recruiting tool. Data indicate that 16% to 25% of annual enlistments started as ASVAB program leads. The program influences young people's consideration of the military as a career and creates a positive impression of the military, regardless of whether a participant decides to enlist. Although there is evidence that the ASVAB program enhances recruiting, obstacles to its effectiveness include recruiter training deficiencies.     

Effects of the testing situation on item responding: cause for concern

The effects of faking on personality test scores have been studied previously by comparing (a) experimental groups instructed to fake or answer honestly, (b) subgroups created from a single sample of applicants or nonapplicants by using impression management scores, and (c) job applicants and nonapplicants. In this investigation, the latter 2 methods were used to study the effects of faking on the functioning of the items and scales of the Sixteen Personality Factor Questionnaire. A variety of item response theory methods were used to detect differential item/test functioning, interpreted as evidence of faking. The presence of differential item/test functioning across testing situations suggests that faking adversely affects the construct validity of personality scales and that it is problematic to study faking by comparing groups defined by impression management scores.     

Measuring Cognitive Aptitude Using Unobtrusive Knowledge Tests: A New Survey Technology

Five knowledge tests and one implicit-reasoning task were developed to be: (1) exceptionally short, (2) correlated with general cognitive aptitude, (3) unobtrusive, i.e., appear similar to attitudinal survey items as opposed to maximal performance measures, and (4) without formally “correct” answers. The intent was to design scales that could be administered in non-proctored environments to directly measure general cognitive aptitude while avoiding the possibility that participants could use references to provide “good” answers. The five knowledge tests used a Likert format to assess knowledge in verbal and practical domains, and were scored by computing distances between examinee and reference ratings. The implicit-reasoning task appeared to be a series completion “game” that required a dichotomous response. The scales were administered to 288 Air Force recruits and were validated against the Armed Services Vocational Aptitude Battery (ASVAB). Individual unobtrusive knowledge scales and ASVAB tests were substantially correlated with sample correlations ranging to .39 and population correlation estimates to .66 after correcting for range restriction. Two sets of factor scores, which were separately derived from the unobtrusive test battery and the ASVAB, were highly correlated in our sample, .54, yielding a population correlation of .80 after correcting for range restriction. This technology is important because few paper- or Internet-based surveys, and virtually no mail-based surveys accurately measure general cognitive aptitude, while many of these surveys address important social issues and commercial questions that could be better understood given an unobtrusive but accurate estimate of general cognitive aptitude.     

Individual and group differences in intelligence and speed of information processing

A battery of eight different reaction time (RT) tests, measuring the speed with which individuals perform various elementary cognitive processes, and a group test of scholastic aptitude (the Armed Services Vocational Aptitude Battery, ASVAB) were given to 50 black and 56 white male vocational college students. The regression of the general factor scores of the ASVAB on the RT measures yielded a shrunken multiple correlation of 0.465. Although discriminant analyses, when applied separately to the ASVAB subtests and to the RT variables, showed highly comparable overall discrimination (over 70% correct classification) between the black and white groups, factor scores derived from the general factor (labeled ‘speed of information processing’) of the RT battery show only about one-third as large a mean black-white difference as the mean group difference on the general factor scores derived from the ASVAB. Comparisons were also made between the 106 vocational college students and 100 university students of higher average academic aptitude who had previously been tested on the same RT battery (Vernon, 1983a). These groups showed marked differences on the RT variables, the largest differences occuring on the tests that required more complex cognitive processing. The more complex RT tests also correlate most highly with the psychometric measures of ability within each group. The results are consistent with the hypothesis that individual differences and the mean differences between groups in psychometric abilities and scholastic achievement are related to differences in the speed of information processing as measured in elementary cognitive tasks.     

Alternative Techniques for Predicting Success in Air Controller School

The Armed Services Vocational Aptitude Battery (ASVAB) is the principal cognitive test battery used for military classification-that is, for assignment of individuals to specific job categories. Successful entrance into a specific job category requires satisfactory completion of relevant training. Therefore, it is necessary that the ASVAB accurately predict the likelihood that individuals will complete training. Presently, the U.S. Navy classifies enlisted personnel based on a unit-weighted linear combination of scores derived from a subset of tests currently composing the ASVAB test battery. This research explored the development of new classification models estimated by using linear regression, logistic regression, discriminant analysis, and artificial neural networks, and it compared their classification accuracy with models developed using unit- weighted regression. Models were estimated and cross-validated using data from individuals admitted into the Navy's Air Controlman training during fiscal years 1988 and 1989. Accuracy was measured by the proportion of individuals correctly classified as graduates or as academic failures and by the ability to rank students-that is, to predict the highest scores for graduates and the lowest scores for attrites.     

Development of a Cyber/Information Technology Knowledge Test for Military Enlisted Technical Training Qualification

An Armed Services Vocational Aptitude Battery (ASVAB) Review Panel, with expertise in personnel selection, job classification, psychometrics, and cognitive psychology developed recommendations for changes to the military enlistment test battery. One recommendation was to develop and evaluate a test of cyber/information and communications technology literacy to supplement current ASVAB content. This article summarizes a multiphased Cyber Test development process: (a) a review of information/computer technology literacy definitions and measures, (b) development and pilot testing of a cyber knowledge measure, (c) validation of test scores against final school grades (FSGs) for selected technical training courses, (d) development of an operational reporting metric and subgroup norms, and (e) examination of construct validity. Results indicate the Cyber Test has predictive validity versus technical training school grades and incremental validity comparable to the ASVAB technical knowledge tests when used with the ASVAB Armed Forces Qualification Test (AFQT) verbal/math composite as a baseline.     

Using Invariance to Examine Cheating in Unproctored Ability Tests

Despite their widespread use in personnel selection, there is concern that cheating could undermine the validity of unproctored Internet‐based tests. This study examined the presence of cheating in a speeded ability test used for personnel selection. The same test was administered to applicants in either proctored or unproctored conditions. Item response theory differential functioning analyses were used to evaluate the equivalence of the psychometric properties of test items across proctored and unproctored conditions. A few items displayed different psychometric properties, and the nature of these differences was not uniform. Theta scores were not reflective of widespread cheating among unproctored examinees. Thus, results were not consistent with what would be expected if cheating on unproctored tests was pervasive.     

The effect of variations in answer-sheet format on aptitude test performance

Three different answer sheets were used in the administration of the General Aptitude Test Battery to groups of young adults (n = 302). The answer sheets, designed for use with different brands of Optical Mark Reader, varied in the shape of the response space. A two-factor, repeated measures analysis of variance revealed a significant interaction between subtests and type of answer sheet. Further analyses showed that performance on ‘speed’ tests was affected by the format of answer sheet used, whereas performance on mainly ‘power’ tests was not. The implications of the results for testing practice are discussed.     

The Magnitude and Extent of Cheating and Response Distortion Effects on Unproctored Internet‐Based Tests of Cognitive Ability and Personality

The use of unproctored internet‐based testing (UIT) for employee selection is quite widespread. Although this mode of testing has advantages over onsite testing, researchers and practitioners continue to be concerned about potential malfeasance (e.g., cheating and response distortion) under high‐stakes conditions. Therefore, the primary objective of the present study was to investigate the magnitude and extent of high‐ and low‐stakes retest effects on the scores of a UIT speeded cognitive ability test and two UIT personality measures. These data permitted inferences about the magnitude and extent of malfeasant responding. The study objectives were accomplished by implementing two within‐subjects design studies ( Study 1 N=296; Study 2 N=318) in which test takers first completed the tests as job applicants (high‐stakes) or incumbents (low‐stakes) then as research participants (low‐stakes). For the speeded cognitive ability measure, the pattern of test score differences was more consonant with a psychometric practice effect than a malfeasance explanation. This result is likely due to the speeded nature of the test. And for the UIT personality measures, the pattern of higher high‐stakes scores compared with lower low‐stakes scores is similar to those reported for proctored tests in the extant literature. Thus, our results indicate that the use of a UIT administration does not uniquely threaten personality measures in terms of elevated scores under high‐stakes testing that are higher than those observed for proctored tests in the extant literature.     

Contextualized personality measures in employee selection: Extending frame‐of‐reference research with job applicant samples

This study compared contextualized and non‐contextualized personality measures in operational hiring situations, unlike previous research which has largely relied on student or job incumbent samples. Comparisons were made with regard to (A) the frames‐of‐reference adopted by applicants when responding to the measures, (B) relations with subsequent employment interview scores, (C) applicant reactions, and (D) mean scores for the personality scales. The findings highlight potential concerns with using non‐contextualized personality measures for employee selection, as job applicants will likely adopt various frames‐of‐reference unrelated to the intended focus of measurement (i.e., work). Results also indicated that it may be premature to assume contextualized measures elicit improved applicant reactions despite their apparent job‐relatedness. The discussion concludes with suggested directions for future research.     

Measuring gratitude in youth: assessing the psychometric properties of adult gratitude scales in children and adolescents

Before the developmental trajectory, outcomes, and related interventions of gratitude can be accurately and confidently studied among the youth, researchers must ensure that they have psychometrically sound measures of gratitude that are suitable for this population. Thus, considering that no known scales were specifically designed to measure gratitude in youth, this study aimed to answer an important question: Are the existing gratitude scales used with adults valid for use with youth? The present study is an empirical investigation, based on a large youth sample (N = 1,405) with ages ranging from 10 to 19 years old, of the psychometric properties of scores of the Gratitude Questionnaire-6 (GQ-6; M. E. McCullough, R. A. Emmons, & J.-A. Tsang, 2002), the Gratitude Adjective Checklist (GAC; M. E. McCullough, R. A. Emmons, & J.-A. Tsang, 2002), and the Gratitude Resentment and Appreciation Test (GRAT)-short form (M. Thomas & P. Watkins, 2003). Single-group and multiple-group confirmatory factor analyses indicated that the factor structures of these gratitude scales resemble those found with adults and were invariant across age groups. Scores of all three gratitude scales revealed acceptable internal consistency estimates (i.e., >.70) across age groups. Results showed that whereas scores of all three gratitude scales were positively correlated with each other for 14- to 19-year-olds, GRAT-short form scores tended to display relatively low correlations with scores of the other two measures for younger children (10-13 years old). Furthermore, the nomological network analysis showed that scores of all three gratitude scales were positively correlated with positive affect and life satisfaction scores across the age groups. The relationships with negative affect and depression scores, however, seemed dependent on the child's age. Pending results from subsequent research recommendations for researchers interested in studying gratitude in youth are offered.     

Application patterns when applicants know the odds: implications for selection research and practice

Unlike previous research that found small differences between population standard deviations and applicant pool standard deviations (P. R. Sackett & D. J. Ostgaard, 1994; D. S. Ones & C. Viswesvaran, 2003), this study revealed a 23% disparity between Law School Admission Test (LSAT) scores of all LSAT test takers and those of LSAT test takers who applied to law school. This study also illustrated robust applicant self-selection behavior across different law school ranks. These findings are important, because predictor scores of applicants who know their scores in advance and perceive small selection ratios necessitate substantially smaller range restriction corrections than those that would be required by population standard deviations. Furthermore, these findings more generally reveal that applicants who know their scores in advance behave quite differently from applicants who do not.     

Interrelatedness With Others as a Measure of Psychological Stability

Psychological disturbance is presumed to express itself in a relatively small and unique patter of responses; a disturbed individual is somewhat unlike most other individuals, both normal and disturbed. A pertinent assessment approach is to select a representative scoring sample. Using the answer sheets of these subjects as keys, all other subjects are scored on them. A subject's total score is the sume of his scores on the answer sheets of the scoring sample.     

The Use of Mobile Devices in High‐stakes Remotely Delivered Assessments and Testing

With Internet access no longer restricted to desktop and laptop computers, job applicants now have the opportunity to complete remotely delivered assessments on mobile, handheld small screen devices such as smartphones, and personal digital assistants. In this study, a large dataset is used to investigate demographic and score differences between job applicants who completed a remotely delivered high‐stakes assessment on a mobile device and those who completed it on a nonmobile device. Based on a sample of 3,575,207 job applicants who completed an unproctored Internet‐based assessment between January 2011 and April 2012, the percentage of applicants completing the assessment on a mobile device was small, 1.93%, but nevertheless represented more than 69,000 people. Overall, there were small test‐taker demographic differences in the use of mobile devices versus nonmobile devices in that mobile devices were slightly more likely to be used by women, African‐Americans and Hispanics, and younger applicants. Scores on a personality measure were similar for mobile and nonmobile devices but scores on a general mental ability test were substantially lower for mobile devices. Tests of measurement invariance also indicated equivalence across the mobile and nonmobile samples. Test taker and organizational implications for completing remotely delivered high‐stakes noncognitive and cognitive assessments on mobile versus nonmobile devices are discussed.     

The Validity of Verifiable and Non‐verifiable Biodata Items: An Examination Across Applicants and Incumbents

This study examines the influence of item verifiability (non‐verifiable vs. verifiable), context (applicant vs. incumbent), and keying procedure on biodata mean test scores and validity. Concurrent and predictive validation studies were conducted using a sample of 425 call center incumbents and a sample of 410 call center applicants. Although applicants did not obtain significantly higher mean biodata scores, results provide support for the hypothesis that the non‐verifiable biodata composite would be less valid in the applicant context, while the verifiable biodata composite would be equally valid across both the applicant and incumbent contexts. The same pattern of results was obtained using both item‐ and option‐keying procedures. Implications for research and practice are discussed.     

MOTIVATIONAL COMPONENTS OF TEST TAKING

The first phase of this research effort describes an effort to directly measure the attitudes and opinions of employment test takers toward the tests they just took; the instrument is called the Test Attitude Survey (TAS). Nine factors were developed which reflect test takers' expressed effort and motivation on the test, the degree of concentration, perceived test ease, and the like. Several studies were conducted showing that TAS factors were significantly sensitive to differences in test types and administration permitting the inference that the TAS possessed construct validity. The second phase of this study tested several propositions and hypotheses. In one study, it is shown that the applicants report significantly higher effort and motivation on the employment tests compared to incumbents, even when ability is held constant. A second study showed that a small but significant relationship exists between TAS factor scores, test performances, and the person factors. Moreover, some of the racial differences on test performances can be accounted for via the TAS factor scores; it is observed that after holding these TAS factors constant, racial differences on the employment tests scores diminished. In a third study, very limited evidence was found for the incremental and moderating effects of these attitudes, but there were several limitations to the study associated with small sample sizes, unknown reliabilities in the criterion scales, and so forth. Discussion focussed on the potential practical applications of the TAS instrument and factor scores. It is suggested that further research could have some utility in this domain.     

A Research Note on the Incremental Validity of Job Knowledge and Integrity Tests for Predicting Maximal Performance

In a sample of industrial job applicants, relationships among scores on an integrity test, a job knowledge measure, and maximal performance as assessed by a work sample measure were investigated. The observed correlation between the personality-based integrity test and maximal performance was .27, indicating that integrity tests can be predictive of maximal performance. Furthermore, integrity test scores correlated .14 with job knowledge and job knowledge scores had a validity of .36 for maximal performance. Theoretical implications for the maximal/typical performance distinction are discussed.     

A Third Moment Adjusted Test Statistic for Small Sample Factor Analysis

Goodness-of-fit testing in factor analysis is based on the assumption that the test statistic is asymptotically chi-square, but this property may not hold in small samples even when the factors and errors are normally distributed in the population. Robust methods such as Browne's (1984) asymptotically distribution-free method and Satorra Bentler's (1988, 1994) mean scaling statistic were developed under the presumption of nonnormality in the factors and errors. This article finds new application to the case where factors and errors are normally distributed in the population but the skewness of the obtained test statistic is still high due to sampling error in the observed indicators. An extension of Satorra Bentler's statistic is proposed that not only scales the mean but also adjusts the degrees of freedom based on the skewness of the obtained test statistic in order to improve its robustness under small samples. A simple simulation study shows that this third moment adjusted statistic asymptotically performs on par with previously proposed methods and at a very small sample size offers superior Type I error rates under a properly specified model. Data from Mardia, Kent, and Bibby's (1980) study of students tested for their ability in 5 content areas that were either open or closed book were used to illustrate the real-world performance of this statistic.