Review of psychomotor skills in pilot selection research of the U.S. military services

This review provides an historical perspective of the use of psychomotor, perceptual--cognitive paper-and-pencil, and automated tests for the selection of pilot trainees by the U.S. military services. Automated versions of vintage psychomotor tests (developed in the 1930s and 1940s) seem to be as predictive of military pilot/aviator performance today as in the past. The psychomotor tests receiving the most attention today are the Complex Coordination and Two-Hand Coordination tests originally developed by Mashburn and colleagues [correction of colleges] before World War II (Mashburn, 1934). These tests were significant predictors of Air Force and Navy pass--fail criteria in the past, and automated versions are similarly predictive today. The U.S. Army and Air Force are now using a combination of paper-and-pencil and automated psychomotor--cognitive tests for initial selection (Air Force) or helicopter assignment (Army). It appears that the Navy is considering the use of automated cognitive and psychomotor tests in a selection battery of the future.     

Cross Validation of Experimental USAF Pilot Training Performance Models

In a recent study, several experimental computerized tests demonstrated utility for supporting United States Air Force (USAF) pilot candidate selection de- cisions and adding to the predictive validity of the Air Force Officer Qualifying Test (AFOQT), the ability measure currently included in the USAF pilot candidate selection process (Carretta, 1989). Pilot candidates with good hand-eye coordination, who make quick and accurate decisions, and who exhibited a cautious risk-taking style were more likely to complete training successfully. The purpose of this study was to cross validate these results to determine their stability. To perform the cross validation, 885 USAF Undergraduate Pilot Training (UPT) students were assigned randomly to two groups (478 of these students also were used in Carretta, 1989). Pilot candidate selection models were developed independently for each group. The "best fitting" regression weights from each group then were applied to subjects in the other group to determine the stability of the regression solutions. In the model development phase, subjects in both groups who had good hand-eye coordination and who made quick and accurate decisions were more likely to complete pilot training successfully. Although there was some reduc- tion in the validity coefficients in the cross-validation phase, the selection models were related significantly to final training outcome in both groups. These results suggest that the experimental pilot candidate selection models are robust enough to be used as adjuncts to operational USAF pilot candidate selection procedures.     

Pilot selection in the Norwegian Air Force: a validation and meta-analysis of the test battery

The purpose of this study was to evaluate the test battery currently used for pilot selection to the Norwegian Air Force. Selection is currently based on a standard battery of 20 different psychological tests as well as on medical tests and on an interview by a licensed psychologist. First, two-factor analyses were conducted to examine the relation between the tests in the battery. Then, a correlation study was conducted to evaluate the predictive validity of the tests against two criteria of pilot performance collected during the basic training period. Finally, a small-scale meta-analysis of previous validation studies in Norway was conducted. me best predictors of success in training, based on the meta-analysis, were Instrument Comprehension (mean r = .29), Mechanical Principles (mean r = .23), and Aviation Information (mean r = .22)     

On the Treatment of Outliners in Cognitive and Psychomotor Test Data

Authors of many statistical texts and review articles have pointed to the possi- ble adverse effects that outliers can have on the calculation of sample statistics and have suggested several methods for detecting and treating outliers. We investigated two different methods-data censoring and transformation-for treating outliers in aptitude test data at the item level and total-score level and their effects on the internal consistency and predictive validity of six computer- ized tests being evaluated by the U.S. Air Force. Results from our sample of more than 2,000 pilot training candidates indicated that neither outlier treat- ment method at either level of analysis had significant effects on the tests' internal consistencies or predictive validities. Possible reasons for these findings include the frequency with which outliers occur and the robustness of linear modeling methods.     

Air force officer qualifying test validity for predicting pilot training performance

The AFOQT was validated for the prediction of pilot training criteria. Subjects were 7,563 men and women selected for pilot training on the basis of educational attainment and AFOQT scores. Criterion variables included daily flight training grades, check flight grades in subsonic and transonic aircraft, and overall academic performance in the 53 week pilot training course. Test validities were presented as observed, corrected for multivariate range restriction, and corrected for multivariate range restriction and unreliability. The Aviation Information and Instrument Comprehension tests, measures of job knowledge, were most predictive of daily and check flights in the initial subsonic jet aircraft. This reflects the relative greater importance of prior job knowledge early in training. The Scale Reading test, a measure of perceptual speed, was most predictive for daily and check flights in the advanced transonic training aircraft. The Arithmetic Reasoning test, a good measure of general cognitive ability, was most predictive of aeronautics in ground school. The development of an improved pilot selection composite is suggested by the results of the validity analyses.The views expressed are those of the authors and not necessarily those of the Department of the Air Force, Department of Defense, or the Government of the United States.     

The stability of the factorial pattern of aircrew classification tests in four analyses

This study consists of four factor analyses of the Army Air Forces Aircrew Classification Batteries. The first was an analysis of the 1945 wartime battery, while the other three were analyses of the 1947 postwar battery, consisting of essentially the same variables, but using different samples. Eleven factors were found which had been identified and reported in previous analyses. An additional factor, possibly an artifact, was identified as an age-education doublet. The only factor which differed significantly in the analyses was pilot or flying interest. These factor analyses show that the factorial content of the tests remains quite similar in both wartime and postwar populations.The data reported in this study were collected as part of the United States Air Force Human Resources Research and Development Program and described in Research Bulletin 52-16. The opinions or conclusions contained in this report are those of the authors. They are not to be construed as reflecting the view or indorsement of the Department of the Air Force.     

Group Differences on US Air Force Pilot Selection Tests

Sex and ethnic group differences were examined on the operational composites and tests used to select applicants for US Air Force officer commissioning programmes and for pilot training. Results showed that large mean score differences in applicant samples were substantially reduced among the pilot trainees. Despite differences in test performance, there was no evidence of differential validity for groups. When group differences in predicted pilot training completion rate were observed, performance was overestimated for the minority group relative to the majority group. When regression equations were adjusted for unreliability of the predictors, the observed differences in intercepts were reduced or eliminated. No prediction bias was observed against the minority groups.     

Validation of the Micropat Battery

Micropat is a battery of computer-based aptitude tests designed to assess aspects of psychomotor coordination and information management. Initial research and development work on Micropat started in 1980 with the Army Air Corps and the tests were successfully validated against rotary wing pilot training outcome. In 1985, funding for the project was taken over by the Royal Navy (RN) with a view to improving selection for both RN Pilots and RN Observers. Validation of the main battery was carried out on Pilot and Observer trainees at Britannia Royal Naval College (BRNC), Dartmouth, UK. The paper outlines the development of the Micropat programme over the past decade, focusing on a number of key practical methodological issues. These include discussion of the task-based approach to test construction, the use made of the potential afforded by computer-based assessment and the problem of developing robust composite predictors from small samples. In particular, the relative merits of unit-weighted, rationally weighted and empirically weighted composite predictors are examined. Evidence for the validity of the tests is summarized. It is concluded that we are unlikely to improve substantially on the levels of prediction which were being obtained in the 1950s. Continual development and improvement in selection testing is needed simply to maintain levels of prediction as the demands of flying change. The new forms of test made possible by computer-based assessment technology provide the means of maintaining useful levels of prediction as flying training courses become longer and more complex.     

Performance Measurement in Distributed Environments: Initial Results and Implications for Training

Advances in telecommunications and simulation technologies have created opportunities for conducting distributed team training through networked simulations. In distributed team training, in which simulators may be geographically dispersed, military teams train together in the same battlespace despite the physical separation. In these types of training environments, multiple users are located at multiple sites; consequently, the efficient and effective conduct of training is a challenge. One area that is particularly challenging is the measurement of team performance. Two case studies are reported in which team performance measurement instruments were developed and tested in a distributed training environment. The measurement tools were designed within the context of an instructional approach known as event-based training, which relies on the creation of explicit linkages among learning objectives, exercise events, performance measures, and after-action review or feedback. Active duty, reserve, and National Guard personnel from the Marine Corps, Air Force, and Army participated in several days of training exercises conducted within a network of simulators that were geographically distributed across the United States. The development and use of the measurement instruments are described, data from both case studies are presented, and implications for training are discussed.     

Predictive Validity of the Air Force Officer Qualifying Test for Non-Rated Officer Specialties

The Air Force Officer Qualifying Test (AFOQT) is used to qualify applicants for officer commissioning and aircrew training programs. The current study examined its predictive validity for 14 officer technical training courses for which there are no additional AFOQT minimum qualification requirements beyond those for officer commissioning. Sample sizes ranged from 16 to 2,190 with a mean and median size of 753 and 319 officers. Ninety percent (63 of 70) of the observed correlations between the AFOQT composites and average technical training grades were statistically significant. Meta-analyses were conducted to determine whether the AFOQT validities were generalizable across training specialties. Analyses were done on the observed data, after correction for range restriction, and after correction for both range restriction and criterion unreliability. The pattern of validities was similar for all three meta-analyses. The lower bound of the 95% confidence interval and the 95% credibility interval around the weighted mean validities were greater than zero for all five AFOQT composites supporting its utility for making personnel selection decisions for these jobs. The Verbal composite had the lowest and the Academic Aptitude composite had the highest weighted mean validity. The weighted mean of the validity coefficients across training specialties ranged from .261 to .326 for the observed data, .322 to .387 for the range-restriction corrected data, and .360 to .433 for the fully corrected data. Additional efforts are required to examine the generalizability of the results for a broader range of occupational specialties and to set minimum qualifying scores.     

An Examination of Five Indexes of Test Battery Performance: Analysis of the ECAT Battery

This investigation evaluated potential revisions to the Armed Services Vocational Aptitude Battery (ASVAB). The data analyzed were collected from trainees in 17 U.S. Air Force, Army, and Navy jobs as part of the Joint-Services Enhanced Computer-Administered Test (ECAT) battery validation study. Predictors included the trainees’ preenlistment scores for the 10 tests in the current ASVAB, plus the 9 experimental ECAT battery tests. The criteria were measures of training performance. All possible combinations of tests that (a) included the Word Knowledge and Arithmetic Reasoning tests of the ASVAB and (b) could be administered in a 134- to 164-min interval were evaluated with respect to 5 indexes of test battery performance: criterion-related validity, classification efficiency, and 3 types of subgroup differences (White vs. Black, White vs. Hispanic, and male vs. female). The 5 indexes were calculated for each of the 16,437 possible combinations of tests. The standard deviations of the indexes across the combinations of tests showed that (a) values on the validity index varied little, (b) values on the classification efficiency and White versus Black and White versus Hispanic subgroup differences indexes varied moderately, and (c) values on the male versus female difference index varied substantially. The validity index of the combinations showed a moderate correlation with the classification efficiency index and a nearly zero correlation with subgroup differences. However, the classification efficiency index showed a small-to-moderate positive correlation with the subgroup difference indexes. The subgroup difference indexes showed moderate-to-high positive correlations with one another. Examinations of the top 20 combinations of tests identified by each index demonstrated that tests that optimize one type of index usually do not optimize each of the other indexes. In particular, trade-offs were observed between (a) the maximization of validity (and classification efficiency) versus the minimization of all 3 types of subgroup differences and (b) the minimization of differences between Whites and Blacks (or between Whites and Hispanics) versus the minimization of differences between men and women. These results suggest that no combination of the tests considered in this investigation simultaneously optimizes all 5 test battery performance indexes.     

Computerized Testing in the United States Air Force

Computerized testing in the United States Air Force finds its historical beginnings in apparatus tests developed during World War II. The advent of fast, small, and inexpensive computers led to the establishment of a research program to use computers in pilot selection. Later, computerized tests were demonstrated for more familiar aptitude measures such as verbal and quantitative ability. Several large-scale efforts to implement computerized testing have proved informative. Many Air Force tests are routinely administered by computer. Continued advances in computer technology promise to make computerized testing more common and more commonplace. Computer use in measuring physiological correlates of human ability is discussed.     

Multivariate information transmission

A multivariate analysis based on transmitted information is presented. It is shown that sample transmitted information provides a simple method for measuring and testing association in multi-dimensional contingency tables. Relations with analysis of variance are pointed out, and statistical tests are described.This work was supported in part by the Air Force Human Factors Operations Research Laboratories, and in part jointly by the Army, Navy, and Air Force under contract with the Massachusetts Institute of Technology.Several of the indices and tests discussed in this paper have been developed independently by J. E. Keith Smith (11) at the University of Michigan, and by W. R. Garner at Johns Hopkins University.     

Personality characteristics of fighter pilots and ground personnel

The training of fighter plane pilots is expensive, yet instruments are rarely used to screen potential pilots prior to flight training. Hence, a measure that can differentiate fighter pilots from other military personnel is needed. This study was an investigation of personality traits that are contributing factors for becoming a pilot in the Air Force, which could serve as screening tools. One hundred and twelve males were recruited from the Air Force university in Taiwan. Follow-up was conducted 2 years later with 73 students who became pilots and 39 who became ground personnel. A further 53 male military personnel of the same age served as controls. Structural equation modeling was used to show that pilots were more extraverted and less neurotic than the ground personnel, and more extraverted than the controls. Paternal overprotection had an indirect association with becoming a pilot through the mediation of the personality traits of extraversion and neuroticism. Mental health was not associated with becoming a pilot. The optimal cut-off point of 4/5 on a scale of extraversion resulted in a high sensitivity (96%) for differentiating between fighter pilots and controls. Independent of psychosocial stressors (mental health), extraversion is associated with the biological mechanisms of an individual, and plays a unique role in the process of becoming a pilot. Therefore, an extraversion index can be used for screening potential military pilots prior to flight training, as a means of reducing costs and managing human resources.     

A Study of Active Duty Military Personnel With the MMPI-2

The Minnesota Multiphasic Personality Inventory (MMPI) has a long history of application in military settings. Our investigation is an exploration of the relevance and appropriateness of the revised version of this instrument, the MMPI-2, in military settings. The question of whether special norms are needed for use in the military was addressed. Results showed that military personnel (N = 1,156 men from the Army, Navy, Air Force, and Marine Corps) responded, as a group, in a manner similar to the MMPI-2 restandardized sample. Similar mean scores and factor structures were obtained. Age effects were also studied. Consistent with past research, there were clear age differences on the MMPI-2 with younger subjects producing more scale elevation, particularly on the Psychopathic Deviate (Pd), Psychastenia (Pt), Schizophrenia (Sc), and Hypomania (Ma) scales. The results of the study show that special norms for military settings are not needed.     

A statistical critique of the USAFI tests of general educational development

Data resulting from the administration of the USAFI Tests of General Educational Development to more than 1000 junior Air Force officers have been statistically analyzed to indicate the reliability of these tests, their correlation with school achievement, the comparability of their intercorrelations with intercorrelations among grades in school subjects, their capacity for differential diagnosis, their factorial composition, and their average item-test correlations. In the light of these findings and the finding that there is a low gradient between achievement on these tests and amount of formal education among these officers, the tests have been evaluated as possessing the practical validity suggested by their face validity for selection of young Air Force officers for assignment to study at civilian colleges and universities.Paper read before the National Council on Measurements Used in Education, Atlantic City, New Jersey, February 24, 1948. The writers are indebted to Dr. James E. Greene, formerly Deputy for Research of the Educational Advisory Staff, The Air University, for aid in planning and sponsoring this study.     

Lack of Ability Is Not Always the Problem

Managers request validation studies and stringent standards to stem training attrition. These studies while legally mandated and important are not necessarily the solution. The current study was motivated by such a request. The validity of four composites from an aptitude battery for predicting training success for weapons directors was evaluated. Participants were 353 Air Force personnel who completed training and duty in a previous job. All four composites, Mechanical, Administrative, General, and Electronics were valid. General was most valid. We observed only three academic failures among 32 failed participants. The General composite scores for the successful and non-academic failures were very similar. It was speculated that low motivation and job design features were contributing factors for most eliminees.     

Suitability Screening During Basic Military Training

Results are presented for suitability screening conducted by the United States Air Force during basic military training. The Lackland Behavioral Questionnaire has been used operationally since 2007, and its use has led to an increase in predictive validity. For the prediction of outcomes during the first 4 years of service, one can identify trainees with a 49% likelihood of being separated for unsuitability and a 56% likelihood of being diagnosed with a mental disorder. A formal suitability screening program allows referrals and recommendations to be made using a combination of empirical evidence and clinical judgment.     

A revised law of comparative judgment

In contrast to Thurstone's Law of Comparative Judgment, a model in which a comparison pair and its complement are assumed to give rise to two different distributions of differences is considered. The consequences of this revised model on scaling problems is developed.The research in this article was supported jointly by the Army, Navy, and Air Force under contract with the Massachusetts Institute of Technology.