Large sample validations of three general predictors of pilot training success

This article addresses one of the perennial problems of pilot selection research: obtaining an adequate sample size for reliable statistical analysis of predictive validity. Results from three studies involving the same computerized tests of instrument comprehension and psychomotor ability were combined in a meta-analysis to determine whether the validities of these tests generalized across three contexts. These were Royal Air Force and Turkish Air Force fixed-wing pilot training and British Army Air Corps rotary-wing pilot training. In this article, we discuss the adequacy of samples for estimating the validity of the tests, and the persistence of predictive validity to later stages of training as shown by British Army Air Corps data. Reference is also made to data from a fourth independent study of Qantas pilot training.     

Employment test usage as a predictor of gross domestic product

The relationship between the usage of employment tests—particularly ability tests-and job performance criteria has been extensively studied. The usage of testing was reported to have declined after the landmark personnel selection caseGriggs (1971). Models of selection utility were reviewed, and an ecological systems theory of test incidence and the American economy was developed. For both ability tests and non-personality employment tests, a positive relationship was hypothesized between the historical incidence of firms using tests and American Gross Domestic Product. Using historical survey and economic data, results were consistent with the hypothesis involving ability testing incidence.The views expressed are those of the author and not necessarily those of the U.S. Air Force, the Department of Defense, or the U.S. government.     

Cross Validation of Experimental USAF Pilot Training Performance Models

In a recent study, several experimental computerized tests demonstrated utility for supporting United States Air Force (USAF) pilot candidate selection de- cisions and adding to the predictive validity of the Air Force Officer Qualifying Test (AFOQT), the ability measure currently included in the USAF pilot candidate selection process (Carretta, 1989). Pilot candidates with good hand-eye coordination, who make quick and accurate decisions, and who exhibited a cautious risk-taking style were more likely to complete training successfully. The purpose of this study was to cross validate these results to determine their stability. To perform the cross validation, 885 USAF Undergraduate Pilot Training (UPT) students were assigned randomly to two groups (478 of these students also were used in Carretta, 1989). Pilot candidate selection models were developed independently for each group. The "best fitting" regression weights from each group then were applied to subjects in the other group to determine the stability of the regression solutions. In the model development phase, subjects in both groups who had good hand-eye coordination and who made quick and accurate decisions were more likely to complete pilot training successfully. Although there was some reduc- tion in the validity coefficients in the cross-validation phase, the selection models were related significantly to final training outcome in both groups. These results suggest that the experimental pilot candidate selection models are robust enough to be used as adjuncts to operational USAF pilot candidate selection procedures.     

Review of psychomotor skills in pilot selection research of the U.S. military services

This review provides an historical perspective of the use of psychomotor, perceptual--cognitive paper-and-pencil, and automated tests for the selection of pilot trainees by the U.S. military services. Automated versions of vintage psychomotor tests (developed in the 1930s and 1940s) seem to be as predictive of military pilot/aviator performance today as in the past. The psychomotor tests receiving the most attention today are the Complex Coordination and Two-Hand Coordination tests originally developed by Mashburn and colleagues [correction of colleges] before World War II (Mashburn, 1934). These tests were significant predictors of Air Force and Navy pass--fail criteria in the past, and automated versions are similarly predictive today. The U.S. Army and Air Force are now using a combination of paper-and-pencil and automated psychomotor--cognitive tests for initial selection (Air Force) or helicopter assignment (Army). It appears that the Navy is considering the use of automated cognitive and psychomotor tests in a selection battery of the future.     

A method for clustering eigenvalues

A method of computing communalities is presented which attempts to cluster eigenvalues about zero. Results are exhibited which show reduction of rank using computed communalities.This paper is based on work performed under Contract AF 30(602)-3300 monitored for the U. S. Air Force by Mr. Rocco Iuorno, Rome Air Development Center. The use, under the above contract, of the IBM 7094 computer located at Aeronautical Systems Division, Wright-Patterson Air Force Base, is gratefully acknowledged. Permission is granted for reproduction, translation, publication, use, and disposal in whole and in part by or for the United States Government.     

Multivariate information transmission

A multivariate analysis based on transmitted information is presented. It is shown that sample transmitted information provides a simple method for measuring and testing association in multi-dimensional contingency tables. Relations with analysis of variance are pointed out, and statistical tests are described.This work was supported in part by the Air Force Human Factors Operations Research Laboratories, and in part jointly by the Army, Navy, and Air Force under contract with the Massachusetts Institute of Technology.Several of the indices and tests discussed in this paper have been developed independently by J. E. Keith Smith (11) at the University of Michigan, and by W. R. Garner at Johns Hopkins University.     

Better Candidates Fly Fewer Training Hours: Another Time Testing Pays Off

The relationship between Pilot Candidate Selection Method (PCSM) scores and the number of flying training hours required to complete United States Air Force Undergraduate Pilot Training (UPT) was investigated on a sample of 1,082 graduates. This was done to demonstrate that ability testing has real world correlates and that cost savings accrued as a result of ability testing. The criterion ‘extra flying hours’ was computed by subtracting each student's cumulative flying hours from the sample mean. The correlations (corrected for range restriction) between PCSM scores and primary and advanced flying training extra hours were ?0.206 and ?0.270, respectively. Demonstrating that UPT graduates with higher PCSM scores required fewer flying hours to complete training facilitates the estimation of the cost avoidance achieved by ability tests.     

Ethical considerations in the use of computers in psychological testing and assessment

Ethical considerations in the use of computers in psychological testing and assessment are discussed. Particular attention is given to existing ethics and standards that provide guidance to users of computerized test interpretation and report-writing programs. Issues covering professional accountability and confidentiality are also discussed as they relate to computer-assisted assessment. Areas of appropriate use of computers in testing and assessment are suggested, including test scoring, the analysis of scores, and generating remedial recommendations. Findings are summarized and several guidelines are suggested for users of computers in testing and assessment.     

The computer as psychometrician

The application of computers to the functions traditionally carried out by a psychometrician or psychologist is examined. Diagnosis, psychotherapy, interviewing, psychological testing, test interpretation, and report writing are reviewed. The advantages and disadvantages of using computers in assessment applications are discussed. Patient and professional acceptance of automated psychological services is reviewed, as well as the parameters to consider in the development of computerized interviewing and testing. Brief comments are made regarding suitable computer systems for these applications. It is argued that computers can provide a cost-effective enhancement to the delivery of mental health services.     

Computerized Testing in Neurobehavioural Toxicology

Cet article décrit quelques-uns des avantages que peut procurer l'informatisation du testing par rapport à l'administration manuelle traditionnelle. On dresse une liste des principales batteries de tests informatisées utilisées dans le champ de la toxicologie comportementale. Les conséquences éventuelles des progrès réalisés dans le software et le hardware sont traitées. On parle aussi de l'absence relative de guides théoriques pour l'élaboration de tests en toxicologie comportementale et l'on présente enfin des recommandations pour le développement à venir et I'utilisation de batteries de tests neurocomportementaux informatisées.
This paper discusses some of the advantages that may be gained from using computerized performance testing, as compared to traditional manual testing. A list of the major computerized test batteries used in the field of behavioural toxicology is presented, and possible implications of advances in computer hardware and software are discussed. The implications of the relative lack of theoretical guidelines for the development of tests within behavioural toxicology are discussed, and recommendations for the future development and use of computerized neurobehavioural test batteries are also presented.     

适应性测验的发展:历史与现状

适应性测验作为一种测验形式,与传统的纸笔测验形式相比,具有省时、高效等很多优越性。测验应该适合于被试的理念最初可见于比内的智力测验。上世纪七十年代以来,适应性测验的研究从经典测量理论阶段发展到项目反应理论阶段,经历了从二阶段、三阶段、多阶段测验、固定分支测验和分层适应性测验的发展,到现在的计算机化适应性测验研究。随着项目反应理论和计算机技术的发展,计算机化适应性测验已经在教育和心理测验领域中得到广泛应用。目前对它的研究日益深入,主要有项目克隆、项目曝光、多维适应性测验、被试诊断、人格适应性测验等问题。     

A statistical critique of the USAFI tests of general educational development

Data resulting from the administration of the USAFI Tests of General Educational Development to more than 1000 junior Air Force officers have been statistically analyzed to indicate the reliability of these tests, their correlation with school achievement, the comparability of their intercorrelations with intercorrelations among grades in school subjects, their capacity for differential diagnosis, their factorial composition, and their average item-test correlations. In the light of these findings and the finding that there is a low gradient between achievement on these tests and amount of formal education among these officers, the tests have been evaluated as possessing the practical validity suggested by their face validity for selection of young Air Force officers for assignment to study at civilian colleges and universities.Paper read before the National Council on Measurements Used in Education, Atlantic City, New Jersey, February 24, 1948. The writers are indebted to Dr. James E. Greene, formerly Deputy for Research of the Educational Advisory Staff, The Air University, for aid in planning and sponsoring this study.     

Percentage points of Wilks'L mvc andL vc criteria

Likelihood ratio tests have been proposed by Wilks for testing the hypothesis of equal means, variances, and covariances (H _mvc) and the hypothesis of equal variances and covariances (H _vc) in ap-variate normal distribution. Using exact distributions of the appropriate likelihood ratio statistics, tables of the .05 and .01 points of these distributions are constructed forp = 4, 5, 6, 7 and sample sizen = 25 (5) 60 (10) 100. A correction factor is recommended for largern. Two numerical examples illustrate use of the tables. A nonparametric test is proposed forH _mvc when the multivariate parent population is known to be non-normal.This research was supported partly by the Office of Naval Research under Contract No. Nonr-855(06) and partly by the United States Air Force through the Air Force Office of Scientific Research of the Air Research and Development Command, under Contract No. 18(600)-83. Reproduction in whole or in part for any purpose of the United States Government is permitted.     

Taskomat: Evaluation of a Computerized Test Battery

A case study on a computerized test battery is presented based on more than a decade of use in military and civil settings. The study sheds light on the general issues associated with computerized testing. The accent is on the contrast between computerized and traditional paper-and-pencil testing. The computerized battery, called Taskomat, is founded on concepts arising from human performance theory. The tests of the battery are described, as well as issues related to the hardware (cost of purchasing, hardware reliability and industrial styling). Criteria used to evaluate the content of the tests were: reliability, robustness of effects, uniqueness or newness of abilities relative to paper-and-pencil testing, and validity of these abilities for predicting occupational success. A conclusion is that a theoretical foundation helps to increase the likelihood of assessing new abilities.     

THE PEN-BASED COMPUTER AS AN ALTERNATIVE PLATFORM FOR TEST ADMINISTRATION

Pen-based computers are similar to paper and pencil (P&P) tests in the method of responding, and thus, may more closely match paper and pencil administration in construct equivalence than keyboard-entry computers. A study was conducted comparing P&P, pen-based note-book computer, and keyboard-entry PC versions of two test batteries. Participants completed tests administered using different administration modes on separate days; construct equivalence was evaluated by comparing Day 1-Day 2 correlations across conditions. Although construct equivalence was found for the power tests, differences emerged for the speeded tests. For the pen-based computer, solid evidence of equivalence to P&P appeared for all but one of the speeded tests, whereas the keyboard PC showed borderline equivalence for only one of the three speeded tests. These findings suggested that the pen-entry computer may be more capable than the keyboard-entry computer in maintaining construct equivalence to P&P tests.     

Application of the quartimax method of rotation to Thurstone's primary mental abilities study

This study compares a quartimax rotation of the centroid factor loadings for Thurstone's Primary Mental Abilities Test Battery with factorings of the same correlation matrix by Thurstone (simple structure), Zimmerman (revised simple structure), Holzinger and Harman (bi-factor analysis), and Eysenck (group factor analysis). The quartimax results agree very closely with the solutions of Holzinger and Harman and of Eysenck, and reasonably well with the two simple structure analyses. The principal difference is the general factor provided by the quartimax solution. Reproduction of the factorial structure is sufficiently good to justify its use at least as the first stage of rotation. More extensive trial of the method will be needed with more varied data before it will be possible to decide whether quartimax factors meet psychological requirements sufficiently well without further rotation.We wish to thank Professor L. G. Henyey and the University of California Computer Center for making the IBM 701 electronic computer available for this study, and the National Science Foundation for its support of the work of the Computer Center. Professor H. F. Kaiser of the University of Illinois has made helpful criticisms of the paper, and Mr. Louis S. Davis of the University of California has assisted with preparation of the tables. The research was supported in part by the United States Air Force under Contract No. AF 33 (038)-25726 monitored by the Air Force Personnel and Training Research Center. Permission is granted for reproduction, translation, publication, use and disposal in whole and in part by or for the United States Government.A 701 program for calculation of the quartimax and varimax loadings, prepared by Professor H. F. Kaiser, is available in the library of computer programs held by the Computer Center at the University of California (Program No. 464). Mr. J. O. Neuhaus and Mr. K. W. Dickman have prepared a quartimax program for Illiac at the University of Illinois. This Illiac program will be usable on three other computers recently built or under construction: Mistic (Michigan State University), Silliac (University of Sydney), and the machine being constructed by Iowa State College.     

An IBM method for computing intraserial correlations

A method for computing intraserial correlations using a 602-A Calculating Punch, an 077 Collator, a 513 Gang Punch, and a 403 Tabulator is described. An example of the use of the procedure and an estimate of the time needed with each machine are given. This procedure is compared with another method, which makes use of a more powerful IBM machine.This research was supported in part by the United States Air Force under Contract No. AF 33(038)-25726, monitored by the Air Force Personnel and Training Research Center. Permission is granted for reproduction, translation, publication, use and disposal in whole and in part by or for the United States Government.     

Computer models of cognitive processes

Conclusion From the small sample of achievements that I have had time to mention, we can only conclude that automation is here to stay. Nor is there any doubt that more powerful automata will be built. A great many of the higher human abilities will be given to machines. The great rush to automation is sure to stimulate psychologists to learn more about the human symbolic processes being mimicked by the machines. And the computers, which are the ultimate cause of the feverish scramble toward automation, are providing both the framework for describing complex models of behavior and also the means for testing these models. With both the means and the motivation at hand, psychologists are sure to make rapid progress in understanding complex human behavior.Operated with support from the United States Army, Navy, and Air Force.     

TESTAN: An integrated modular system for personality assessment and test development on MS-DOS personal computers

TESTAN (TEST and ANalysis) is an authoring shell and integrated psychometric package for development of computerized multiple-choice questionnaires, tests, and personality scales on personal computers that use MS-DOS. TESTAN allows (1) writing and editing items, interpretation messages, keys, norms, a bank of profiles, and so on, (2) collecting data in on-line or paper-and-pencil modes, (3) selecting the most discriminating items by means of correlation and factor analysis for practical use, (4) validating test scales and items according to external criteria and expert ratings, and (5) making multifactor assessment decisions after testing. TESTAN can be used to test student conceptual knowledge in any area in the form of multiple-choice questions. This report describes essential functional properties and facilities of TESTAN for psychometrists and applied psychologists.