Measuring Cognitive Aptitude Using Unobtrusive Knowledge Tests: A New Survey Technology

Five knowledge tests and one implicit-reasoning task were developed to be: (1) exceptionally short, (2) correlated with general cognitive aptitude, (3) unobtrusive, i.e., appear similar to attitudinal survey items as opposed to maximal performance measures, and (4) without formally “correct” answers. The intent was to design scales that could be administered in non-proctored environments to directly measure general cognitive aptitude while avoiding the possibility that participants could use references to provide “good” answers. The five knowledge tests used a Likert format to assess knowledge in verbal and practical domains, and were scored by computing distances between examinee and reference ratings. The implicit-reasoning task appeared to be a series completion “game” that required a dichotomous response. The scales were administered to 288 Air Force recruits and were validated against the Armed Services Vocational Aptitude Battery (ASVAB). Individual unobtrusive knowledge scales and ASVAB tests were substantially correlated with sample correlations ranging to .39 and population correlation estimates to .66 after correcting for range restriction. Two sets of factor scores, which were separately derived from the unobtrusive test battery and the ASVAB, were highly correlated in our sample, .54, yielding a population correlation of .80 after correcting for range restriction. This technology is important because few paper- or Internet-based surveys, and virtually no mail-based surveys accurately measure general cognitive aptitude, while many of these surveys address important social issues and commercial questions that could be better understood given an unobtrusive but accurate estimate of general cognitive aptitude.