A comparison of paper-and-pencil and computerized forms of Line Orientation and Enhanced Cued Recall Tests

The purpose of this study was to assess whether a computerized battery of neuropsychological tests could produce similar results as the conventional forms. Comparisons on 77 volunteer undergraduates were carried out with two neuropsychological tests: Line Orientation Test and Enhanced Cued Recall Test. Firstly, students were assigned randomly across the test medium (paper-and-pencil versus computerized). Secondly, the groups were given the same test in the other medium after a 30-day interval between tests. Results showed that the Enhanced Cued Recall Test-Computer-based did not correlate with the Enhanced Cued Recall Test-Paper-and-pencil results. Line Orientation Test-Computer-based scores, on the other hand, did correlate significantly with the Line Orientation Test-Paper-and-pencil version. In both tests, scores were higher on paper-and-pencil tests compared to computer-based tests. Total score difference between modalities was statistically significant for both Enhanced Cued Recall Tests and for the Line Orientation Test. In both computer-based tests, it took less time for participants to complete the tests.     

Computerized assessment in neuropsychology: A review of tests and test batteries

This article contains detailed reviews of 13 computerized neuropsychological and performance test batteries and six stand-alone computer tests. Tasks found on these instruments are described and tables illustrate which batteries employ which measures. In addition to issues of reliability and validity, special considerations apply to computerized assessment. These issues are discussed and readers are provided information to help them assess computerized tests in relation to their particular clinical and research needs. Since many computerized tests were developed as performance assessment tools, the relationship between performance and neuropsychological assessment is examined.     

Computerized Neurocognitive Testing in the Management of Sport-Related Concussion: An Update

Since the late nineties, computerized neurocognitive testing has become a central component of sport-related concussion (SRC) management at all levels of sport. In 2005, a review of the available evidence on the psychometric properties of four computerized neuropsychological test batteries concluded that the tests did not possess the necessary criteria to warrant clinical application. Since the publication of that review, several more computerized neurocognitive tests have entered the market place. The purpose of this review is to summarize the body of published studies on psychometric properties and clinical utility of computerized neurocognitive tests available for use in the assessment of SRC. A review of the literature from 2005 to 2013 was conducted to gather evidence of test-retest reliability and clinical validity of these instruments. Reviewed articles included both prospective and retrospective studies of primarily sport-based adult and pediatric samples. Summaries are provided regarding the available evidence of reliability and validity for the most commonly used computerized neurocognitive tests in sports settings.     

Computerized adaptive personality testing: a review and illustration with the MMPI-2 Computerized Adaptive Version

Computerized adaptive testing in personality assessment can improve efficiency by significantly reducing the number of items administered to answer an assessment question. Two approaches have been explored for adaptive testing in computerized personality assessment: item response theory and the countdown method. In this article, the authors review the literature on each and report the results of an investigation designed to explore the utility, in terms of item and time savings, and validity, in terms of correlations with external criterion measures, of an expanded countdown method-based research version of the Minnesota Multiphasic Personality Inventory-2 (MMPI-2), the MMPI-2 Computerized Adaptive Version (MMPI-2-CA). Participants were 433 undergraduate college students (170 men and 263 women). Results indicated considerable item savings and corresponding time savings for the adaptive testing modalities compared with a conventional computerized MMPI-2 administration. Furthermore, computerized adaptive administration yielded comparable results to computerized conventional administration of the MMPI-2 in terms of both test scores and their validity. Future directions for computerized adaptive personality testing are discussed.     

ITC Guidelines on Quality Control in Scoring,Test Analysis,and Reporting of Test Scores

The Quality Control (QC) Guidelines are intended to increase the efficiency, precision, and accuracy of the scoring, analysis, and reporting process of testing. The QC Guidelines focus on large-scale testing operations where multiple forms of tests are created for use on set dates. However, they may also be used for a wide variety of other testing situations and assessment techniques and for almost any situation in which assessment occurs. The QC Guidelines are applicable in any form of test administration, including paper and pencil tests and the ever-increasing computerized assessments via the Internet or offline.     

Testing attention: Comparing the ANT with TVA-based assessment

Posner’s attention network model and Bundesen’s theory of visual attention (TVA) are two influential accounts of attention. Each model has led to the development of a test method: the attention network test (ANT) and TVA-based assessment, respectively. Both tests have been widely used to investigate attentional function in normal and clinical populations. Here we report on the first direct comparison of the ANT to TVA-based assessment. A group of 68 young healthy participants were tested in three consecutive sessions that each contained standard versions of the two tests. The parameters derived from TVA-based assessment had better internal reliability and retest reliability than did those of the standard version of the ANT, where only the executive network score reached comparable levels. However, when corrected for differences in test length, the retest reliability of the orienting network score equaled the least reliable TVA parameters. Both tests were susceptible to practice effects, which improved performance for some parameters while leaving others constant. All pairwise correlations between the eight attention parameters measured by the two tests were small and nonsignificant, with one exception: A strong correlation (r?=?0.72) was found between two parameters of TVA-based assessment, visual processing speed and the capacity of visual short-term memory. We conclude that TVA-based assessment and the ANT measure complementary aspects of attention, but the scores derived from TVA-based assessment are more reliable.     

The differential aptitude test (language usage and spelling): A clinical study of a computerized form

As part of the Leicester/DHSS project on microcomputer-aided assessment, 103 subjects were tested and retested on standard and computerized versions of the Differential Aptitude Tests for Language Usage and Spelling (Forms S and T). Subjects were generally faster on the computerized versions than on the standard versions. On the Language Usage test, subjects scored significantly higher on the computerized than on the standard test. The correlations found between the standard and computerized versions were modest in comparison to the original test-retest reliabilities. It is concluded that these data argue against the claim that the current computerized versions of the tests are psychometrically parallel to the standard versions. This research, which was carried out when the authors were at the Department of Psychology, University of Leicester, was supported by the United Kingdom Department of Health and Social Security.     

Administration and Environment Considerations in Computer-Based Sports-Concussion Assessment

Computer-based testing has become a vital tool for the assessment of sport-related concussion (SRC). An increasing number of papers have been published on this topic, focusing on subjects such as the purpose and validity of baseline testing, the performance of special populations on computer-based tests, the psychometric properties of different computerized neurocognitive tools, and considerations for valid and reliable administration of these tools. The current paper describes several considerations regarding computerized test design, input and output devices, and testing environment that should be described explicitly when administering computer-based cognitive testing, regardless of whether the assessment is used for clinical or research purposes. The paper also reviews the conclusions of recent literature (2007–2013) using computer-based testing for the assessment of SRC, with special attention to the methods used in these studies. We also present an appendix checklist for clinicians and researchers that may be helpful in ensuring proper attention to factors that could influence the reliability and validity of computer-based cognitive testing. We believe that explicit attention to these technological factors may lead to the development of standards for the development and implementation of computer-based tests. Such standards have the potential to enhance the accuracy and utility of computer-based tests in SRC.     

A process dissociation approach to objective-projective test score interrelationships

Even when self-report and projective measures of a given trait or motive both predict theoretically related features of behavior, scores on the 2 tests correlate modestly with each other. This article describes a process dissociation framework for personality assessment, derived from research on implicit memory and learning, which can resolve these ostensibly conflicting results. Research on interpersonal dependency is used to illustrate 3 key steps in the process dissociation approach: (a) converging behavioral predictions, (b) modest test score intercorrelations, and (c) delineation of variables that differentially affect self-report and projective test scores. Implications of the process dissociation framework for personality assessment and test development are discussed.     

DEVELOPMENT AND VALIDATION OF A COMPUTERIZED INTERPRETATION SYSTEM FOR PERSONNEL TESTS

A computerized system was developed for generating narrative interpretations of scores from a battery of personnel screening tests. The report structure and interpretive statement library were designed to capture the test expertise and interpretive strategies of a panel of testing experts. This was accomplished by enumerating the questions that the experts believed the battery could answer, developing answers to these questions, and devising rules for selecting the appropriate answers based on test-battery scores. The accuracy, thoroughness, readability, and coherence of the computer-generated reports were evaluated in comparison to reports generated by human experts for the same examinees. Results of the evaluation showed the computerized reports to be more accurate and thorough, as readable, and somewhat less coherent than interpretations generated by the typical human expert. The computerized system development and validation strategies described are useful for other applications in which numbers are interpreted in a narrative report format.     

Further explorations of perceptual speed abilities in the context of assessment methods, cognitive abilities, and individual differences during skill acquisition

Measures of perceptual speed ability have been shown to be an important part of assessment batteries for predicting performance on tasks and jobs that require a high level of speed and accuracy. However, traditional measures of perceptual speed ability sometimes have limited cost-effectiveness because of the requirements for administration and scoring of paper-and-pencil tests. There have also been concerns about the validity of previous computer approaches to administering perceptual speed tests (e.g., see Mead & Drasgow, 1993). The authors developed two sets of computerized perceptual speed tests, with touch-sensitive monitors, that were designed to parallel several paper-and-pencil tests. The reliability and validity of the tests were explored across three empirical studies (N = 167, 160, and 117, respectively). The final study included two criterion tasks with 4.67 and 10 hours of time-on-task practice, respectively. Results indicated that these new measures provide both high levels of reliability and substantial validity for performance on the two skill-learning tasks. Implications for research and application for computerized perceptual speed tests are discussed.     

Computerization and adaptive administration of the NEO PI-R

This study asks, how well does an item response theory (IRT) based computerized adaptive NEO PI-R work? To explore this question, real-data simulations (N = 1,059) were used to evaluate a maximum information item selection computerized adaptive test (CAT) algorithm. Findings indicated satisfactory recovery of full-scale facet scores with the administration of around four items per facet scale. Thus, the NEO PI-R could be reduced in half with little loss in precision by CAT administration. However, results also indicated that the CAT algorithm was not necessary. We found that for many scales, administering the "best" four items per facet scale would have produced similar results. In the conclusion, we discuss the future of computerized personality assessment and describe the role IRT methods might play in such assessments.     

Correlates of performance on the Gollin and Mooney tests of visual closure

One hundred and twenty-seven undergraduate students variously performed a computerized version of the Gollin (1960) Incomplete Figures Test, the Mooney (1957) Test of Incomplete Face Perception, the Poppelreuter (1917) Overlapping Figures Test, and a visual search task. Performance of male subjects was superior to that of female subjects on the Mooney test but inferior on the visual search task. Correlation and regression analyses showed that the only significant predictor of Gollin test scores was latency to identify all items in the Overlapping Figures Test. There was no relationship between performances on the Gollin and Mooney tests or between Gollin or Mooney test performance and visual search latency. The Gollin and Mooney tests appear to access different perceptual processes, none of which is dependent on the efficiency of visual search.     

Gs Invaders: Assessing a computer game-like test of processing speed

Computer games potentially offer a useful research tool for psychology but there has been little use made of them in assessing cognitive abilities. Two studies assessing the viability of a computer game-like test of cognitive processing speed are described. In Experiment 1, a computerized coding task that uses a mouse responsemethod (McPherson & Burns, 2005) was the basis for a simple computer game-like test. In Experiment 2, dynamic game-like elements were added. Validity was assessed within a factor analytic framework using standardized abilities tests as marker tests. We conclude that computer game-like tests of processing speed may provide an alternative or supplementary tool for research and assessment. There is clearly potential to develop game-like tests for other cognitive abilities.     

RETEST EFFECTS IN OPERATIONAL SELECTION SETTINGS: DEVELOPMENT AND TEST OF A FRAMEWORK

This study proposes a framework for examining the effects of retaking tests in operational selection settings. A central feature of this framework is the distinction between within-person and between-person retest effects. This framework is used to develop hypotheses about retest effects for exemplars of 3 types of tests (knowledge tests, cognitive ability tests, and situational judgment tests) and to test these hypotheses in a high stakes selection setting (admission to medical studies in Belgium). Analyses of within-person retest effects showed that mean scores of repeat test takers were one-third of a standard deviation higher for the knowledge test and situational judgment test and one-half of a standard deviation higher for the cognitive ability test. The validity coefficients for the knowledge test differed significantly depending on whether examinees' test scores on the first versus second administration were used, with the latter being more valid. Analyses of between-person retest effects on the prediction of academic performance showed that the same test score led to higher levels of performance for those passing on the first attempt than for those passing on the second attempt. The implications of these results are discussed in light of extant retesting practice.     

Validity and utility of computer-based test interpretation

Computers have been important to applied psychology since their introduction, and the application of computerized methods has expanded in recent decades. The application of computerized methods has broadened in both scope and depth. This article explores the most recent uses of computer-based assessment methods and examines their validity. The comparability between computer-administered tests and their pencil-and-paper counterparts is discussed. Basic decision making in psychiatric screening, personality assessment, neuropsychology, and personnel psychology is also investigated. Studies on the accuracy of computerized narrative reports in personality assessment and psychiatric screening are then summarized. Research thus far appears to indicate that computer-generated reports should be viewed as valuable adjuncts to, rather than substitutes for, clinical judgment. Additional studies are needed to support broadened computer-based test usage.     

The theory of test validity and correlated errors of measurement

In the theory of test validity it is assumed that error scores on two distinct tests, a predictor and a criterion, are uncorrelated. The expected-value concept of true score in the calssical test-theory model as formulated by Lord and Novick, Guttman, and others, implies mathematically, without further assumptions, that true scores and error scores are uncorrelated. This concept does not imply, however, that error scores on two arbitrary tests are uncorrelated, and an additional axiom of “experimental independence” is needed in order to obtain familiar results in the theory of test validity. The formulas derived in the present paper do not depend on this assumption and can be applied to all test scores. These more general formulas reveal some unexpected and anomalous properties of test validty and have implications for the interpretation of validity coefficients in practice. Under some conditions there is no attenuation produced by error of measurement, and the correlation between observed scores sometimes can exceed the correlation between true scores, so that the usual correction for attenuation may be inappropriate and misleading. Observed scores on two tests can be positively correlated even when true scores are negatively correlated, and the validity coefficient can exceed the index of reliability. In some cases of practical interest, the validity coefficient will decrease with increase in test length. These anomalies sometimes occur even when the correlation between error scores is quite small, and their magnitude is inversely related to test reliability. The elimination of correlated errors in practice will not enhance a test's predictive value, but will restore the properties of the validity coefficient that are familiar in the classical theory.     

Analysis of effects of distribution of practice in learning and retention of a continuous and a discrete skill presented on a computer

This investigation examined the effects of distributed and massed practice on the learning and retention of a discrete computerized skill (Exp. 1) and a continuous computerized skill (Exp. 2). 40 men were randomly assigned to one of four groups, of which two groups took part in Exp. 1 and two groups in Exp. 2. Performance was assessed at various points during acquisition and then on 8 retention tests conducted at varying times after acquisition. Learning curves for practice were highly similar for the two conditions. Participants in the distributed-practice group performed significantly better than those in the massed-practice group at the end of practice on both the discrete and continuous skills. However, participants in the distributed-practice group performed significantly more poorly on retention during 24 hr. and after acquisition. Participants in the massed-practice condition performed significantly better on retention tests than did those who learned in the distributed-practice condition.     

Correlates of intelligence in computer measured aspects of prose vocabulary: word length, diversity, and rarity

Computer measured aspects of prose vocabulary as correlates of intelligence are of interest because they offer the potential of assessing intelligence in situations where more direct assessment (e.g. through IQ tests) is either impractically expensive or (as in the case of populations that lived in the past) impossible. This study assessed a word-length measure (average number of letters), two word-diversity measures (ratio of number of different to number of total words, and Yule's Characteristic K, which indicates the repeat rate for words), and a word-rarity measure (proportion of words present on a rare-words list). In the first part of the study, essays of 120 students in Grade 11 and 12 in a private American high-school for whom Cooperative School and College Ability Test (SCAT) scores (which correlate with IQ test scores) were available, were assessed in terms of the vocabulary measures. Only the word-rarity and word-length measures correlated significantly with SCAT scores, and the highest correlations were manifested by the word-rarity measure. In the study's second part, the vocabulary measures were applied to articles selected from American newspapers representing African American (119 articles), general (110 articles), and Jewish-American (109 articles) communities, among which, for whatever reasons, reliable average IQ performance differences have been found. Only the word-rarity measure discriminated in the predicted way among the three sorts of newspapers. Implications for other potential uses of the computerized word-rarity measure for assessing temporal, social, and geographic group differences in intelligence are discussed.