Equivalence of Computerized and Standard Administration of the Piers-Harris Children's Self-Concept Scale

Two hundred seventy-nine 8- to 17-year-old children and adolescents were randomly assigned to complete paper-and-pencil or computer-administered versions of the Piers-Harris Children's Self-Concept Scale. The equivalence of these two administration modes was assessed by evaluating the comparability of scale means, variances, reliabilities, and validities. Results indicate that, of 27 comparisons, only 1 produced a statistically significant difference. We concluded that the computerized administration of the Piers-Harris scale did not affect its psychometric properties and that paper-and-pencil and computerized modes of testing for this questionnaire may be regarded as equivalent.     

Response latencies and the computerized assessment of intelligence

A study was undertaken to examine the relationship between response latencies to verbal ability test items administered by computer and overall verbal intelligence test scores. Sixty-four undergraduate students responded to a test of verbal ability under four conditions of alternate test forms (A or B) and modes of administration (computerized vs paper-and-pencil). The response latencies recorded during computerized testing, averaged for each subject, showed a negative correlation with overall test scores as would be predicted from a speed-of-information-processing perspective of human intelligence (Jensen, 1982a, b; Vernon, 1983). This inverse relationship was evident in every condition of test form and mode of administration, thereby demonstrating the generalizability of these findings. Discussion considered the implications of test speededness for the results of this study and provided suggestions for future research employing response latency data as a means for studying the cognitive processes underlying intelligent behaviour.     

Paper-and-pencil or online? Evaluating mode effects on measures of emotional functioning and attachment

The viability of using the World Wide Web to collect data from three widely used instruments by clinicians and researchers was investigated. The instruments were the Inventory of Parental and Peer Attachment, the Negative Mood Regulation Scale, and the Trait Meta-Mood Scale. Data were collected from two comparable groups of college students, and differences in response patterns on paper-and-pencil and World Wide Web versions of the measures, at both the item level and scale score level, were documented. Although mode of administration effects were statistically significant, the magnitude of the effects was in general very small. The basic similarity of the properties of the measures using paper-and-pencil and online Internet modes of administration suggests the viability of the Internet for assessing these and other psychological phenomena.     

Assessing the measurement invariance of latent job satisfaction ratings across survey administration modes for respondent subgroups: A MIMIC modeling approach

To date, many studies have assessed the measurement invariance of a wide variety of measures across Internet and paper-and-pencil conditions; however a relative dearth exists in the literature investigating measurement invariance across administration modes for differing subgroups of respondents. Using MIMIC modeling, this study assessed whether gender and age of the respondent systematically influenced responding according to administration mode above and beyond measurement invariance. Consistent with past research, this study demonstrated that job satisfaction ratings were indeed measurement invariant across Internet and paper-and-pencil conditions, however, older respondents tend to differentially rate job satisfaction according to administration mode. Implications are discussed.     

网络测验和纸笔测验的测量不变性研究：以生活满意度量表为例

以生活满意度量表为例,运用实证性因素分析,考察在中国文化下网络测验和传统纸笔测验之间的测量不变性。结果显示,网络测验和纸笔测验之间存在弱不变性,即网络测验和纸笔测验有着相同的测量单位;但网络测验和纸笔测验只存在部分的强不变性和部分的严格不变性,测验实施环境对结果的影响不可忽视。该研究表明,恰当设计的网络测验是可靠的,同时还提示,当一个测验在不同情境下运用时,检验测量不变性十分必要     

Further explorations of perceptual speed abilities in the context of assessment methods, cognitive abilities, and individual differences during skill acquisition

Measures of perceptual speed ability have been shown to be an important part of assessment batteries for predicting performance on tasks and jobs that require a high level of speed and accuracy. However, traditional measures of perceptual speed ability sometimes have limited cost-effectiveness because of the requirements for administration and scoring of paper-and-pencil tests. There have also been concerns about the validity of previous computer approaches to administering perceptual speed tests (e.g., see Mead & Drasgow, 1993). The authors developed two sets of computerized perceptual speed tests, with touch-sensitive monitors, that were designed to parallel several paper-and-pencil tests. The reliability and validity of the tests were explored across three empirical studies (N = 167, 160, and 117, respectively). The final study included two criterion tasks with 4.67 and 10 hours of time-on-task practice, respectively. Results indicated that these new measures provide both high levels of reliability and substantial validity for performance on the two skill-learning tasks. Implications for research and application for computerized perceptual speed tests are discussed.     

Differences in paper-and-pencil versus computerized line bisection according to ADHD subtype and hand-use

Two versions of the line bisection task, paper-and-pencil and computerized, were administered to non-medicated children (5-12 years) with and without Attention-Deficit/Hyperactivity Disorder (ADHD). Fifteen children were classified with ADHD-Inattentive type (ADHD-I), 15 were classified with ADHD-Combined or Hyperactive-Impulsive type (ADHD-C), and 15 children served as controls. During the paper-and-pencil task, and irrespective of hand-use, participants with ADHD-C bisected lines with a right bias, whereas participants with ADHD-I showed a leftwards bias. Interestingly, during the computerized version, an opposite pattern of hemineglect was observed with a leftwards bias for participants with ADHD-C and a rightwards bias for participants with ADHD-I. These findings suggest that different task demands are associated with the paper-and-pencil and computerized tasks. The findings also suggest that the two subtypes differ according to their cognitive profile, and possibly differ as to their underlying neural impairment.     

Taskomat: Evaluation of a Computerized Test Battery

A case study on a computerized test battery is presented based on more than a decade of use in military and civil settings. The study sheds light on the general issues associated with computerized testing. The accent is on the contrast between computerized and traditional paper-and-pencil testing. The computerized battery, called Taskomat, is founded on concepts arising from human performance theory. The tests of the battery are described, as well as issues related to the hardware (cost of purchasing, hardware reliability and industrial styling). Criteria used to evaluate the content of the tests were: reliability, robustness of effects, uniqueness or newness of abilities relative to paper-and-pencil testing, and validity of these abilities for predicting occupational success. A conclusion is that a theoretical foundation helps to increase the likelihood of assessing new abilities.     

Development of an automated performance test system for environmental and behavioral toxicology studies

In developing a menu of computerized performance tests for repeated-measures applications the metric properties of selected tests have been examined. Factors of chief concern have been stability and reliability, as well as the practical issue of the length of time it takes to achieve high levels of both. In this study, these factors, as well as predictive validity, are examined. 25 subjects were tested repeatedly (10 sessions) with 11 tests previously identified as "good" candidates for repeated-measures research in paper-and-pencil (marker test) versions. The 11 tests were administered concurrently in their traditional paper-and-pencil modes and newly implemented microcomputer-based versions, along with the Wechsler Adult Intelligence Scale (WAIS). Nine of the 11 microcomputer-based tests achieved stability. Reliabilities were generally high, with r greater than or equal to .77 for 3 min. of testing. Cross-correlations of microbased tests with traditional paper-and-pencil versions suggest equivalency between the test constructs in the different media. Correlations between six of the microbased subtests and the WAIS identified common variance, and these might comprise an efficient short (6 min.) battery of tests.     

Personality research on the Internet: A comparison of Web-based and traditional instruments in take-home and in-class settings

Students, faculty, and researchers have become increasingly comfortable with the Internet, and many of them are interestedin using the Web to collectdata. Few published studies have investigated the differences between Web-based data and data collected with more traditional methods. In order to investigate these potential differences, two important factors were crossed in this study: whether the data were collected on line or not and whether the data were collected in a group setting at a fixed time or individually at a time of the respondent’s choosing. The Visions of Morality scale (Shelton & McAdams, 1990) was used, and the participants were assigned to one of four conditions: in-class Web survey, in-class paper-and-pencil survey; take-home Web survey, and take-home paper-and-pencil survey. No significant differences in scores were found for any condition; however, response rates were affected by the type of survey administered, with the take-home Web-based instrument having the lowest response rate. Therefore, researchers need to be aware that different modes of administration may affect subject attrition and may, therefore, confound investigations of other independent variables.     

Computerizing the Mental Rotations Test: Are gender differences maintained?

A computerized version of the Revised Mental Rotations Test using touch-screen technology was tested and compared with the paper-and-pencil version. One hundred ninety-two participants--133 women and 59 men--took the paper-and-pencil version; 91 participants--47 women and 44 men--took the touch-screen version. Standard scores x test half and the entire test, proportion of errors x type, and proportion correct x alternative type were calculated. The expected gender differences occurred in standard scores and proportion correct x alternative type for both test versions. Men performed better than women in all instances, but gender difference effect sizes (measured by Cohen's d) were reduced from large for the paper-and-pencil version to medium for the computerized version. Scores declined at least nominally from the first to the second half of both versions, and significantly for women taking the paper-and-pencil version.     

Equivalence of computerized and conventional versions of the Beck Depression Inventory-II (BDI-II)

This study examined the equivalence of the conventional and computerized versions of the Beck Depression Inventory-II (BDI-II), taking into account that computer aver-sion may negatively impact computer-administered BDI-II scores by elevating them. Participants were 180 psychology undergraduate students from a medium-sized midwestern university. Participants were divided into four experimental groups. Each group was administered the BDI-II twice in various combinations (conventional only, computerized only, conventional and computerized and vice versa). All participants completed measures of computer aversion and computer experience. Participants who received both versions of the BDI-II were also asked to specify their preference for method of administration. Independent samples t-test results indicated that the com-puterized and paper-and-pencil versions of the BDI-II may be considered equivalent in terms of measurement validity. Implications for future research are discussed. Portions of this article were presented at the 1999 meetings of the American Psychological Association, Boston. The authors thank Dr. Michael Granaas for statistical assistance with the Latin Squares analysis.     

Equivalence of computerized and conventional versions of the Beck Depression Inventory-II (BDI-II)

This study examined the equivalence of the conventional and computerized versions of the Beck Depression Inventory-II (BDI-II), taking into account that computer aver-sion may negatively impact computer-administered BDI-II scores by elevating them. Participants were 180 psychology undergraduate students from a medium-sized midwestern university. Participants were divided into four experimental groups. Each group was administered the BDI-II twice in various combinations (conventional only, computerized only, conventional and computerized and vice versa). All participants completed measures of computer aversion and computer experience. Participants who received both versions of the BDI-II were also asked to specify their preference for method of administration. Independent samples t-test results indicated that the com-puterized and paper-and-pencil versions of the BDI-II may be considered equivalent in terms of measurement validity. Implications for future research are discussed. Portions of this article were presented at the 1999 meetings of the American Psychological Association, Boston. The authors thank Dr. Michael Granaas for statistical assistance with the Latin Squares analysis.     

Evaluation of a networked self-testing program

The use of a computerized, multiple-choice test bank to present practice and assessment tests on a network was evaluated with 46 men and 119 women from a first-year class in psychology. A correlation of .65 (p < .001) between scores on a traditional paper-and-pencil test and scores on a computerized test provided some validity for the computerized assessment. Regression analysis showed that ability (previous academic performance) and motivation (number of practice tests taken) accounted for 73% of the explained variance in computerized test scores. Sex differences did not enter the regression equation significantly.     

A comparison of paper-and-pencil and computerized forms of Line Orientation and Enhanced Cued Recall Tests

The purpose of this study was to assess whether a computerized battery of neuropsychological tests could produce similar results as the conventional forms. Comparisons on 77 volunteer undergraduates were carried out with two neuropsychological tests: Line Orientation Test and Enhanced Cued Recall Test. Firstly, students were assigned randomly across the test medium (paper-and-pencil versus computerized). Secondly, the groups were given the same test in the other medium after a 30-day interval between tests. Results showed that the Enhanced Cued Recall Test-Computer-based did not correlate with the Enhanced Cued Recall Test-Paper-and-pencil results. Line Orientation Test-Computer-based scores, on the other hand, did correlate significantly with the Line Orientation Test-Paper-and-pencil version. In both tests, scores were higher on paper-and-pencil tests compared to computer-based tests. Total score difference between modalities was statistically significant for both Enhanced Cued Recall Tests and for the Line Orientation Test. In both computer-based tests, it took less time for participants to complete the tests.     

The equivalence of computerized and paper-and-pencil psychological instruments: Implications for measures of negative affect

The use of computerized psychological assessment is a growing practice among contemporary mental health professionals. Many popular and frequently used paper-and-pencil instruments have been adapted into computerized versions. Although equivalence for many instruments has been evaluated and supported, this issue is far from resolved. This literature review deals with recent research findings that suggest that computer aversion negatively impacts computerized assessment, particularly as it relates to measures of negative affect. There is a dearth of equivalence studies that take into account computer aversion’s potential impact on the measurement of negative affect. Recommendations are offered for future research in this area.     

TESTAN: An integrated modular system for personality assessment and test development on MS-DOS personal computers

TESTAN (TEST and ANalysis) is an authoring shell and integrated psychometric package for development of computerized multiple-choice questionnaires, tests, and personality scales on personal computers that use MS-DOS. TESTAN allows (1) writing and editing items, interpretation messages, keys, norms, a bank of profiles, and so on, (2) collecting data in on-line or paper-and-pencil modes, (3) selecting the most discriminating items by means of correlation and factor analysis for practical use, (4) validating test scales and items according to external criteria and expert ratings, and (5) making multifactor assessment decisions after testing. TESTAN can be used to test student conceptual knowledge in any area in the form of multiple-choice questions. This report describes essential functional properties and facilities of TESTAN for psychometrists and applied psychologists.