Comparing Paper-and-Pencil and Internet Survey Methods Conducted in a Combat-Deployed Environment

The Unit Behavioral Health Needs Assessment Survey (UBHNAS) is a tool used to aid behavioral health providers in assessing the needs of service members of a military unit. Traditionally, the survey has been administered by paper-and-pencil; however, implementing the survey via the Internet may be a faster and easier way to administer the survey. There are, however, potential drawbacks to Internet survey administration, and previous studies comparing the 2 methods have yielded inconsistent findings. Using a between-subjects design with a combat-deployed military sample, the present study compared the 2 survey administration methods. Specifically, we assessed comparability for (a) psychological constructs, (b) auxiliary measures (such as completion time, frequency of missing data, and length of written comments), and (c) ratings and perceptions of the UBHNAS, and preference in methodology. The method of survey administration did not predict differences in the psychological outcome variables. However, participants in the paper-and-pencil condition reported higher rates of comfort answering questions, belief that responses would remain anonymous, and frankness and honesty in answering questions. Nevertheless, participants in both conditions preferred to take the survey via the Internet as opposed to paper-and-pencil in the future. Finally, participants in the Internet condition completed the survey faster and tended to write longer comments, although in terms of the comments being informative, there was no difference between the groups. Based on our findings, we recommend the use of the Internet for administering the UBHNAS; however, we caution that there may be tradeoffs regarding concerns about comfort, honesty, and anonymity.     

Evaluating the generalization of math fact fluency gains across paper and computer performance modalities

Computer-based interventions are being used more in the classroom. Student responses to these interventions often contribute to decisions making regarding important outcomes. It is important to understand the effect of these interventions within the context of the intervention as well as across related context. The current study examined the generalization of math fact fluency gains resulting from a computer-based intervention to paper-and-pencil performance. A total of 31 second grade students completed fluency drills on the computer or with paper and pencil. Pretest-posttest performance on both computer and paper and pencil for all students was evaluated using a doubly multivariate repeated measure ANOVA. Results indicated that gains achieved on the computer did not generalize to paper-and-pencil performance.     

Differences in responses to Web and paper surveys among school professionals

This experiment investigated the effects of survey administration mode (Web vs. paper and pencil) and survey length (short or long) on the responses of a large sample of primary and secondary school professionals in the U.S. The 812 participants in this study were part of an initial random sample of 1,000 individuals representing 5.81% of the membership of a national professional organization. The participants were randomly assigned to each of the four treatment conditions. Results indicated (1) substantially lower response rates for Web surveys than for same-length paper surveys; (2) a higher response rate for short surveys than for long surveys with paper, but not Web, questionnaires; and (3) a younger age for Web respondents, as compared with their paper counterparts. In light of prior research, we suggest that paper-and-pencil methods be used for surveying professionals in primary and secondary school settings.     

Personality assessment: Does the medium matter? No

The equivalence of an Internet administration of personality tests with two other administration formats was assessed using Item Response Theory (IRT) and various other statistical methods. The analyses were conducted on measures of Neuroticism, Extroversion, Agreeableness, and Conscientiousness. A total of 728 participants took part in the study. Participants were randomly assigned to one of three administrative conditions: paper-and-pencil, proctored computer lab, and unproctored Internet. Analyses with IRT, factor analysis, criterion-related validity, and mean differences supported the equivalence of Internet and traditional paper-and-pencil administrations of personality tests.     

The development and evaluation of a computer-administered measure of cognitive complexity

The present study presents and evaluates the Computer-Administered Rep Test (CART). The CART is a micro computer program that automates both the administration and scoring of a repertory grid based measure of cognitive complexity. Preliminary evaluation of the CART indicates that it provides a useful alternative to traditional paper-and-pencil administered repertory grid measures of cognitive complexity. We also examine the effect of an alternative scoring approach on the reliability of scores derived from role repertory grid measures of cognitive complexity. Results indicate that this scoring approach substantially enhances the reliability of cognitive complexity scores derived from both the paper-and-pencil and computer-administered measures.     

Personality research on the Internet: A comparison of Web-based and traditional instruments in take-home and in-class settings

Students, faculty, and researchers have become increasingly comfortable with the Internet, and many of them are interestedin using the Web to collectdata. Few published studies have investigated the differences between Web-based data and data collected with more traditional methods. In order to investigate these potential differences, two important factors were crossed in this study: whether the data were collected on line or not and whether the data were collected in a group setting at a fixed time or individually at a time of the respondent’s choosing. The Visions of Morality scale (Shelton & McAdams, 1990) was used, and the participants were assigned to one of four conditions: in-class Web survey, in-class paper-and-pencil survey; take-home Web survey, and take-home paper-and-pencil survey. No significant differences in scores were found for any condition; however, response rates were affected by the type of survey administered, with the take-home Web-based instrument having the lowest response rate. Therefore, researchers need to be aware that different modes of administration may affect subject attrition and may, therefore, confound investigations of other independent variables.     

New approaches to the design of computerized interviewing and testing systems

Most computer interviewing and testing systems have adopted paper-and-pencil approaches to information gathering with little modification. However, computer technology offers two fundamental advantages over paper-and-pencil technology for psychological information gathering: (1) A computer can record ancillary data such as latencies and pressure on response keys during an interviewing session, and (2) A computer can react adaptively to special events as these arise during a session. Ways to capitalize on these advantages are outlined. A pilot study of interviewee behavior during a computer problem-screening interview is described, and the implications of the results for future research in the area are discussed. Passive and active computer testing systems occupy positions on a continuum between paper-based psychological testing and the flexible, but less well controlled, technology represented by the human. With its unique capabilities, computer technology has a special role to play in the future of psychological measurement.     

Varieties of Inconsistency Across Test Occasions: Effects of Computerized Test Administration and Repeated Testing

This article investigates different types of instability in subjects' responses to the Minnesota Multiphasic Personality Inventory (MMPI). Indices were constructed to measure the effects of the automated testing format and repeated testing on 150 undergraduate subjects who took a computer-administered and pencil-and-paper MMPI 1 week apart. One set of indices measures systematic shifting, attributable to format or time alone. Two families of six indices each are computed measuring unsystematic changes in responding, tendencies to shift in particular directions between true, false, and cannot say responses across all 566 items. Unsystematic changes were assessed both between formats and across times, although these factors are confounded. Subjects tend to change their responses to cannot say from the pencil-and-paper to the computer administration and to shift from false to true on the second administration. Systematic shifting due to the test format is related to the more general, unsystematic tendency to shift between true and false responses. The number of cannot say responses in the computerized testing situation, although greater in magnitude, is correlated with the use of cannot say in the pencil-and-paper condition. Systematic shifting attributable to time is distinct from the other types of inconsistency. Subjects are more inconsistent across occasions when the test format changes than would be expected from repeated testing alone, although the consequences of this inconsistency for clinical interpretation are not yet clear. Inconsistent responding is related to subjects' personality characteristics, but not to age or prior experience with computers.     

The reliability of a computer interview for drug use/abuse information

A computer interview was developed to collect information about drug use/abuse from high school students. Responses to this interview were compared with those obtained from an identical paper-and-pencil questionnaire given to the same students, in counterbalanced order. Results were comparable for the two methods, but missing data and logically inconsistent answers obtained with the paper-and-pencil questionnaire were not possible with the computer interview. The overwhelming majority of respondents preferred the computer interview as a means of data collection.     

Computer-assisted versus paper-and-pencil administration of questionnaires

In a field experiment using 95 employees, the results of computer-assisted versus paper-and-pencil administration of questionnaires during computer training were compared. Unlike past research, the computer mode of administration was made as similar as possible, in format, to the paper mode. No differences in means or reliabilities were found between any variables across the two conditions. However, when the correlations were examined, different patterns of relationships were found across the two conditions. Specifically, subjective or affective measures (software efficacy, playfulness, and training reaction) were more highly related when the measures were gathered by computer as opposed to paper-and-pencil. The relationships between subjective measures and more objective or factual measures (experience, performance, training transfer) were not affected. Thus, it appears that, while differences in mode of administration may not be reflected in differences in means, there are differences in nomological validity.     

Using personal digital assistants (PDAs) for the collection of safety belt use data in the field

From 1975 through 2003, because of the use of safety belts, an estimated 180,000 deaths from traffic crashes have been prevented. In order to assess the gains that have been made in safety belt use across the U.S., in individual states, and in communities, belt use surveys are conducted at regular intervals to determine use rates. The most valid method for surveying safety belt use is through direct observation. Direct observation surveys are conducted along roadways by trained researchers looking into passing vehicles and recording safety belt use. This method of data collection has been effective in the past through the use of paper-and-pencil data recording, yet it could be improved through the use of electronic and communication technology. Reported here is a study designed to compare electronic data collection, using personal digital assistants (PDAs), with collection using the traditional paperand-pencil method during the annual statewide survey of safety belt use in Michigan that we have been conducting since 1984. The goals of the study were to develop a PDA database program for data entry in the field, to directly compare the PDA data collection process with the paper-and-pencil method on both accuracy and speed, and to assess mechanical and environmental factors, such as battery life, screen visibility, and reaction to adverse weather, that may act as limitations to the PDA method, in comparison with the paper-and-pencil method. In a direct comparison of methods, two observers collected data at the same roadway intersections, one using paper and a pencil and one using a PDA equipped with our custom software. The study showed that the PDA method was as fast and as accurate as the paper-and-pencil method. There were no adverse effects on the PDA caused by environmental conditions. The PDA was superior to the paper-and-pencil method in rainy weather and for data collector supervision. In addition, the use of the PDA obviated the need for entry of paper-recorded data into an electronic format. We conclude that the use of PDAs for safety belt field data collection is superior to the paper-and-pencil method. A Web address is given where the custom PDA safety belt data collection software can be obtained free of charge.     

Examining the impact of administration medium on examine perceptions and attitudes

The present study explored the impact of administration medium on examinees' affective reactions. The research compared managers' reactions with 3 versions of the Conflict Resolution Skills Assessment (J. B. Olson-Buchanan et al., 1998) that were identical in content but varied in terms of the level of technology used: a paper-and-pencil form, a written form administered by computer (i.e., a computerized page-turner), and a multimedia form administered by computer. Managers completing the multimedia assessment perceived the assessment as more face valid and had more positive attitudes, relative to managers who completed the other 2 assessments. Computerization, however, was not enough to make a difference; instead, it was the multimedia nature of the computer presentation that resulted in the most positive affective reactions. Study limitations and implications for research and practice are discussed.     

Further explorations of perceptual speed abilities in the context of assessment methods, cognitive abilities, and individual differences during skill acquisition

Measures of perceptual speed ability have been shown to be an important part of assessment batteries for predicting performance on tasks and jobs that require a high level of speed and accuracy. However, traditional measures of perceptual speed ability sometimes have limited cost-effectiveness because of the requirements for administration and scoring of paper-and-pencil tests. There have also been concerns about the validity of previous computer approaches to administering perceptual speed tests (e.g., see Mead & Drasgow, 1993). The authors developed two sets of computerized perceptual speed tests, with touch-sensitive monitors, that were designed to parallel several paper-and-pencil tests. The reliability and validity of the tests were explored across three empirical studies (N = 167, 160, and 117, respectively). The final study included two criterion tasks with 4.67 and 10 hours of time-on-task practice, respectively. Results indicated that these new measures provide both high levels of reliability and substantial validity for performance on the two skill-learning tasks. Implications for research and application for computerized perceptual speed tests are discussed.     

The effects of episodic and semantic memory on the identification of sentences masked by white noise

Two studies examine how episodic and semantic memory affect subjects’ abilities to repeat sentences masked by white noise. Subjects first hear a list of 70 sentences that are not masked. Subjects in the framework conditions are told prior to hearing the list that all sentences refer to a contextual framework concerning a deserted island. Subjects in the no-framework groups are not given this information. Subjects in the framework-after condition are given this information only after hearing the list of sentences. Subjects then perform a white-noise identification task. The results indicate that framework subjects are able to identify both old and new framework-related information better than other subjects. Subjects in the no-framework and framework-after conditions identify old information better than control subjects who do not participate in an acquisition phase. Emphasis is placed on the interdependence of episodic and semantic memory, including conditions leading to such interdependence.     

A preliminary study of achievement, attitudes toward success in mathematics, and mathematics anxiety with technology-based instruction in brief calculus

This study was designed to compare achievement, attitudes toward success in mathematics, and mathematics anxiety of college students taught brief calculus using a graphic calculator, with the achievement and attitudes and anxiety of students taught using the computer algebra system Maple, using a technology based text book. 50 men and 50 women, students in three classes at a large public university in the southwestern United States, participated. Students' achievement in brief calculus was measured by performance on a teacher-made achievement test given at the end of the study. Analysis of variance showed no significant difference in achievement between the groups. To measure change in attitudes and anxiety, responses to paper-and-pencil inventories indicated significant differences in favor of students using the computer.     

Long-term consequences of childhood physical and psychological maltreatment

This study assessed whether some of the correlates of learned helplessness—depression, low self-esteem, and a maladaptive attributional style—are long-term consequences of child abused. In this study, 260 subjects were identified as physically abused, psychologically abused, both physically and psychologically abused, or nonabused, based on responses to the Child Abuse Questionnaire (CAQ). Subjects were tested for levels of nonclinical depression via the Beck Depression Inventory (BDI), for levels of self-esteem via the Rosenberg Self-Esteem Scale (RSE), and for the adaptiveness of attributional style via the Attributional Style Questionnaire (ASQ). It was hypothesized that each of the three abuse groups would differ from the control group on the three dependent measures; differences among the three abuse groups were also explored. Three one-way analyses of variance indicated that (1) BDI scores reflected a greater tendency toward depression in subjects reporting both types of abuse than in nonabused subjects or in subjects reporting either psychological or physical abuse, (2) subjects reporting psychological abuse only or both psychological and physical abuse showed lower self-esteem than did nonabused subjects, and (3) abused subjects did not seem to exhibit a more maladaptive attributional style than that of nonabused subjects. In addition, multiple linear regression analyses pointed to psychological abuse as a critical variable in predicting levels of depression, self-esteem, and attributional style, when physical abuse effects were controlled. Controlling for the effects of psychological abuse, however, indicated that physical abuse did not significantly contribute to the variance in these variables. Implications for the learned helplessness model and for future research are discussed. © 1992 Wiley-Liss, Inc.     

Socially Desirable Responding in Computerized Questionnaires: When Questionnaire Purpose Matters More Than the Mode1

The impact of questionnaire purpose (job screening interview vs. consumer survey) and the impact of questionnaire mode (paper-and-pencil vs. computer) on multiple measures of socially desirable responding (SDR) were examined. Students (N= 85) participated in experimental job screening (high SDR demand) and consumer survey (low SDR demand) conditions. Dependent measures included the Marlowe-Crowne Social Desirability Scale and the impression management subscale of the Balanced Inventory of Desirable Responding, Version 6. Significant questionnaire purpose effects suggest one possible, context-related explanation for mixed SDR findings between earlier experiments. Mode effects were nonsignificant, adding further evidence of paper-and-pencil and computer equivalence with respect to SDR.     

Tonic and phasic stimulus control of blood pressure responses using opposing unconditional stimuli

The transswitching paradigm was used in the present study to investigate the effects of tonic (long duration) environmental stimuli paired with opposing unconditional stimuli on human blood pressure. Sixty volunteers participated for two sessions, one week apart. Subjects were presented with a pseudorandom sequence of four trials of red lights (five minutes each) and four trials of blue lights (five minutes each) within each session. All subjects received the cold pressor test during the red lights. During the blue lights, half the subjects received warm water and the other half received a neutral temperature water. Subjects received the presentations of the water in one of three methods: as determined by the experimenter, preceded by a warning signal, or self-administered. The results indicated that conditional and unconditional responses during the red lights were progressive increases in blood pressure. During the blue lights associated with warm water, conditional and unconditional responses were often progressive decreases in blood pressure. The method of administration of the water affected the magnitude of the unconditional responses. Evidence for phasic stimulus control was not as unequivocal as tonic stimulus control. The present results indicated that it is possible to modify blood pressure in two directions (increases and decreases) when two high contrast, opposing unconditional stimuli are presented.     

A Comparison of the Effectiveness of Values Clarification Presented as a Personal Computer Program versus a Traditional Therapy Group:

Eight adolescent inpatients were assigned to two groups equated as much as possible by matching the subjects for age, sex, and diagnosis. The experimental group viewed an individual self-paced computer program on values clarification, and the control group attended a single group therapy session covering identical content. Pretesting and posttesting was done using the Tennessee Self-concept Scale and the Internal-External Control Scale. No significant differences were found using these tests as measures of effectiveness, but on the post-treatment attitude survey, subjects in the computer group reported greater perceived effectiveness than those in the control group. The computer program was well accepted and not related to prior exposure to computers. Subjects completed the computer program in less than half the time required to present the material in the therapy group.