Assessing the measurement invariance of latent job satisfaction ratings across survey administration modes for respondent subgroups: A MIMIC modeling approach

To date, many studies have assessed the measurement invariance of a wide variety of measures across Internet and paper-and-pencil conditions; however a relative dearth exists in the literature investigating measurement invariance across administration modes for differing subgroups of respondents. Using MIMIC modeling, this study assessed whether gender and age of the respondent systematically influenced responding according to administration mode above and beyond measurement invariance. Consistent with past research, this study demonstrated that job satisfaction ratings were indeed measurement invariant across Internet and paper-and-pencil conditions, however, older respondents tend to differentially rate job satisfaction according to administration mode. Implications are discussed.     

Personality assessment: Does the medium matter? No

The equivalence of an Internet administration of personality tests with two other administration formats was assessed using Item Response Theory (IRT) and various other statistical methods. The analyses were conducted on measures of Neuroticism, Extroversion, Agreeableness, and Conscientiousness. A total of 728 participants took part in the study. Participants were randomly assigned to one of three administrative conditions: paper-and-pencil, proctored computer lab, and unproctored Internet. Analyses with IRT, factor analysis, criterion-related validity, and mean differences supported the equivalence of Internet and traditional paper-and-pencil administrations of personality tests.     

Computer-administered versus paper-and-pencil surveys and the effect of sample selection

Airport patrons answered a self-administered questionnaire regarding their satisfaction with various airport facilities and operations. The questionnaire was administered either by a computer touch-sensitive screen or by a contextually identical paper-and-pencil version. For the latter method, respondents were selected randomly, and for the former, they were either randomly selected or self-selected. The effect of the method of questionnaire administration on the rating scales was very small when the samples were selected at random. However, there were substantial differences in ratings between self-selected and randomly selected respondents: The former gave consistently more negative ratings. These differences are probably due to psychological factors such as motivation to participate. Also, it was found that self-selected persons using the computer were more likely to make comments. The findings of this study are discussed with emphasis on their implications for computer interactive surveys.     

RELAXATION TREATMENT OF PSEUDOINSOMNIA AND IDIOPATHIC INSOMNIA: AN ELECTROENCEPHALOGRAPHIC EVALUATION

Twenty-nine insomniacs underwent four consecutive sleep laboratory evaluations before and after receiving tension-release relaxation training, no-tension-release relaxation training, or no-treatment. On the basis of the discrepancy between subjective and EEG-defined measures of latency to sleep onset, subjects were classified as pseudoinsomniacs or idiopathic insomniacs. As predicted, tension-release relaxation was significantly more effective than the other two conditions on subjective sleep measures, regardless of insomnia subtype and on objective sleep measures only for idiopathic insomniacs. Subjective improvement was maintained at 12-month followup. Numerous differences between the two subtypes emerged on pretherapy and during-therapy measures distinct from the latency measures, but changes on those variables were unrelated to outcome improvement.     

Socially Desirable Responding in Computerized Questionnaires: When Questionnaire Purpose Matters More Than the Mode1

The impact of questionnaire purpose (job screening interview vs. consumer survey) and the impact of questionnaire mode (paper-and-pencil vs. computer) on multiple measures of socially desirable responding (SDR) were examined. Students (N= 85) participated in experimental job screening (high SDR demand) and consumer survey (low SDR demand) conditions. Dependent measures included the Marlowe-Crowne Social Desirability Scale and the impression management subscale of the Balanced Inventory of Desirable Responding, Version 6. Significant questionnaire purpose effects suggest one possible, context-related explanation for mixed SDR findings between earlier experiments. Mode effects were nonsignificant, adding further evidence of paper-and-pencil and computer equivalence with respect to SDR.     

Paper-and-pencil or online? Evaluating mode effects on measures of emotional functioning and attachment

The viability of using the World Wide Web to collect data from three widely used instruments by clinicians and researchers was investigated. The instruments were the Inventory of Parental and Peer Attachment, the Negative Mood Regulation Scale, and the Trait Meta-Mood Scale. Data were collected from two comparable groups of college students, and differences in response patterns on paper-and-pencil and World Wide Web versions of the measures, at both the item level and scale score level, were documented. Although mode of administration effects were statistically significant, the magnitude of the effects was in general very small. The basic similarity of the properties of the measures using paper-and-pencil and online Internet modes of administration suggests the viability of the Internet for assessing these and other psychological phenomena.     

网络测验和纸笔测验的测量不变性研究：以生活满意度量表为例

以生活满意度量表为例,运用实证性因素分析,考察在中国文化下网络测验和传统纸笔测验之间的测量不变性。结果显示,网络测验和纸笔测验之间存在弱不变性,即网络测验和纸笔测验有着相同的测量单位;但网络测验和纸笔测验只存在部分的强不变性和部分的严格不变性,测验实施环境对结果的影响不可忽视。该研究表明,恰当设计的网络测验是可靠的,同时还提示,当一个测验在不同情境下运用时,检验测量不变性十分必要     

Response latencies and the computerized assessment of intelligence

A study was undertaken to examine the relationship between response latencies to verbal ability test items administered by computer and overall verbal intelligence test scores. Sixty-four undergraduate students responded to a test of verbal ability under four conditions of alternate test forms (A or B) and modes of administration (computerized vs paper-and-pencil). The response latencies recorded during computerized testing, averaged for each subject, showed a negative correlation with overall test scores as would be predicted from a speed-of-information-processing perspective of human intelligence (Jensen, 1982a, b; Vernon, 1983). This inverse relationship was evident in every condition of test form and mode of administration, thereby demonstrating the generalizability of these findings. Discussion considered the implications of test speededness for the results of this study and provided suggestions for future research employing response latency data as a means for studying the cognitive processes underlying intelligent behaviour.     

Internet administration of paper-and-pencil questionnaires used in couple research: assessing psychometric equivalence

This study examined the psychometric equivalence of paper-and-pencil and Internet formats of key questionnaires used in couple research. Self-report questionnaires assessing interpersonal constructs (relationship satisfaction, communication/conflict management, partner support, emotional intimacy) and intrapersonal constructs (individual traits, psychological symptoms, contextual influences) were administered to young adults in committed dating relationships. The same measures were administered twice via paper-and-pencil and/or Internet methods over a 2-week period. Method order was counterbalanced among participants, and temporal stability was controlled. Intrapersonal and interpersonal measures generally remained reliable when administered online and demonstrated quantitative and qualitative equivalence across methods. The implications of online administration of questionnaires are discussed, and specific recommendations are made for researchers who wish to transition to online data collection.     

WEB‐BASED AND PAPER‐AND‐PENCIL TESTING OF APPLICANTS IN A PROCTORED SETTING: ARE PERSONALITY,BIODATA, AND SITUATIONAL JUDGMENT TESTS COMPARABLE?

This quasi-experimental study compares the equivalence of proctored Web-based tests to paper-and-pencil tests in a selection setting. The predictor battery was composed of measures of Conscientiousness, Agreeableness, Emotional Stability, a biodata form, and a situational judgment test. Three samples were administered the same test items, but differed with respect to context and test format: 425 incumbents completed a paper-and-pencil version, 2,544 applicants completed a paper-and-pencil version, and 2,356 applicants completed an identical Web-based version (although these individuals were not randomly assigned to test format, the job and organization were the same for all samples). The results found effects for both context and test format. Relative to the applicants completing the paper-and-pencil measures, the Web-based measures showed (a) better distributional properties, (b) lower means, (c) more variance, (d) higher internal consistency reliabilities, and (e) stronger intercorrelations. Although the applicant context resulted in higher mean responses than the incumbent context, this difference was about twice as large for the paper-and-pencil test than for the Web-based test. All such differences were greater for personality measures than for biodata and situational judgment tests. Overall, these results suggest that proctored Web-based testing has some positive benefits relative to paper-and-pencil measures, and we identify several implications of these findings for research and practice.     

Training teachers as mediators: Efficacy of a workshop versus the bug-in-the-ear technique

The implementation of behavior modification techniques in school settings often relies on effective training of teachers as mediators. At the conclusion of Phase I, the performance of an experimental group of 13 teachers who were taught behavior modification by means of an inservice workshop was compared with a no-treatment control group. Although the experimental group scored higher on a paper-and-pencil test of behavior modification, there were no differences between the two groups on selected observed in-class behaviors. During Phase II, half of the experimental group of teachers received two hours of bug-in-the-ear (BIE) training in their own classrooms. BIE resulted in changes in the classroom behaviors of praise and contingency statements. These results are discussed in terms of the situation and response mode specificity of behavior and in terms of the implications for future workshops and research.     

Further explorations of perceptual speed abilities in the context of assessment methods, cognitive abilities, and individual differences during skill acquisition

Measures of perceptual speed ability have been shown to be an important part of assessment batteries for predicting performance on tasks and jobs that require a high level of speed and accuracy. However, traditional measures of perceptual speed ability sometimes have limited cost-effectiveness because of the requirements for administration and scoring of paper-and-pencil tests. There have also been concerns about the validity of previous computer approaches to administering perceptual speed tests (e.g., see Mead & Drasgow, 1993). The authors developed two sets of computerized perceptual speed tests, with touch-sensitive monitors, that were designed to parallel several paper-and-pencil tests. The reliability and validity of the tests were explored across three empirical studies (N = 167, 160, and 117, respectively). The final study included two criterion tasks with 4.67 and 10 hours of time-on-task practice, respectively. Results indicated that these new measures provide both high levels of reliability and substantial validity for performance on the two skill-learning tasks. Implications for research and application for computerized perceptual speed tests are discussed.     

Personality research on the Internet: A comparison of Web-based and traditional instruments in take-home and in-class settings

Students, faculty, and researchers have become increasingly comfortable with the Internet, and many of them are interestedin using the Web to collectdata. Few published studies have investigated the differences between Web-based data and data collected with more traditional methods. In order to investigate these potential differences, two important factors were crossed in this study: whether the data were collected on line or not and whether the data were collected in a group setting at a fixed time or individually at a time of the respondent’s choosing. The Visions of Morality scale (Shelton & McAdams, 1990) was used, and the participants were assigned to one of four conditions: in-class Web survey, in-class paper-and-pencil survey; take-home Web survey, and take-home paper-and-pencil survey. No significant differences in scores were found for any condition; however, response rates were affected by the type of survey administered, with the take-home Web-based instrument having the lowest response rate. Therefore, researchers need to be aware that different modes of administration may affect subject attrition and may, therefore, confound investigations of other independent variables.     

Individual differences in spatial learning from computer-simulated environments

A multivariate study examined relationships between the following factors: paper-and-pencil assessments of verbal and spatial ability, ability to form an accurate spatial representation of a large real-world environment, gender, computer attitudes and experience, proficiency with the navigational interface of a virtual environment (VE), and the ability to acquire and transfer spatial knowledge from a VE. Psychometrically assessed spatial ability and proficiency with the navigational interface were found to make substantial contributions to individual differences in the ability to acquire spatial information from a VE. Gender influenced many VE-related tasks, primarily through its relationship with interface proficiency and spatial ability. Measures of spatial knowledge of a VE maze were highly predictive of subsequent performance in a similar real-world maze, suggesting that VEs can be useful for training people about real-world spaces.     

The relationships of basic information processing measures with fluid and crystallized intelligence

The aim of the present research is to study the nature of Basic unit of Information Processing (BIP) by means of two related experiments. The first of these experiments uses structural equation models to attempt to confirm the relationships between these measures and crystallized intelligence proposed by Draycott and Kline (Personality and Individual Differences, 20, 201–210, 1994). The second attempts to find out more about the relationships between this and other Chronometrie measures both in computer and paper and pencil administration. Results seem to confirm the relationship between BIP and crystallized intelligence and it seems to be extensive to other paper and pencil measures such as the visual forms test. The relationships between task complexity and the size of the correlation between Chronometric measures and intelligence are also discussed.     

The Relationship of Paper-and-Pencil Wellness Measures to Objective Physiological Indexes

College freshmen completed a paper-and-pencil wellness assessment instrument and a supplemental questionnaire and were given a number of physical tests including measures of body composition, cholesterol, blood pressure, and pulse rate. Analyses indicated there were no significant relationships between specific paper-and-pencil physical scores and specific objective physiological indicators but that when several wellness scores were considered simultaneously and general indicators of health were used, significant relationships did exist. There were also significant relationships between wellness scores and people's perceptions of their health. Limitations and implications of the results are discussed.     

Visual analogue scales for mode-independent measurement in self-administered questionnaires

Previous research has shown evidence for mode differences between computer-assisted self-administered interviews (CASI) and paper-and-pencil interviews, especially in the case of sensitive questions. Some of these differences are explained by higher degrees of self-disclosure for CASI than for paper-and-pencil interviews, due to the more private situation with CASIs. This analysis examines the existence of different degrees of self-disclosure for CASI versus paper-and-pencil questionnaires and whether these differences can be reduced by the use of a specific response format. Judgments of items on a self-control scale with discrete 5-point (Likert-type) scales are compared with judgments on continuous visual analogue scales (VAS). Because a categorization effect for Likert-type items is assumed when pressure for social desirability is present, it is hypothesized that VAS compared to Likert-type response formats are less sensitive to mode differences.     

Assessment of role-play generalization across qualitatively different situations

Role-play generalization of newly acquired assertive behavior was assessed across different response situations in a clinical outpatient sample. Prior to assertive training, subjects completed two paper-and-pencil inventories and a series of role-play situations. Behavioral role-play scenes included four response forms: making requests, expressing affection, standing up for rights, and expressing displeasure. Subjects were randomly assigned to treatment or test-retest conditions and the treatment subjects received six 2-hr sessions of assertion training using modeling, behavior rehearsal, instruction, positive reinforcement, assignedin vivo homework, bibliotherapy, and daily recordings of assertive behavior. Posttreatment, subjects completed the paper-and-pencil measures, trained role-play scenes, and a set of dissimilar untrained scenes requiring different response forms (giving compliments, receiving compliments; refusing unreasonable requests and expressing justified anger). Videotapes were rated blindly and in random order at the conclusion of the study by trained judges. Treatment and test-retest subjects were equivalent prior to training. Posttreatment for the two groups differed in self-report, role-play behavior in trained situations, and role-play behavior in untrained dissimilar situations (allp<.001). MANOVA with repeated measures confirmed that trained and generalization role-play scenes were quantitatively, as well as qualitatively, different, confirmed that treatment and test-retest subjects exhibited different performance levels posttreatment on both trained and generalization scenes, and revealed four significant interactions (allp<.001). Implications of the interactions are discussed. Canonical correlation revealed that subjects displayed significant commonality in their behavioral profiles across different response forms. The findings indicate that complex training resulted in generalization from trained situations to untrained and dissimilar generalization scenes for the clinical outpatient sample.     

The reliability of a computer interview for drug use/abuse information

A computer interview was developed to collect information about drug use/abuse from high school students. Responses to this interview were compared with those obtained from an identical paper-and-pencil questionnaire given to the same students, in counterbalanced order. Results were comparable for the two methods, but missing data and logically inconsistent answers obtained with the paper-and-pencil questionnaire were not possible with the computer interview. The overwhelming majority of respondents preferred the computer interview as a means of data collection.