Comparing continuous and dichotomous scoring of the balanced inventory of desirable responding

The Balanced Inventory of Desirable Responding (BIDR; Paulhus, 1994) is a widely used instrument to measure the 2 components of social desirability: self-deceptive enhancement and impression management. With respect to scoring of the BIDR, Paulhus (1994) authorized 2 methods, namely continuous scoring (all answers on the continuous answer scale are counted) and dichotomous scoring (only extreme answers are counted). In this article, we report 3 studies with student samples, and continuous and dichotomous scoring of BIDR subscales are compared with respect to reliability, convergent validity, sensitivity to instructional variations, and correlations with personality. Across studies, the scores from continuous scoring (continuous scores) showed higher Cronbach's alphas than those from dichotomous scoring (dichotomous scores). Moreover, continuous scores showed higher convergent correlations with other measures of social desirability and more consistent effects with self-presentation instructions (fake-good vs. fake-bad instructions). Finally, continuous self-deceptive enhancement scores showed higher correlations with those traits of the Five-factor model for which substantial correlations were expected (i.e., Neuroticism, Extraversion, and Conscientiousness). Consequently, these findings indicate that continuous scoring may be preferable to dichotomous scoring when assessing socially desirable responding with the BIDR.     

Personality and Creativity in Art and Writing: Innovation Motivation,Psychoticism, and (Mal)Adjustment

Undergraduates (N = 68) completed inventories measuring innovation motivation (need to be different and innovation expectancy), psychoticism, extraversion, neuroticism, and symptomatic distress, as well as a sentence completion measure of adjustment. They also wrote lyric poems using an associative procedure and completed house–tree–person drawings. Poems were scored for originality and arousal potential and independently judged for quality by two college writing instructors. Drawings were scored for original features and independently judged for quality by two art therapists. The need to be different and current adjustment problems correlated significantly with both originality scores and judged creativity on both tasks, often interacting to explain much of the variance in creativity. Innovation expectancy and psychoticism displayed significant correlations with some creativity measures. Originality scoring and expert judgment correlated well on both tasks. Originality and judged creativity correlated significantly across creative domains, supporting a domain-general view of creativity.     

Predicting academic success with the Big 5 rated from different points of view: Self‐rated,other rated and faked

Self‐ratings of personality predict academic success above general intelligence. The present study replicated these findings and investigated the increment of other‐ratings or intentionally distorted self‐ratings. Participants (N = 145) had to compile a personality questionnaire twice. First they were given neutral instructions. The second time they were asked to imagine a specific applicant setting. Furthermore, two peers rated each participant. Additionally, verbal, numerical and figural reasoning scores were obtained. Grades on a statistics exam obtained 2 months later served as the criterion. Results replicated prior findings and showed incremental validity for self‐ and other‐rated personality, which was stable after controlling for intelligence. Faking had no impact on the domain‐score level, but results on the facet‐score level were less encouraging. Copyright © 2010 John Wiley & Sons, Ltd.     

Be Careful What You Ask For: Effects of Response Instructions on the Construct Validity and Reliability of Situational Judgment Tests

The aim of this study was to examine how six different types of situational judgment test (SJT) instructions, used frequently in practice, influence the psychometric characteristics of SJTs. The six SJT versions used the exact same items and differed only in their instructions; these versions were administered in two phases. Phase I was a between–subjects design (n= 486) that had participants complete one version of the SJTs. Phase II was a within–subjects design (n= 231) held several weeks later that had participants complete all six versions of the SJTs. Further, 146 of these individuals completed both phases, allowing for an assessment of test–retest reliability. A variety of objective and subjective criteria were collected, including self and peer ratings. Results indicated that instructions had a large effect on SJT responses, reliability, and validity. In general, instructions asking what one ‘would do’ showed more favorable characteristics than those that asked what one ‘should do’. Correlations between these two types were relatively low despite the fact that the same items were used, and criterion–related validities differed substantially in favor of the ‘would do’ instructions. Overall, this study finds that researchers and practitioners must give careful consideration to the types of SJT instructions used; failing to do so could influence criterion–related validity and cloud inferences of construct validity.     

Psychometric properties of the Psychological Flourishing Scale in an Egyptian setting

The specific goals of the present research were twofold: i) to test the structure validity, and internal reliability of scores from the Psychological Flourishing Scale (PFS) with Egyptian college students; and ii) to test the PFS’s temporal stability and convergent validity properties. Two samples of Egyptian college students completed the PFS (n = 374; n = 173) to address the two research aims in two studies. The students also completed the Mental Health Continuum-Short Form (MHC-SF) and the Satisfaction With Life Scale (SWLS) for determining the concurrent validity of scores. In the first study, the internal consistency was evaluated. Exploratory Factor Analysis (EFA) and Confirmatory Factor Analysis (CFA) were performed. In the second, the temporal stability and convergent validity were tested. The Arabic version of PFS showed a high internal reliability, structure, and convergent validity, and moderate temporal stability.     

Multimedia instructions and cognitive load theory: Effects of modality and cueing

Background: Recent research on the influence of presentation format on the effectiveness of multimedia instructions has yielded some interesting results. According to cognitive load theory (Sweller, Van Merriënboer, & Paas, 1998) and Mayer's theory of multimedia learning (Mayer, 2001), replacing visual text with spoken text (the modality effect) and adding visual cues relating elements of a picture to the text (the cueing effect) both increase the effectiveness of multimedia instructions in terms of better learning results or less mental effort spent. Aims: The aim of this study was to test the generalisability of the modality and cueing effect in a classroom setting. Sample: The participants were 111 second‐year students from the Department of Education at the University of Gent in Belgium (age between 19 and 25 years). Method: The participants studied a web‐based multimedia lesson on instructional design for about one hour. Afterwards they completed a retention and a transfer test. During both the instruction and the tests, self‐report measures of mental effort were administered. Results: Adding visual cues to the pictures resulted in higher retention scores, while replacing visual text with spoken text resulted in lower retention and transfer scores. Conclusions: Only a weak cueing effect and even a reverse modality effect have been found, indicating that both effects do not easily generalise to non‐laboratory settings. A possible explanation for the reversed modality effect is that the multimedia instructions in this study were learner‐paced, as opposed to the system‐paced instructions used in earlier research.     

Development and content validity of a screening instrument for gaming addiction in adolescents: The Gaming Addiction Identification Test (GAIT)

This study describes the development of a screening tool for gaming addiction in adolescents – the Gaming Addiction Identification Test (GAIT). Its development was based on the research literature on gaming and addiction. An expert panel comprising professional raters (n = 7), experiential adolescent raters (n = 10), and parent raters (n = 10) estimated the content validity of each item (I‐CVI) as well as of the whole scale (S‐CVI/Ave), and participated in a cognitive interview about the GAIT scale. The mean scores for both I‐CVI and S‐CVI/Ave ranged between 0.97 and 0.99 compared with the lowest recommended I‐CVI value of 0.78 and the S‐CVI/Ave value of 0.90. There were no sex differences and no differences between expert groups regarding ratings in content validity. No differences in the overall evaluation of the scale emerged in the cognitive interviews. Our conclusions were that GAIT showed good content validity in capturing gaming addiction. The GAIT needs further investigation into its psychometric properties of construct validity (convergent and divergent validity) and criterion‐related validity, as well as its reliability in both clinical settings and in community settings with adolescents.     

An Abbreviated Screening Instrument for Autism Spectrum Disorders

The reliability and validity of the Parent's Observations of Social Interactions (POSI), a new, seven‐item screening instrument for autism spectrum disorders (ASD), is examined in two substudies. In Study 1, parents of 217 children (18–48 months) evaluated at a developmental clinic completed intake questionnaires that included the POSI and the Modified Checklist for Autism in Toddlers (M‐CHAT) checklist. POSI and M‐CHAT scores were compared to clinical evaluation results to assess reliability and validity. In Study 2, parents of 232 children (16–36 months) from both primary care and subspecialty settings completed the POSI, the M‐CHAT, and a report of their child's diagnoses. POSI and M‐CHAT scores were compared to reported diagnoses to assess reliability and validity. In both studies, the POSI demonstrated adequate internal reliability (Cronbach α = 0.83 and 0.86, respectively). In Study 1, POSI sensitivity (89%) was higher than that for the M‐CHAT (71%; p < .05); specificities were not significantly different (POSI: 54%, M‐CHAT: 62%). In Study 2, sensitivity (83%) compared favorably to that for the M‐CHAT (50%), although specificity was lower (75 vs. 84%). Despite its brevity, the POSI demonstrated good internal reliability and comparable sensitivity/specificity to the M‐CHAT checklist in two independent populations. If results are reproduced in larger, more diverse samples, the POSI may provide an efficient method for ASD screening in young children.     

Factor Analysis of the Mystical Experience Questionnaire: A Study of Experiences Occasioned by the Hallucinogen Psilocybin

A large body of historical evidence describes the use of hallucinogenic compounds, such as psilocybin mushrooms, for religious purposes. But few scientific studies have attempted to measure or characterize hallucinogen‐occasioned spiritual experiences. The present study examined the factor structure of the Mystical Experience Questionnaire (MEQ), a self‐report measure that has been used to assess the effects of hallucinogens in laboratory studies. Participants (N = 1,602) completed the 43‐item MEQ in reference to a mystical or profound experience they had had after ingesting psilocybin. Exploratory factor analysis of the MEQ retained 30 items and revealed a four‐factor structure covering the dimensions of classic mystical experience: unity, noetic quality, sacredness (F1); positive mood (F2); transcendence of time/space (F3); and ineffability (F4). MEQ factor scores showed good internal reliability and correlated with the Hood Mysticism Scale, indicating convergent validity. Participants who endorsed having had a mystical experience on psilocybin, compared to those who did not, had significantly higher factor scores, indicating construct validity. The four‐factor structure was confirmed in a second sample (N = 440) and demonstrated superior fit compared to alternative models. The results provide initial evidence of the validity, reliability, and factor structure of a 30‐item scale for measuring single, hallucinogen‐occasioned mystical experiences, which may be a useful tool in the scientific study of mysticism.     

The effects of instructions and calculation procedures on observers' accuracy, agreement, and calculation correctness

Although the quality of observational data is generally evaluated by observer agreement, measures of both observer agreement and accuracy were available in the present study. Videotapes with a criterion protocol were coded by 16 observers. All observers calculated agreement scores both on their own and their partner's data and on a contrived data set misrepresented as data collected by other observers. Compared with agreement scores calculated by the experimenter, observers erroneously inflated their own agreement scores and deflated the agreement scores on the contrived data. Half of the observers (n = 8) had been given instructions emphasizing the importance of accuracy during observation while the other half had been given instructions emphasizing interobserver agreement. Accuracy exceeded agreement for the former group, whereas agreement exceeded accuracy for the latter group. The implications are that agreement should be calculated by the experimenter and that the accuracy-agreement relationship can be altered by differential observer instructions.     

Dyscontrol Experiences Questionnaire: Development, Reliability and Validity

This paper describes development of a retrospective measure of childhood exposure to parental dyscontrol related to specific conditions such as a parent's drinking, anger, or other negative emotional states (e.g., stress, anxiety, depression). A 45-item questionnaire, the Dyscontrol Experiences Questionnaire (DEQ), was designed to assess individuals' childhood experiences with parental loss of control. Two student samples (n = 465 and n = 536) completed a battery of self-report measures. Responses were used to examine the psychometric properties of the DEQ. Factor analysis revealed three lower-order factors accounting for 54.4% of the variance in DEQ item scores. Cronbach's coefficient alphas indicated acceptable internal consistency. Moderate intercorrelations with two other retrospective measures of childhood experiences suggested construct validity. Moderate intercorrelations with two measures of psychological distress suggested predictive validity. In summary, initial psychometric testing of the DEQ suggests it is a reliable and valid tool for investigating an important developmental antecedent of adult psychological distress.     

Psychometric evaluation of the situation test of heterosocial skill

This study examined various psychometric properties of Forms A and B of the Situation Test, developed by Rehm and Marston (Journal of Consulting and Clinical Psychology, 1968, 37, 565–574) for the assessment of heterosocial skill and anxiety. A third test form composed of heterosocially irrelevant items was also examined for comparison purposes. Split-half, alternate-form, and interresponse consistency was determined for measures of skill, anxiety, response latency, and response duration. Differences across test forms on these measures were also investigated. Subsequently, criterion-related validity was examined relative to three criteria of heterosocial skill. Results indicated that two measures, anxiety and response duration, displayed adequate internal consistency, while that of skill and response latency was marginal. Interresponse consistency was moderately low for all three test forms. Comparisons of mean performances across forms revealed significant differences, with heterosocially irrelevant items appearing easier, in general, than the heterosocial items of Forms A and B. Lastly, significant predictions of peer-reported heterosocial behavior were obtained for all three test forms, but two self-report criteria were not found to be related to test behavior. Various implications of these findings are discussed.Portions of this research were completed while the authors were affiliated with the University of Georgia, Athens, Georgia. The advice and consultation provided by the late William K. Boardman is gratefully acknowledged. Appreciation is also extended to Michael Breakwell, Linda Maertzweller, Steven Ray, and Jan Rockley for their assistance with data collection.Copies of assessment material and specific instructions to subjects are available upon request.     

Validating the semantic misattribution procedure as an implicit measure of gender stereotyping

The current research tested the validity of the semantic misattribution procedure (SMP)—a variant of the affect misattribution procedure—as an implicit measure of gender stereotyping. In three studies (N = 604), prime words of gender‐stereotypical occupations (e.g., nurse, doctor) influenced participants' guesses of whether unknown Chinese ideographs referred to male or female names in a stereotype‐congruent manner. Priming scores of gender stereotyping showed high internal consistency and construct‐valid correlations with explicit measures of sexism. Discriminant validity of gender stereotyping scores was tested by investigating relations with priming effects involving grammatical gender (e.g., mother, father). Evidence for discriminant validity was obtained when (1) trials from the two priming measures were presented in a blocked rather than interspersed manner and (2) the measure of stereotypical gender priming preceded the measure of grammatical gender priming. Overall, the SMP showed good psychometric properties and construct validity for the assessment of gender stereotyping.     

The use of the peabody picture vocabulary test with emotionally disturbed adolescents

The validity and reliability for the Peabody Picture Vocabulary Test (PPVT) was estimated with 52 nonpsychotic emotionally disturbed adolescents. Approximately 6 months after an initial administration of the PPVT and the Peabody Individual Achievement Test (PIAT), all children were retested. Significant increases in PPVT IQ and PIAT reading comprehension scores were found upon readministration. The internal consistency estimate for the initial PPVT administration showed considerable consistency in responding. A modest temporal stability estimate indicated a fair amount of fluctuation in scores over time. Concurrent validity coefficients for the PPVT indicated moderate to dependable relationships with PIAT subtests. Predictive validity estimates for the PPVT suggested moderate relationships with future achievement, accounting for some 26% of the variability in total achievement.     

Attachment security,values, and prosocial attitudes

The objective of this study was to determine whether individuals with different attachment styles held different prosocial values and attitudes given their different models of self and of others. A sample of 717 university students completed the ECR ‐S (Experiences in Close Relationships‐Spanish) and the PVQ (Portrait Values Questionnaire), and evaluated different prosocial attitudes. The results showed that secure individuals reported higher scores on self‐transcendence and in the value self‐directions of the dimension openness to change , whereas insecure individuals had higher scores on conservation and self‐enhancement . No differences were found in the attitudes of individuals with different attachment styles. These findings show a link between the dimensions of the universal values theory, and the models of self and of others.     

Alternative Predictors for Dealing with the Diversity–Validity Dilemma in Personnel Selection: The constructed response multimedia test

In the context of the diversity–validity dilemma in personnel selection, the present field study compared ethnic subgroup differences on an innovative constructed response multimedia test to other commonly used selection instruments. Applicants (N = 245, 27% ethnic minorities) for entry‐level police jobs completed a constructed response multimedia test, cognitive ability test, language proficiency test, personality inventory, structured interview, and role play. Results demonstrated minor ethnic subgroup differences on constructed response multimedia test scores as compared to other instruments. Constructed response multimedia test scores were related to the selection decision, and no evidence for predictive bias was found. Subgroup differences were also examined on the dimensional level, with cognitively loaded dimension scores displaying larger differences.     

Cognitive ability in virtual reality: Validity evidence for VR game‐based assessments

Virtual reality (VR) technology may provide unique benefits for assessing individual characteristics, in contrast with traditional assessments. This study examined validity evidence for the use of VR game (VRG) scores to indicate specific cognitive abilities. Participants completed three VRGs, three computer‐based assessments (CBAs), and additional scales. To evaluate convergent validity, VRG scores were compared with CBA scores. For divergent validity, VRG scores were compared with five factors of personality based on the Five Factor Model. For criterion‐related validity, VRG scores were compared with academic achievement (i.e., grade point average). Findings generally provided support that the VRG scores are related to certain spatial reasoning subdimensions of cognitive ability. However, further research is needed to examine the qualities of VRGs as assessment tools.     

Effects of an expert-modeled attentional focus cue structure on skilled jump rope performance and learning

Instructions that direct attention externally have been shown to enhance motor performance. However, research on skilled performers and on learning effects has produced some distinct findings. Further, many studies have presented an overly simplistic view of attention and learning, such that all-internal focus protocols are contrasted with all-external focus protocols. Contrary to this approach, skilled performers have reported adopting combined focus strategies, revealing the need to test more realistic instructional protocols. The current study provided an experimental test of focus instructions that were modeled after the strategies of elite jump rope athletes. Four groups of skilled jump rope athletes practiced novel skills under various focus instructions. The internal focus (IF) and external focus (EF) groups were given traditional internal and external focus instructions, respectively. The expert modeled (EM) group was given a set of instructions that were based on experts' reported focus strategies. The expert modeled-autonomous (EM-A) group was allowed to choose how they used each of the expert-modeled instructions. All groups completed a baseline assessment, four practice sessions, and a learning assessment. Results of a chi-square test of independence revealed no relationship between group assignment and performance during baseline or practice. There was a significant relationship between group assignment and performance during the learning assessment (p < .05). Specifically, the IF group performed below expected values while the EM group performed better than expected. Findings support previous research showing learning detriments associated with internal focus instructions and also provide new insight into the advantages of using instructional approaches that are modeled after experts' strategies. Implications and suggestions for future research are discussed.