Comparing Personality Test Formats and Warnings: Effects on criterion‐related validity and test‐taker reactions

The potential for applicant response distortion on personality measures remains a major concern in high‐stakes testing situations. Many approaches to understanding response distortion are too transparent (e.g., instructed faking studies) – or are too subtle (e.g., correlations with social desirability measures as indices of faking). Recent research reveals more promising approaches in two methods: using forced‐choice (FC) personality test items and warning against faking. The present study examined effects of these two methods on criterion‐related validity and test‐taker reactions. Results supported incremental validity for an FC and Likert‐scale measure in warning and no‐warning conditions, above and beyond cognitive ability. No clear differences emerged between the FC vs Likert measures or warning vs no‐warning conditions in terms of validity. However, some evidence suggested that FC measures and warnings may produce negative test‐taker reactions. We conclude with implications for implementation in selection settings.     

INTERRATER CORRELATIONS DO NOT ESTIMATE THE RELIABILITY OF JOB PERFORMANCE RATINGS

Interrater correlations are widely interpreted as estimates of the reliability of supervisory performance ratings, and are frequently used to correct the correlations between ratings and other measures (e.g., test scores) for attenuation. These interrater correlations do provide some useful information, but they are not reliability coefficients. There is clear evidence of systematic rater effects in performance appraisal, and variance associated with raters is not a source of random measurement error. We use generalizability theory to show why rater variance is not properly interpreted as measurement error, and show how such systematic rater effects can influence both reliability estimates and validity coefficients. We show conditions under which interrater correlations can either overestimate or underestimate reliability coefficients, and discuss reasons other than random measurement error for low interrater correlations.     

Implicit measures of association in psychopathology research

Studies obtaining implicit measures of associations in Diagnostic and Statistical Manual of Mental Disorders (4th ed., Text Revision; American Psychiatric Association, 2000) Axis I psychopathology are organized into three categories: (a) studies comparing groups having a disorder with controls, (b) experimental validity studies, and (c) incremental and predictive validity studies. In the first category, implicit measures of disorder-relevant associations were consistent with explicit beliefs for some disorders (e.g., specific phobia), but for other disorders evidence was either mixed (e.g., panic disorder) or inconsistent with explicit beliefs (e.g., pain disorder). For substance use disorders and overeating, expected positive and unexpected negative associations with craved substances were found consistently. Contrary to expectation, implicit measures of self-esteem were consistently positive for patients with depressive disorder, social phobia, and body dysmorphic disorder. In the second category, short-term manipulations of disorder-relevant states generally affected implicit measures as expected. Therapeutic interventions affected implicit measures for one type of specific phobia, social phobia, and panic disorder, but not for alcohol use disorders or obesity. In the third category, implicit measures had predictive value for certain psychopathological behaviors, sometimes moderated by the availability of cognitive resources (e.g., for alcohol and food, only when cognitive resources were limited). The strengths of implicit measures include (a) converging evidence for dysfunctional beliefs regarding certain disorders and consistent new insights for other disorders and (b) prediction of some psychopathological behaviors that explicit measures cannot explain. Weaknesses include (a) that findings were inconsistent for some disorders, raising doubts about the validity of the measures, and (b) that understanding of the concept "implicit" is incomplete.     

Measuring everyday adaptation after imprisonment: The post-release living inventory for ex-prisoners (PORLI-ex)

BackgroundThe number of ex-prisoners worldwide has constantly been increasing in recent years. Currently, little is known about post-release daily adaptation, not to mention valid and reliable instruments for post-release daily routines pertinent to mental health.ObjectiveThis study aims to develop and validate a self-report instrument, hereafter referred to as Post Release Living Inventory for Ex-prisoners (PORLI-ex).MethodsThree separate samples of ex-prisoners were recruited to complete an online survey (N=1,277, age range=17–89 years, 53.2% male, 72% white).ResultsThe final model evidenced acceptable goodness-of-fit and consisted of 45 items on nine dimensions, which loaded on three second-order factors: Consolidation (three dimensions; e.g., Institutional Routines), Replacement (two dimensions; e.g., Maladaptive Behaviors), and Addition (four dimensions; e.g., Socializing with Ex-prisoner Friends) (α=.695–.915). Convergent validity was demonstrated in the positive correlations with IADL, SOLI, MLQ, GSE-6, and MSPSS. Discriminant validity was demonstrated in the weak correlations with the LEC-5 and perceived social and personal cost of punishment. Criterion-related validity was demonstrated in the correlations with psychiatric symptoms and crime-related outcomes and incremental validity in the correlations with these measures independent of the scores on IADL, SOLI, MLQ, GSE-6, and MSPSS.ConclusionThis study calls for more resources on fostering psychological strengths and resilience through regularizing basic daily life experiences on top of traditional interventions for risk management among the ex-prisoners.     

The Physical Appearance Perfectionism Scale: Development and Preliminary Validation

Eight studies with data from 2316 students are presented describing the development and preliminary validation of the Physical Appearance Perfectionism Scale (PAPS), a brief measure with two subscales: Worry About Imperfection and Hope For Perfection. Results from exploratory and confirmatory factor analyses confirmed the measure’s two-dimensional structure. Moreover, correlation analyses provided first evidence for the two subscales’ differential validity: Worry About Imperfection showed negative correlations with positive self-perceptions of one’s appearance (e.g., appearance self-esteem) and positive correlations with maladaptive concerns aspects of perfectionism, physical appearance concerns (e.g., body image disturbances), and body weight control whereas Hope For Perfection showed positive correlations with positive striving aspects of perfectionism, positive self-perceptions, and impression management. In addition, all PAPS scores showed high reliability (Cronbach’s alpha) and temporal stability (test-retest). Overall the findings suggest that the PAPS is a reliable and valid instrument to assess positive and negative aspects of physical appearance perfectionism.     

How does higher in‐group social value lead to positive mental health? An integrated model of in‐group identification and support

Previous research has indicated that the effects of in‐group social value on mental health are mediated by the extent to which individuals identify with their in‐group. Other research has shown that in‐group identification leads to positive mental health because it provides in‐group members with a psychological basis for social support. We examine how the individual's perception of the social value of the in‐group leads to positive mental health, integrating the effects of identification with and support from the in‐group. As predicted, the relationship between higher social value and decreased mental health (e.g. depression, perceived stressful events) is mediated by higher in‐group identification, which in turn leads the members to expect support from the in‐group but not the out‐group. An integrated model of the effects of perceived in‐group social value, identification, and support on mental health is proposed, and the implications are discussed.     

Validating the semantic misattribution procedure as an implicit measure of gender stereotyping

The current research tested the validity of the semantic misattribution procedure (SMP)—a variant of the affect misattribution procedure—as an implicit measure of gender stereotyping. In three studies (N = 604), prime words of gender‐stereotypical occupations (e.g., nurse, doctor) influenced participants' guesses of whether unknown Chinese ideographs referred to male or female names in a stereotype‐congruent manner. Priming scores of gender stereotyping showed high internal consistency and construct‐valid correlations with explicit measures of sexism. Discriminant validity of gender stereotyping scores was tested by investigating relations with priming effects involving grammatical gender (e.g., mother, father). Evidence for discriminant validity was obtained when (1) trials from the two priming measures were presented in a blocked rather than interspersed manner and (2) the measure of stereotypical gender priming preceded the measure of grammatical gender priming. Overall, the SMP showed good psychometric properties and construct validity for the assessment of gender stereotyping.     

Assessing Religious Faith in Medical Patients: Cross-Validation of the Santa Clara Strength of Religious Faith Questionnaire

Growing attention has focused on the relationship between religiosity and health outcomes. However, research has been constrained by the limited availability of measures suitable for use with medical patients. This study examined the psychometric properties of the Santa Clara Strength of Religious Faith Questionnaire (SCSORF) in two well-defined samples of medical patients, representing a range of illness severity: 1) 175 women seen at a gynecology clinic; and 2) 104 cancer patients treated at a bone marrow transplantation program. Evidence for convergent validity was demonstrated by moderately high correlations with other measures of religious involvement. Divergent validity was demonstrated by small associations with social desirability. The measure was not strongly associated with demographic background or medical variables. Modest correlations with conceptually-related measures commonly used in health research (e.g., social support, optimism, existential meaning, purpose in life) further supported the divergent validity of the instrument. Results suggest that this may be a useful, concise measure for assessing religious involvement in medical patients.     

Assessment of methods for measuring social support

A rapidly expanding literature documents the effects of social support on physical and psychological well-being. Although definitions vary, most include both tangible components (e.g., financial assistance and physical aid) and intangible components (e.g., encouragement and guidance). Social support has been implicated in the mediation of stressful life events, recovery from illness, and increased program adherence. There are many inconsistent findings in the literature, however, and it is difficult to resolve discrepancies because measures of social support vary widely from study to study. To guide in the selection of measurement methods for research and applied work, 23 techniques for assessing social support are reviewed and evaluated. Criteria for favorable evaluation included reliability coefficients greater than .8 and documentation of validity. Correlations between various social support and criterion measures are simulated in order to demonstrate the consequences of choosing a measure with low reliability. Scale developers reported reliability data for 19 of the reviewed measures. Internal consistency coefficients ranged from .31 to .98. Test-retest coefficients ranged from .22 to .96. At least some validity documentation was available for 13 of the scales. Discriminant validity evidence, however, is almost universally absent. Despite psychometric weaknesses and variability among the scales, researchers have several instruments available to them.     

Comparing continuous and dichotomous scoring of the balanced inventory of desirable responding

The Balanced Inventory of Desirable Responding (BIDR; Paulhus, 1994) is a widely used instrument to measure the 2 components of social desirability: self-deceptive enhancement and impression management. With respect to scoring of the BIDR, Paulhus (1994) authorized 2 methods, namely continuous scoring (all answers on the continuous answer scale are counted) and dichotomous scoring (only extreme answers are counted). In this article, we report 3 studies with student samples, and continuous and dichotomous scoring of BIDR subscales are compared with respect to reliability, convergent validity, sensitivity to instructional variations, and correlations with personality. Across studies, the scores from continuous scoring (continuous scores) showed higher Cronbach's alphas than those from dichotomous scoring (dichotomous scores). Moreover, continuous scores showed higher convergent correlations with other measures of social desirability and more consistent effects with self-presentation instructions (fake-good vs. fake-bad instructions). Finally, continuous self-deceptive enhancement scores showed higher correlations with those traits of the Five-factor model for which substantial correlations were expected (i.e., Neuroticism, Extraversion, and Conscientiousness). Consequently, these findings indicate that continuous scoring may be preferable to dichotomous scoring when assessing socially desirable responding with the BIDR.     

Testing the basic socio‐structural assumptions of social identity theory in the gender context: Evidence from correlational studies on women's leadership

Given negative social identity, different perceptions of the structure of an intergroup relation (i.e., stability, legitimacy, permeability) should be related to different identity‐management strategies (i.e., social competition, social creativity, or individual mobility) depending on group identification. This is among the basic tenets of social identity theory (SIT). There is surprisingly little empirical support for these postulates in the context of one of the most central group identities: gender. Using a sample of women in leadership positions in Spain (N = 649), we tested relations between structural perceptions and identity‐management strategies in a pilot study. Structural equation modeling yielded empirical support regarding social competition, but little for social creativity or individual mobility. Identity‐management strategies were related to one organizational outcome (i.e., identification with the organization). The preregistered main study is intended to replicate and extend these findings using a different sample while improving several of the measures used.     

Managing one's social identity: Successful and unsuccessful identity management

This work explores the motivational dynamics of social identity management. Following social identity theory, we hypothesized that a threat to a positive social identity elicits specific negative emotions (i.e., outgroup‐directed anger) and motivates identity management. Successful identity management restores a positive social identity and decreases outgroup‐directed anger. However, when a successful identity management is blocked (e.g., because of limited cognitive resources), identity management will be unsuccessful and outgroup‐directed anger will remain at a higher level. This effect of unsuccessful identity management on outgroup‐directed anger should be particularly strong for group members who highly value their group (i.e., high group‐based self‐esteem). A negative comparison outcome is discrepant with these group members' positive view of the ingroup, and therefore, unsuccessful identity management should especially elicit negative emotions (i.e., anger) towards the threatening outgroup. Two studies tested these predictions. Study 1 (N = 110) showed that participants' outgroup‐directed anger increased when threatened under cognitive load. Study 2 (N = 99) demonstrated that this was particularly true for participants high in group‐based self‐esteem. The results' implications for research on the motivational processes underlying social identity management are discussed. Copyright © 2014 John Wiley & Sons, Ltd.     

Reliability measures in item response theory: Manifest versus latent correlation functions

For item response theory (IRT) models, which belong to the class of generalized linear or non‐linear mixed models, reliability at the scale of observed scores (i.e., manifest correlation) is more difficult to calculate than latent correlation based reliability, but usually of greater scientific interest. This is not least because it cannot be calculated explicitly when the logit link is used in conjunction with normal random effects. As such, approximations such as Fisher's information coefficient, Cronbach's α, or the latent correlation are calculated, allegedly because it is easy to do so. Cronbach's α has well‐known and serious drawbacks, Fisher's information is not meaningful under certain circumstances, and there is an important but often overlooked difference between latent and manifest correlations. Here, manifest correlation refers to correlation between observed scores, while latent correlation refers to correlation between scores at the latent (e.g., logit or probit) scale. Thus, using one in place of the other can lead to erroneous conclusions. Taylor series based reliability measures, which are based on manifest correlation functions, are derived and a careful comparison of reliability measures based on latent correlations, Fisher's information, and exact reliability is carried out. The latent correlations are virtually always considerably higher than their manifest counterparts, Fisher's information measure shows no coherent behaviour (it is even negative in some cases), while the newly introduced Taylor series based approximations reflect the exact reliability very closely. Comparisons among the various types of correlations, for various IRT models, are made using algebraic expressions, Monte Carlo simulations, and data analysis. Given the light computational burden and the performance of Taylor series based reliability measures, their use is recommended.     

The fading affect bias shows positive outcomes at the general but not the individual level of analysis in the context of social media

Unpleasant affect fades faster than pleasant affect (e.g., Walker, Vogl, & Thompson, 1997); this effect is referred to as the Fading Affect Bias (FAB; Walker, Skowronski, Gibbons, Vogl, & Thompson, 2003a). Research shows that the FAB is consistently related to positive/healthy outcomes at a general but not at a specific level of analysis based on event types and individual differences (e.g., Gibbons et al., 2013). Based on the positive outcomes for FAB and negative outcomes for social media (Bolton et al., 2013; Huang, 2010), the current study examined FAB in the context of social media events along with related individual differences. General positive outcomes were shown in the form of robust FAB effects across social media and non-social media events, a larger FAB for non-social media events than for social media events, negative correlations of FAB with depression, anxiety, and stress as well as a positive correlation of FAB with self-esteem. However, the lack of a negative correlation between FAB and anxiety for social media events in a 3-way interaction did not show positive outcomes at a specific level of analysis. Rehearsal ratings mediated the 3-way interaction. Implications are discussed.     

Linking self and ingroup: Self‐anchoring as distinctive cognitive route to social identification

The present paper investigates how cognitive projection processes instigate social identification. We complement the classical self‐stereotyping approach (i.e., conforming to prototypical group norms) by investigating self‐anchoring (i.e., projection from self to group) as a distinct cognitive route to social identification. Self‐anchoring has mainly been investigated as predictor of intergroup differentiation. Surprisingly, no reliable link has been provided yet between self‐anchoring and social identification. In Study 1, we provide first evidence for this positive link. In Study 2, we add self‐stereotyping to our model and show that self‐anchoring is still positively related to social identification when controlling for self‐stereotyping. Additionally, we show that self‐anchoring is positively related to affective components of identification, while self‐stereotyping is positively related to cognitive components. Moreover, we examined the impact of self‐concept stability on self‐anchoring. Self‐concept stability was positively related to self‐anchoring, and hence to social identification (Study 1), independently from self‐stereotyping (Study 2). In the discussion, we argue that disentangling self‐anchoring from self‐stereotyping is important as it increases our insight in how people identify, and how this may vary depending on self‐concept and group context. Copyright © 2011 John Wiley & Sons, Ltd.     

Social identification effects in social dilemmas: a transformation of motives

Three experimental studies were conducted to examine two alternative explanations for the widely established positive effect of social identification in promoting cooperation in social dilemmas. We hypothesised that social identification effects could be either ascribed to (1) an increase in the value assigned to the collective good (i.e. goal‐transformation hypothesis) or (2) an enhancement of trust in the cooperation of other group members (i.e. goal‐amplification hypothesis). To disentangle these two explanations, we examined the effects of social identification on the contributions to a public good of people with a different social value orientation (i.e. pre‐existing differences in preferred outcome distribution between self and others). Following the goal transformation hypothesis, we predicted that an increased group identification would raise contributions, in particular for people essentially concerned with their personal welfare (i.e. pro‐self value orientation). Alternatively, following the goal amplification hypothesis it was expected that increased group identification would primarily affect decisions of people concerned with the collective welfare (i.e. prosocial value orientation). The results of all three studies provided support for the goal‐transformation rather than goal‐amplification hypothesis, suggesting that ‘selfish’ individuals can be encouraged to cooperate by increasing the salience of their group membership. Copyright © 1999 John Wiley & Sons, Ltd.     

Validation of a Japanese Version of the Work Engagement Scale for Students

We sought to verify the reliability and validity of the Japanese version of the Utrecht Work Engagement Scale for Students (UWES‐S‐J). We examined 824 university students. We calculated the goodness of fit for a single‐factor model and the three‐factor model. The fit to the data was better for the three‐factor than for the single‐factor model, but all factors were highly positively correlated. Additionally, the UWES‐S‐J had good internal consistency and test–retest reliability. For the content validity, there were significant positive correlations between the UWES‐S‐J score and social support, a resilience scale, and subjective happiness. The UWES‐S‐J has good reliability and validity and may therefore be used to assess study engagement among Japanese students.     

A Study of the Psychometric Properties of the Social Support Scale for Children

The Social Support Scale for Children (SSSC; Harter, 1985a) is one of the most commonly used measures of perceived social support and positive regard from others in children and young adolescents. Research on the validity of the SSSC, however, is limited. The current study examined the test-retest reliability, internal validity intersubscale correlations, and convergent validity of the SSSC. A confirmatory factor analysis was performed using structural equation modeling. The participants included 279 youth between the ages of 6 to 15 from three metropolitan areas in the Midwest. The results supported the reliability and validity of the SSSC. The results supported the four factor solution reported by Harter (1985a). Rewording or removal of two items is suggested.     

Development and preliminary validation of a mental toughness inventory for Australian football

ObjectivesThis paper describes the development and preliminary evaluation of the Australian football Mental Toughness Inventory (AfMTI).MethodsConfirmatory and exploratory factor analyses were employed to explore the factor structure of a pool of items designed to capture the key components of mental toughness in Australian football [Gucciardi, D.F., Gordon, S., & Dimmock, J.A. (2008). Towards an understanding of mental toughness in Australian football. Journal of Applied Sport Psychology, 20, 261–281.] Correlations between the four-factor inventory and flow, resilience, and social desirability were examined. The discriminant validity of the inventory was also assessed. Multisource ratings (self, parent, and coach) of the AfMTI were examined in experiment two.ResultsThe AfMTI is a 24-item scale that measures four components of mental toughness in Australian football – thrive through challenge, sport awareness, tough attitude, and desire success. It was shown to have adequate internal reliability estimates across different raters (α = .70–.89). Moderate correlations with flow and resilience were evidenced, while minimal correlations existed with social desirability. Multisource data were somewhat equivocal; correlational data suggested a disagreement between raters, whereas an ANOVA suggested agreement between raters.ConclusionsPreliminary data on the factor structure, internal reliability, and construct validity of the AfMTI were encouraging. However, the factor structure, reliability, and validity of the AfMTI must be verified through further psychometric examinations before it can be considered a useful tool for measuring mental toughness in Australian football.     

Psychometric properties and clinical cut-off scores of the Spanish version of the Social Anxiety Scale for Adolescents

This study examined the reliability and validity evidence drawn from the scores of the Spanish version of the Slovenian-developed Social Anxiety Scale for Adolescents (SASA; Puklek, 1997; Puklek & Vidmar, 2000) using a community sample (Study 1) and a clinical sample (Study 2). Confirmatory factor analysis in Study 1 replicated the 2-factor structure found by the original authors in a sample of Slovenian adolescents. Test-retest reliability was adequate. Furthermore, the SASA correlated significantly with other social anxiety scales, supporting concurrent validity evidence in Spanish adolescents. The results of Study 2 confirmed the correlations between the SASA and other social anxiety measures in a clinical sample. In addition, findings revealed that the SASA can effectively discriminate between adolescents with a clinical diagnosis of social anxiety disorder (SAD) and those without this disorder. Finally, cut-off scores for the SASA are provided for Spanish adolescents.