When “reliability” fails,must a measure be discarded?—The case of Kinesthetic Aftereffect

Critics of Kinesthetic Aftereffect (KAE) recommend abandoning it as a personality measure largely because of poor test-retest reliability. Although no test can be valid if lacking true reliability, to discard a measure because of poor retest reliability is an oversimplification of validation procedures. This pitfall is exemplified here by a reexamination of KAE. KAE scores involve measures before (pretest) and after (test) aftereffect induction. Internal analysis of a KAE study showed: Differential bias is present; its locus is the second session pretest; its form makes second-session pretest scores functionally more similar to first- and second-session test scores and functionally more dissimilar to first-session pretest scores. Given this second session bias, the retest correlation tells us nothing about the true reliability of a one-session KAE score. However, if a measure possesses external validity, it must to some degree show true reliability. Based upon a literature review of one-session KAE validity studies, we conclude that one-session KAE scores are valid and hence show true reliability. KAE remains a promising personality measure.     

Kinesthetic Aftereffect and Augmenting/Reducing: A Two-Session Procedure,and Hence Identification of Stimulus-Governed Subjects,is Contraindicated

A few of the recent researchers of kinesthetic aftereffect (KAE) as an index of augmenting/reducing have continued to employ a two-session procedure. Findings that have accrued in the past decade indicate (a) The first administration of KAE is a reliable (internally consistent) and valid index of augmenting-reducing; (b) there are carry-over effects from the first to the second administration that bias the second session's preinduction scores; (c) KAE scores from sessions after the first do not relate to first-session scores (low retest reliability) and do not measure augmenting/reducing; and (d) unless special procedures are undertaken to avoid using the biased second- (or later-) session preinduction scores, a KAE procedure involving more than one session is contraindicated. When we reached a similar conclusion earlier (Baker et al., 1974), Petrie (1974) disagreed, arguing that a two-session procedure was needed to identify and eliminate an atypical subgroup, the “stimulus governed.” The case for determining which subjects are stimulus-governed is assessed and found wanting. Except in special circumstances, a one-session KAE procedure, in which all preinduction trials precede the first exposure to aftereffect induction, is indicated.     

The Shedler-Westen assessment procedure (SWAP): evaluating psychometric questions about its reliability, validity, and impact of its fixed score distribution

The Shedler-Westen assessment procedure (SWAP) is a personality assessment instrument designed for use by expert clinical assessors. Critics have raised questions about its psychometrics, most notably its validity across observers and situations, the impact of its fixed score distribution on research findings, and its test-retest reliability. We review empirical data addressing its validity, emphasizing the multitrait-multimethod approach to evaluating test validity. To evaluate the hypothesis that the fixed, asymmetric score distribution artifactually inflates correlations between SWAP profiles, we conducted Monte Carlo simulations and also presented empirical data from a large patient sample. We observed a mean correlation of zero between simulated SWAP profiles, indicating that the score distribution does not impact the correlation coefficients. Empirical correlations between SWAP profiles of actual patients were small and similar to those obtained using Diagnostic and statistical manual of mental disorders, fourth edition (DSM-IV) personality disorder scales that had no fixed score distributions, suggesting that the correlations were not a methodological artifact of the SWAP. We report new test-retest reliability data (median coefficient > .85) for the SWAP's trait and personality disorder dimensions. The SWAP appears to be reliable and valid. The data do not support its primary psychometric critiques.     

The persuasiveness of racial arguments as a subtle measure of racism

These studies provide evidence of the reliability and validity of a new indirect measure of racism, the Racial Argument Scale (RAS). On the RAS, participants rate how well arguments support conclusions that are positive or negative toward Blacks rather than their agreement with the arguments and conclusions. These studies show that the RAS has good internal consistency, high levels of test-retest reliability, good convergent validity with other self-report measures of racism, and does not correlate with social desirability or right-wing authoritarianism. Furthermore, these studies show that the RAS predicts behavioral measures of racism and that the RAS is able to predict positivity and negativity toward Blacks that is not measured by other self-report measures of racism. These studies suggest that the RAS is a reliable and valid measure of racial attitudes.     

Validation of the Social Appearance Anxiety Scale: factor, convergent, and divergent validity

The Social Appearance Anxiety Scale (SAAS) was created to assess fear of overall appearance evaluation. Initial psychometric work indicated that the measure had a single-factor structure and exhibited excellent internal consistency, test-retest reliability, and convergent validity. In the current study, the authors further examined the factor, convergent, and divergent validity of the SAAS in two samples of undergraduates. In Study 1 (N = 323), the authors tested the factor structure, convergent, and divergent validity of the SAAS with measures of the Big Five personality traits, negative affect, fear of negative evaluation, and social interaction anxiety. In Study 2 (N = 118), participants completed a body evaluation that included measurements of height, weight, and body fat content. The SAAS exhibited excellent convergent and divergent validity with self-report measures (i.e., self-esteem, trait anxiety, ethnic identity, and sympathy), predicted state anxiety experienced during the body evaluation, and predicted body fat content. In both studies, results confirmed a single-factor structure as the best fit to the data. These results lend additional support for the use of the SAAS as a valid measure of social appearance anxiety.     

Psychometric properties of a European Spanish version of the Perceived Stress Scale (PSS)

This paper presents evidence from a heterogeneous sample of 440 Spanish adults, for the reliability and validity of a European Spanish version of the Perceived Stress Scale (PSS), designed to measure the degree to which situations in one's life are appraised as stressful. The European Spanish version PSS (14-item) demonstrated adequate reliability (internal consistency, alpha = .81, and test-retest, r = .73), validity (concurrent), and sensitivity. Additional data indicate adequate reliability (alpha = .82, test-retest, r = .77), validity, and sensitivity of a 10-item short version of the PSS.     

Reliability and validity of scores from the Singer-Loomis Type Deployment Inventory

This paper focuses on the Singer-Loomis Type Deployment Inventory (SL-TDI), which is an alternative to the Myers-Briggs Type Indicator (MBTI). The SL-TDI utilizes a continuous, non-forced choice format and therefore is a more accurate representation of Jung's personality theory of psychological types. The purpose of the study reported here is to evaluate the reliability and validity of scores from the SL-TDI. Specifically, the goals were to (a) provide estimates of the internal consistency of SL-TDI scores; (b) evaluate the divergent validity of SL-TDI scores by examination of their relationships with the scores on a social desirability responding measure; and (c) examine the test-retest stability of scores from the SL-TDI. Strong support was found for both the reliability and validity of SL-TDI scores.     

Validation of the WHOQOL-BREF in a Sample of Male Treatment-Seeking Veterans

We examined the psychometric properties of a quality of life measure, the WHOQOL-BREF, in male treatment-seeking veterans. A sample of 131 veterans at an outpatient clinic completed a mail-in survey that included the WHOQOL-BREF as well as measures of posttraumatic stress, depression, anxiety symptoms, and functional health status. The WHOQOL-BREF showed evidence of good internal consistency and strong test-retest reliability. In addition, analysis of results demonstrated evidence of construct, convergent, and discriminant validity. The results suggest that the WHOQOL-BREF is a reliable and valid measure that can be used in the planning of psychological services provided to veterans with mental health problems.     

On the validity of narrow and broad personality traits: A response to

Salgado, Moscoso, and Berges (2013) have recently commented on the issue of the comparative validity of broad factor-level personality traits and narrow facet-level personality traits, providing arguments and data suggesting superior validity for the former. In response, we first clarify some of our previous points that were discussed by Salgado et al. and comment on some other aspects of their article. We then provide an empirical example in which a facet-level personality scale does better in predicting a delinquency criterion than does the factor-level personality scale that contains that facet. We also use CFA to demonstrate that it was the specific trait variance in our narrow predictor that added significantly to predictive validity, above and beyond the validity achieved by the common factor variance contained within the trait measure. These results support our previous conclusion that exclusive reliance on broad factor measures can be counterproductive for understanding and predicting behavior.     

Two promising shame and guilt scales: a construct validity comparison

This study compared the validity of two promising measures of shame and guilt proneness: revisions of the Harder Personal Feelings Questionnaire (PFQ2; Harder & Lewis, 1987) and the Hoblitzelle Adapted Shame and Guilt Scale (ASGS; Hoblitzelle, 1982). Internal consistency, test-retest stability, factor structure, and construct validity with convergent and discriminant personality dimensions were examined for both scales. In addition to the shame and guilt measures, 63 (37 male, 26 female) mostly freshman college students completed a randomly ordered battery of personality scales theoretically relevant to shame and guilt proneness. Results support the reliability and shame/guilt factor structure of each scale. ASGS Shame correlations appeared marginally more valid with 11 external construct variables than PFQ2 Shame, whereas PFQ2 Guilt was clearly more valid than its corresponding ASGS subscale. New, potentially improved scales were constructed from the factor analyses and from item analyses. However, the resulting scales did not show improved validity.     

The displaced aggression questionnaire

Previous measures of aggressive personality have focused on direct aggression (i.e., retaliation toward the provoking agent). An original self-report measure of trait displaced aggression is presented. Exploratory and confirmatory factor analyses provided support for a 3-factor conceptualization of the construct. These analyses identified an affective dimension (angry rumination), a cognitive dimension (revenge planning), and a behavioral dimension (general tendency to engage in displaced aggression). The trait measure demonstrated good internal consistency and test-retest reliability as well as convergent and discriminant construct validity. Unlike other related personality measures, trait displaced aggression significantly predicted indirect indicators of real-world displaced aggression (i.e., self-reported domestic abuse and road rage) as well as laboratory displaced aggression in 2 experiments.     

Investigation of the psychometric properties of the Perceived Parental Autonomy Support Scale in the Southeastern European context

The Perceived Parental Autonomy Support Scale (P-PASS) is an instrument designed to measure parental autonomy-support and control of late adolescents and emerging adults. The present paper examines the process of adapting the P-PASS to the Romanian culture. Four studies were conducted, investigating: 1) the adequacy of the translation, using a multidimensional scaling of expert ratings; 2) construct validity, through exploratory approaches; 3) various psychometric properties, such as reliability and construct validity, through confirmatory approaches; convergent validity through comparisons with other measures of parental autonomy support (College-Student Scale of the Perceptions of Parents Scales) and control (Psychological Control Scale–Youth Self-Report), and predictive validity in relation with general self-efficacy; 4) test-retest reliability. The results show that the Romanian version of the P-PASS has sound psychometric properties. Confirmatory Factor Analysis indicates that a structure with two second order factors fits the data best and that the measure is equivalent with the original Canadian version. Also, it shows adequate test-retest reliability at 6 months and one year between administrations, good convergent validity, and a good prediction of general self-efficacy.

     

The Development and Validation of the Multicomponent AIDS Phobia Scale

The development, reliability, and validity of a new instrument, the Multicomponent AIDS Phobia Scale (MAPS), is described. Based on a sample of 181 university students, the results indicate that the MAPS has strong internal consistency and moderate test-retest reliability. In addition, an assessment of concurrent and discriminant validity indicates statistically significant correlations with other self-report measures of anxiety, hypochondriasis, obsessive-compulsive disorder, and fear of AIDS. The results of a factor analysis indicate that the scale consists of two factors: Fear of Infection and Fear of Others/Avoidance. Although further research with the MAPS in a clinical population is warranted to investigate the generalizability of this study's results, the instrument appears to be a reliable and valid measure of AIDS Phobia and may prove useful for improving clinical assessment and documenting treatment outcome.     

Development and validation of the Delaying Gratification Inventory

Deficits in gratification delay are associated with a broad range of public health problems, such as obesity, risky sexual behavior, and substance abuse. However, 6 decades of research on the construct has progressed less quickly than might be hoped, largely because of measurement issues. Although past research has implicated 5 domains of delay behavior, involving food, physical pleasures, social interactions, money, and achievement, no published measure to date has tapped all 5 components of the content domain. Existing measures have been criticized for limitations related to efficiency, reliability, and construct validity. Using an innovative Internet-mediated approach to survey construction, we developed the 35-item 5-factor Delaying Gratification Inventory (DGI). Evidence from 4 studies and a large, diverse sample of respondents (N = 10,741) provided support for the psychometric properties of the measure. Specifically, scores on the DGI demonstrated strong internal consistency and test-retest reliability for the 35-item composite, each of the 5 domains, and a 10-item short form. The 5-factor structure fit the data well and had good measurement invariance across subgroups. Construct validity was supported by correlations with scores on closely related self-control measures, behavioral ratings, Big Five personality trait measures, and measures of adjustment and psychopathology, including those on the Minnesota Multiphasic Personality Inventory-2-Restructured Form. DGI scores also showed incremental validity in accounting for well-being and health-related variables. The present investigation holds implications for improving public health, accelerating future research on gratification delay, and facilitating survey construction research more generally by demonstrating the suitability of an Internet-mediated strategy.     

An evaluation of the Quay-Peterson revised behavior problem checklist

The RBPC is perhaps the most valid of all currently available teacher rating scales in terms of factor structure and related measures of validity. Clearly, however, additional research on the concurrent and predictive validity of this instrument is needed. The RBPC is adequately reliable in terms of test-retest reliability and interrater reliability when comparing two teachers and two parents. The lack of representative national norms creates a serious problem for the use of the RBPC by school psychologists, but some useful normative information is available and it would be a relatively easy task to develop local norms.     

Need for Stimulation as a Factor in Preferences for Different Stimuli

Recent investigations have indicated that “reduction” on the kinesthetic aftereffects (KAE) procedure may indicate high levels of a putative need for stimulation (nStim), while “augmentation” on this procedure may indicate low levels of this motive. The present study supports the interpretation of the KAE task as a measure of nStim by indicating that “reducers” on this measure (a) react favorably to complex auditory and visual stimuli and (b) expose themselves particularly frequently to complex and intense stimulation in their ongoing social activities.     

Psychometric evaluation of the parent situation inventory: a role-play measure of coping in parents of substance-using adolescents

This article reports on the generalizability, reliability, and construct validity of the Parent Situation Inventory (PSI), a role-play measure of coping skills in parents experiencing problems from an adolescent's drug and alcohol use. Generalizability was robust (.80) and alternate form and test-retest reliability were satisfactory. PSI skillfulness was negatively related to the parent's own substance use and to the adolescent's alcohol use. The PSI shows promise as a reliable and potentially valid measure of coping in this population and has direct implications for developing and evaluating skill-based parent training programs.     

Reliability and validity of the Spanish Version of the Crisis in Family Systems-Revised

Increasing the representation of Spanish-speaking study participants requires development and dissemination of reliable and valid translated scales. In the urrent study the construct validity was assessed of the Spanish version of the Crisis n Family Systems-Revised, a measure of contemporary life stressors, with a convenience sample of 377 parents interviewed in a study of childhood asthma, although over half of the respondents did not have children with asthma. Most respondents were foreign-born women between 20 to 60 years old (M=35, SD=7). 52% had not completed high school or its equivalent, and 55% reported a household income of dollar 15,000 or less. For a subsample of 25 respondents test-retest reliability was .86 over 2 wk. Reporting more life stressors was associated with greater depressive symptomatology, poorer physical and mental health function, and lower household income. These relationships support the construct validity of the test in Spanish. This study provided strong evidence that this version is a valid and reliable measure of life stressors for a Spanish-speaking population living in the United States.     

Psychometric properties of the Revised Cheek and Buss Shyness Scale

Although the Revised Cheek and Buss Shyness Scale (RCBS; Cheek, 1983) is widely used, its psychometric properties largely are unknown. In this investigation, we examined the normative data, factor structure, internal consistency, test-retest reliability, and convergent/discriminant validity of the RCBS using a sample of 261 university students. Results provided strong support for the stability of normative data over time, reliability of the measure, and its predicted associations with contemporary measures of shyness, social anxiety, and related constructs. Although support was obtained for a unifactorial conceptualization of shyness, an exploratory factor analysis revealed an alternative 3-factor solution that was supportive of a previously proposed meta-analytic model of shyness (Jones, Briggs, & Smith, 1986) and was consistent with other prominent shyness theories (Buss, 1980; Pilkonis, 1977a, 1977b; Zimbardo, 1977). This factor model was replicable on a holdout sample, and there were some data to support the discriminant validity of factors.     

The assessment of social intimacy

Research exploring the psychological importance of closeness with others has been hampered by the absence of a reliable and valid measure of this variable. The development of the Miller Social Intimacy Scale (MSIS), a 17-item measure of the maximum level of intimacy currently experienced, is presented. Evidence for internal consistency and test-retest reliability as well as for convergent, discriminant and construct validity is discussed in the context of the need for further scientific exploration of this important phenomenon.