Response styles in affect ratings: making a mountain out of a molehill

Ratings of affect words are the most commonly used method to assess pleasant affect (PA) and unpleasant affect (UA). The reliance on self-reports would be problematic if affect ratings were heavily influenced by response styles. Several recent publications have indeed suggested (a) that the influence of response styles on affect ratings is pervasive, (b) that this influence can be controlled by variations of the response format using multitrait-multimethod models, and (c) the discriminant validity of PA and UA is spurious. In this article, we examined the evidence for these claims. We demonstrate that (a) response styles have a negligible effect on affect ratings, (b) multiple response formats produce the same results as a single response format, and (c) the discriminant validity of PA and UA is not a method artifact. Rather, evidence against discriminant validity is due to the use of inappropriate response formats that respondents interpreted as bipolar scales.     

Culture,gender, and the bipolarity of momentary affect: A critical re-examination

This paper re-examines Yik's (2007) hypothesis that dialectic thinking moderates aggregated but not momentary ratings of affect. A reanalysis of Yik's (2007) correlation coefficients shows that culture is a significant moderator of the correlation between ratings of positive affect (PA) and negative affect (NA) for two of the three response formats, and for the latent correlations reported in the original article. The effect size of this difference is similar to the effect size in previous publications with aggregated measures of affect. It is concluded that Asian dialectic thinking styles moderate the correlation of both aggregated and momentary reports of PA and NA.     

A Comparison of the Nomological Networks Associated With Forced-Choice and Likert Formats of the Narcissistic Personality Inventory

The Narcissistic Personality Inventory (NPI) is one of the most popular measures of narcissism. However, its use of a forced-choice response set might negatively affect some of its psychometric properties. The purpose of this research was to compare a Likert version of the NPI, in which only the narcissistic response of each pair was given, to the original NPI, in 3 samples of participants (N = 1,109). To this end, we compared the nomological networks of the forced-choice and Likert formats of the NPI in relation to alternative measures of narcissism, narcissistic personality disorder, entitlement, self-esteem, general personality traits (reported by self and informants), interpersonal styles, and general pathological traits included in the DSM–5. The Likert format NPI—total and subscales—manifested similar construct validity to the original forced-choice format across all criteria with only minor differences that seem to be due mainly to the increased reliability and variability found in the Likert NPI Entitlement/Exploitativeness subscale. These results provide evidence that a version of the NPI that employs a Likert format can justifiably be used in place of the original.     

Measurement and Utility of Continuous Unipolar Ratings for the Myers-Briggs Type Indicator

This study is designed to explore the measurement properties and utilities of the Myers-Briggs Type indicator (MBTI) items when the two competing alternatives of individual items were rated separately on a 7-step, free-response continuum. Data were collected from 125 college students who volunteered for this study. Ratings were also obtained from subjects on occupational preferences reported to be leading occupational choices for the 16 MBTI personality types. Major analytic results include the findings of: (a) high factorial validity of unipolar ratings in separate analyses of the two alternatives of MBTI items, (b) high reliability coefficients for all eight preference poles and high convergent and discriminant validities for each of the four MBTI scales, and (c) four significant factors emerging from the combined factor analysis of subjects scores on occupational preferences and MBTI personality poles. Finally, the empirical evidence was discussed about the linkage between Jung theoretical constructs of personality typologies and the MBTI scales. The possibility of just using unipolar ratings for MBTI items in a continuous-response format was also suggested.     

A new frame for frame-of-reference training: enhancing the construct validity of assessment centers

The authors undertook a comprehensive examination of the construct validity of an assessment center in this study by (a) gathering many different types of evidence to evaluate the strength of the inference between predictor measures and constructs (e.g., reliability, accuracy, convergent and discriminant relationships), (b) introducing a theoretically relevant intervention (frame-of-reference [FOR] training) aimed at improving construct validity, and (c) examining the effect of this intervention on criterion-related validity (something heretofore unexamined in the assessment center literature). Results from 58 assessees and 122 assessors suggested that FOR training was effective at improving the reliability, accuracy, convergent and discriminant validity, and criterion-related validity of assessment center ratings. Findings are discussed in terms of implications and future directions for both FOR training and assessment center practice.     

Psychometric properties of the mobility inventory for agoraphobia: convergent, discriminant, and criterion-related validity

Aims of this study were (a) to summarize the psychometric literature on the Mobility Inventory for Agoraphobia (MIA), (b) to examine the convergent and discriminant validity of the MIA's Avoidance Alone and Avoidance Accompanied rating scales relative to clinical severity ratings of anxiety disorders from the Anxiety Disorders Interview Schedule (ADIS), and (c) to establish a cutoff score indicative of interviewers’ diagnosis of agoraphobia for the Avoidance Alone scale. A meta-analytic synthesis of 10 published studies yielded positive evidence for internal consistency and convergent and discriminant validity of the scales. Participants in the present study were 129 people with a diagnosis of panic disorder. Internal consistency was excellent for this sample, α = .95 for AAC and .96 for AAL. When the MIA scales were correlated with interviewer ratings, evidence for convergent and discriminant validity for AAL was strong (convergent r with agoraphobia severity ratings = .63 vs. discriminant rs of .10–.29 for other anxiety disorders) and more modest but still positive for AAC (.54 vs. .01–.37). Receiver operating curve analysis indicated that the optimal operating point for AAL as an indicator of ADIS agoraphobia diagnosis was 1.61, which yielded sensitivity of .87 and specificity of .73.     

Aggregation, acquiescence, and the assessment of trait affectivity

General affect ratings are subject to problems that lessen their validity. Some critics therefore have argued that aggregated on-line ratings provide better measures of trait affectivity. However, aggregated mood ratings suffer from two significant problems. First, correlations among same-valenced affects are substantially higher, indicating poorer discriminant validity. Second, cross-valence correlations are much weaker, reflecting a virtual absence of bipolarity. We show how both of these effects are due to an inflated acquiescence component in aggregated data. We also demonstrate that (a) these effects are a direct result of the aggregation process itself and (b) controlling for acquiescence counteracts these effects. These results indicate that the validity of aggregated mood ratings is lessened by an augmented acquiescence component.     

The effect of response style bias on the measurement of transformational,transactional, and laissez-faire leadership

Transformational, transactional, and laissez-faire leadership are considered to be three distinct leadership styles. In this research we argue that response style behaviour in the form of acquiescence and extreme response style can distort the measurement of these dimensions of leadership. Using a sample of 864 employees selected from 135 work teams, this research demonstrates that (1) response styles affect measurement; (2) divergent validity of the three dimensions increases when response styles are taken into account; (3) gender is spuriously related to leadership upon response styles; and (4) team ratings substantially change when controlling for response styles bias. As a secondary topic of this research, we elaborate on a relatively new approach in diagnosing response styles, i.e., a confirmatory latent-class factor analysis. We explain the advantages of this approach and illustrate the steps a researcher has to take in conducting this type of analysis.     

Increasing Validity with Forced‐Choice Criterion Measurement Formats

The relative validities of forced‐choice (ipsative) and Likert rating‐scale item formats as criterion measures are examined. While there has been much debate about the relative technical and psychometric merits and demerits of ipsative instruments, the present research focused on the crucial question of whether the use of this format has any practical benefit – in terms of improved validity. An analysis is reported from a meta‐analysis data set. This demonstrates that higher operational validity coefficients (prediction of line‐manager ratings of competencies) are associated with the use of forced‐choice (r=.38) rather than rating scale (r=.25) item formats for the criterion measurement instrument when performance is rated by the same line managers on both formats and where the predictor is held constant. Thus the apparent criterion‐related validity of a predictor can increase by 50% simply by changing the format of the criterion measurement instrument. The implications of this for practice are discussed.     

Less is more: Online consumer ratings' format affects purchase intentions and processing

Online customer ratings of products and services are commonplace in e‐commerce; however, the format in which these ratings are presented to consumers can vary. Although not anticipated by classical models of decision making, latter models such as prospect theory and feelings‐as‐information theory suggest that the presentation format of online customer ratings could affect subsequent consumer decision making. In the present research, 3 empirical studies test whether online customer ratings' formats differentially affect consumer purchase intentions. The results offer support for feeling‐as‐information theory and suggest that online ratings presented in a mean (vs. distribution) format result in higher purchase intentions as a result of increased processing fluency. Implications for the presentation of online consumer ratings in e‐commerce, based on these findings, are addressed.     

A Critique of Current Methods of Rorschach Scoring

Construct validity of the Concern For Appropriateness scale (CFA; Lennox &; Wolfe, 1984), a measure of the tendency to adopt protective self-presentation styles, is examined using a multitrait-multimethod approach. Self-ratings on the CFA and ratings by suitemates constitute the two methods, and concern for appropriateness, interpersonal trust, and boredom susceptibility are assessed with each. Multitrait-multimethod analysis shows that the CFA scale exhibits convergent validity against peer ratings and discriminant validity against boredom susceptibility, the only other variable demonstrating convergence. Results indicate that the CFA exhibits construct validity and might therefore prove useful as an operationalization for self-presentation tendencies.     

Construct validity of the Concern For Appropriateness scale

Construct validity of the Concern For Appropriateness scale (CFA; Lennox & Wolfe, 1984), a measure of the tendency to adopt protective self-presentation styles, is examined using a multitrait-multimethod approach. Self-ratings on the CFA and ratings by suitemates constitute the two methods, and concern for appropriateness, interpersonal trust, and boredom susceptibility are assessed with each. Multitrait-multimethod analysis shows that the CFA scale exhibits convergent validity against peer ratings and discriminant validity against boredom susceptibility, the only other variable demonstrating convergence. Results indicate that the CFA exhibits construct validity and might therefore prove useful as an operationalization for self-presentation tendencies.     

Testing the convergent and discriminant validity of the Decisional Balance Scale of the Transtheoretical Model using the Multi-Trait Multi-Method approach.

The authors extended research on the construct validity of the Decisional Balance Scale for smoking in adolescence by testing its convergent and discriminant validity. Hierarchical confirmatory factor analysis multi-trait multi-method approach (HCFA MTMM) was used with data from 2,334 UK adolescents, both smokers and non-smokers. They completed computerized and paper versions of the questionnaire on 3 occasions over 2 years. The results indicated a 3-factor solution; Social Pros, Coping Pros, and Cons fit the data best. The HCFA MTMM model fit the data well, with correlated methods and correlated trait factors. Subsequent testing confirmed discriminant validity between the factors and convergent validity of both methods of administering the questionnaire. There was, however, clear evidence of a method effect, which may have arisen due to different response formats or may be a function of the method of presentation. Taken with other data, there is strong evidence for construct validity of Decisional Balance for smoking in adolescence, but evidence of predictive validity is required.     

Validation of a Swedish short self-report measure of core affect

This research aimed at validating two self-report composite rating scales of core affect that are useful when quick assessments are required, for instance, of current mood or recalled, anticipated, and experienced emotional reactions. The ratings were derived from the Swedish Core Affect Scales (SCAS) comprising six self-report rating scales of the two orthogonal dimensions of core affect, valence (unpleasantness-pleasantness) and activation (quietness-excitement). In three samples of university students who were requested to rate current moods, affect-inducing sounds, or affect-inducing pictures, the composite ratings were compared to SCAS, two widely used graphical rating scales of valence and activation, and skin conductance responses and the acceleratory peak of heart rate. Three different rating formats were also compared. The results showed that the composite ratings were reliable and yielded the expected correlations with the other ratings and with the physiological affect indicators. No effects of rating format were detected. It is concluded that the composite ratings of valence and activation may be used if quick assessments are called for. Choices can be made of any of three rating formats depending on purpose with the assessments.     

The factor structure of the Multiple Affect Adjective Check List-Revised (MAACL-R): Some statistical considerations

In response to Zuckerman's comments on my factor analyses of the MAACL-R, I focus on the statistical issues of the cutoff criterion for factor loadings and the evidence for the discriminant validity of the five-factor solution. Based upon the conclusions I draw in these two areas, I recommend that researchers either (a) use the two factor solution to the MAACL-R represented in the summary scores of Dysphoria and Positive Affect + Sensation Seeking or (b) include all MAACL-R scales in their studies and analyze their data to determine whether it is statistically appropriate to report results from a single scale (A, D, H, PA, or SS) rather than from the summary scores.     

The Multifaceted Relationship Between Physical Activity and Affect

The purpose of the present paper is to provide a summary of scientific findings on the relationship between physical activity (PA) and affect. We provide separate narrative reviews of (a) the association between PA and well‐being in the general population; (b) PA as a prophylactic and/or treatment for clinical depression; and (c) immediate affective response to individual PA sessions and the implications for adherence to PA programs. Findings suggest an association between engaging in regular PA and more positive well‐being, decreased risk of future depression, and reduction of depressed mood among those who are already depressed. However, despite the fairly strong evidence for these associations, there is less evidence for causal effects of PA on well‐being and depression. Individual sessions of PA generally result in positive shifts in affective valence from pre‐PA to post‐PA, but during PA, affective response is intensity‐dependent, with considerable variability for moderate intensity PA. In sum, despite the “physical activity feels good” narrative often portrayed in the popular media, the relationship between PA and affect is complex and multifaceted.     

Tripartite model of anxiety and depression: psychometric evidence and taxonomic implications

We review psychometric and other evidence relevant to mixed anxiety-depression. Properties of anxiety and depression measures, including the convergent and discriminant validity of self- and clinical ratings, and interrater reliability, are examined in patient and normal samples. Results suggest that anxiety and depression can be reliably and validly assessed; moreover, although these disorders share a substantial component of general affective distress, they can be differentiated on the basis of factors specific to each syndrome. We also review evidence for these specific factors, examining the influence of context and scale content on ratings, factor analytic studies, and the role of low positive affect in depression. With these data, we argue for a tripartite structure consisting of general distress, physiological hyperarousal (specific anxiety), and anhedonia (specific depression), and we propose a diagnosis of mixed anxiety-depression.     

Comparing the validity of structured interviews for managerial‐level employees: Should we look to the past or focus on the future?

The current research investigated questions that persist regarding the criterion‐related and construct validity of situational (SI) versus past‐behaviour (PBI) structured interview formats in predicting the job performance of managers. Analyses of data collected from 157 applicants to managerial positions showed that the PBI format significantly predicted job performance ratings (r = .32, p <.01), whereas the SI format did not (r = .09, ns). Investigation of potential construct differences between the SI and PBI formats showed that the PBI was more highly related to manager‐relevant cognitive ability measures, assessment centre exercises and personality traits, as compared with the SI. Such differences help to explain the predictive validity differences between the SI and PBI observed in current and previous research.     

On the integration of personality assessment methods: the Rorschach and MMPI

Despite being the most studied and used personality assessment tools, data from the Rorschach and MMPI generally disagree (Archer & Krishnamurthy, 1993a, 1993b). Independence is proposed to result from at least 3 factors: (a) the methods tap unique levels of personality, (b) personality has a complex organization, and (c) response styles generate considerable method variance that must be considered in nomothetic research. These ideas led to 5 hypotheses, each of which received support. Rorschach and MMPI response styles are uncorrelated, although response styles are quite consistent within a method family. MMPI-2 and Rorschach constructs of dysphoria, psychosis, or wariness are uncorrelated when response styles are ignored. However, robust convergent validity is evident when patients have similar response styles on each method (e.g., for dysphoria, M r = .59) and dysphoria is expressed in opposing ways on each method when response styles are discordant (i.e., M r = -.54). Data from the latter analyses were correlated with genuine clinical phenomena and implications were discussed for clinical practice and research.