Comparative validity of traditional versus S-R format inventories of interpersonal behavior

The comparative validity of two “S-R format” inventories of interpersonal behavior, Kinnane et al.'s (1969) Schedule of Interpersonal Response and a newly developed Rational S-R Inventory, and two “traditional” inventories, the Personality Research Form and the Interpersonal Check List, was examined with respect to 14 self-report and peer rating criteria selected to broadly tap the interpersonal domain. Sixty-four male and female high school seniors served as Ss.In spite of the situational specificity inherent in the S-R inventories, the traditional inventories substantially outperformed the S-R inventories and, in contrast to previous findings, also outperformed simple self-ratings. The results are discussed not only in terms of the relative advantages of various inventory construction strategies but also in reference to the problem of criterion measurement and theoretical and methodological issues in the assessment of situational parameters.     

Predicting Performance with Contextualized Inventories,No Frame‐of‐reference Effect?

A recent meta‐analysis showed that contextualized personality inventories have incremental predictive validity over generic personality inventories when predicting job performance. This study aimed to investigate the differences between two types of contextualization of items: Adding an ‘at work’ tag versus completely modifying items. One hundred thirty‐nine pharmacy assistants from 29 pharmacies filled out a generic, a tagged, and a completely modified personality inventory. The assistants also provided participant reactions for each of the personality inventories. Performance ratings were collected from the supervising pharmacists. We expected to find incremental criterion validity for both the tagged inventory and the completely modified inventory for predicting job performance. However, the results showed an unexpected decrease in predictive validity for the contextualized inventories. Contextualized inventories were liked less than the generic inventory, but were considered somewhat more face valid and predictive by the participants.     

A COMPARISON OF CRITERIA FOR TEST VALIDATION: A META-ANALYTIC INVESTIGATION

Meta-analyses of validity coefficients from tests of clerical abilities for five criteria—supervisor ratings, supervisor rankings, work samples, production quantity, and production quality—were conducted, and the resulting expected true validities were compared. Ratings, rankings, work samples, and production quantity all resulted in high test validities. Validities resulting from ratings and quantity-of-production criteria were highly similar across tests. Validities resulting from rankings and work samples were on the average higher than those from ratings and quantity of production. The fifth criterion, quality of production, had low predictability and did not generalize across situations.     

MTLD,vocd-D,and HD-D: A validation study of sophisticated approaches to lexical diversity assessment

The main purpose of this study was to examine the validity of the approach to lexical diversity assessment known as the measure of textual lexical diversity (MTLD). The index for this approach is calculated as the mean length of word strings that maintain a criterion level of lexical variation. To validate the MTLD approach, we compared it against the performances of the primary competing indices in the field, which include vocd-D, TTR, Maas, Yule’s K, and an HD-D index derived directly from the hypergeometric distribution function. The comparisons involved assessments of convergent validity, divergent validity, internal validity, and incremental validity. The results of our assessments of these indices across two separate corpora suggest three major findings. First, MTLD performs well with respect to all four types of validity and is, in fact, the only index not found to vary as a function of text length. Second, HD-D is a viable alternative to the vocd-D standard. And third, three of the indices—MTLD, vocd-D (or HD-D), and Maas—appear to capture unique lexical information. We conclude by advising researchers to consider using MTLD, vocd-D (or HD-D), and Maas in their studies, rather than any single index, noting that lexical diversity can be assessed in many ways and each approach may be informative as to the construct under investigation.     

Temporal coherence of criteria for four personality disorders

This study sought to investigate the coherence of changes observed in diagnostic criteria for borderline, schizotypal, obsessive-compulsive, and avoidant personality disorders. Five hundred, forty-nine patients were independently evaluated 2 years apart, and correlations of observed changes in each diagnostic criterion with changes in other criteria were examined to determine if there was within-syndrome consistency in these changes. The observed changes in criteria were consistent within syndrome (median alpha = 0.72 across 4 disorders), and reasonably specific to that syndrome relative to the other disorders. The results support the validity of these criterion sets as representing coherent syndromes.     

Criterion-related validity of sex-restrictive and unisex interest scales: A comparison

This report describes two studies comparing the criterion-related validity of sex-balanced (“unisex”) interest inventory scales, i.e., scales designed such that the distributions of scores are similar for males and females, and traditional, sex-restrictive scales. Approximately 1600 college-bound high school seniors (Study 1) and 2000 college seniors (Study 2) completed both the ACT Interest Inventory (ACT-IV) and the new Unisex Edition of the ACT-IV (UNIACT), which contains sex-balanced items. In both studies, each participant was placed in one of six criterion groups based on the correspondence of expressed occupational choice (Study 1) and actual college major (Study 2) to Holland types. Comparable levels of criterion-related validity were obtained with the unisex scales, sex-restrictive scales, and with sex-balanced scores obtained by the traditional procedure of using same-sex norms. Study results and the results of previous research indicate that (a) psychometrically sound interest inventories can be constructed with sex-balanced items, and (b) counselors may use inventories which provide sex-balanced score reports without sacrificing validity.     

Evaluation of the SCID-II personality disorder traits for DSM-IV: coherence, discrimination, relations with general personality traits, and functional impairment

The utility of the DSM personality disorder (PD) system remains a concern. The strategy employed represents one approach designed to evaluate and improve the diagnostic efficiency of the SCID-II PDs. Using a sample of 203 patients, SCID-II PD items-based on the criterion sets of the 10 DSM-IV PDs-were evaluated with respect to (a) convergent validity; (b) divergent validity; (c) relation to general personality traits; and (d) association with functional impairment. Only Borderline PD items were satisfactory on all four evaluation criteria. Histrionic and Obsessive-Compulsive PD items met criteria for convergent and divergent validity and relation to personality dimensions of the Five-Factor Model of Personality (FFM) but were not related to functional impairment, suggesting they might be reconsidered as disorders. Schizotypal PD items met three of the four criteria but showed no relation to FFM dimensions, suggesting that it may be a candidate for reassignment to Axis I.     

Emotional Intelligence in Personnel Selection: Applicant reactions,criterion, and incremental validity

In two studies, based on four samples and more than 600 participants, the authors examined applicant reactions, criterion and incremental validity, and differential prediction of emotional intelligence (EI) in personnel selection using the Mayer–Salovey–Caruso Emotional Intelligence Test. The first study examined applicant reactions in terms of face validity and fairness evaluation of the EI test. The second study examined across three samples (salespeople, front‐desk public employees, and hospital chief executive officers) the criterion validity, incremental validity, and differential prediction of EI. Results suggest that the EI test is perceived as a fair selection tool, and is predictive of performance. The EI test has incremental validity over cognitive abilities and personality (big five) when predicting subjective and objective performance criteria. Based on these results, the authors encourage further research on the use of EI in selection settings.     

Introduction

This article reports a comprehensive meta-analysis of the criterion-oriented validity of the Big Five personality dimensions assessed with forced-choice (FC) inventories. Six criteria (i.e., performance ratings, training proficiency, productivity, grade-point average, global occupational performance, and global academic performance) and three types of FC scores (i.e., normative, quasi-ipsative, and ipsative) served for grouping the validity coefficients. Globally, the results showed that the Big Five assessed with FC measures have similar or slightly higher validity than the Big Five assessed with single-stimulus (SS) personality inventories. Quasi-ipsative measures of conscientiousness (K = 44, N = 8794, ρ = .40) are found to be better predictors of job performance than normative and ipsative measures. FC inventories also showed similar reliability coefficients to SS inventories. Implications of the findings for theory and practice in academic and personnel decisions are discussed, and future research is suggested.     

Examining the validity of self‐reports on scales measuring students' strategic processing

Background. Self‐report inventories trying to measure strategic processing at a global level have been much used in both basic and applied research. However, the validity of global strategy scores is open to question because such inventories assess strategy perceptions outside the context of specific task performance. Aims. The primary aim was to examine the criterion‐related and construct validity of the global strategy data obtained with the Cross‐Curricular Competencies (CCC) scale. Additionally, we wanted to compare the validity of these data with the validity of data obtained with a task‐specific self‐report inventory focusing on the same types of strategies. Sample. The sample included 269 10th‐grade students from 12 different junior high schools. Methods. Global strategy use as assessed with the CCC was compared with task‐specific strategy use reported in three different reading situations. Moreover, relationships between scores on the CCC and scores on measures of text comprehension were examined and compared with relationships between scores on the task‐specific strategy measure and the same comprehension measures. Results. The comparison between the CCC strategy scores and the task‐specific strategy scores suggested only modest criterion‐related validity for the data obtained with the global strategy inventory. The CCC strategy scores were also not related to the text comprehension measures, indicating poor construct validity. In contrast, the task‐specific strategy scores were positively related to the comprehension measures, indicating good construct validity. Conclusion. Attempts to measure strategic processing at a global level seem to have limited validity and utility.     

Convergent,discriminant, and predictive validity of aggressive driving inventories: they drive as they live

The authors hypothesized that aggressive driving is indicative of a more general trait that leads individuals to behave aggressively in different situations. This hypothesis was tested by having 85 male and 237 female undergraduates complete the Aggression Questionnaire, three measures of aggressive driving (the Driver Anger Scale, Driver Vengeance Questionnaire, and Driving Behavior Inventory), and questions about violating traffic laws. High correlations among the aggressive driving inventories provided evidence for the convergent validity, but a confirmatory factor analyses raised questions about the discriminant validity of aggressive driving inventories. The inventories did not attain significance in hierarchical analyses of traffic violations that controlled for gender, driving experience, and measures of trait aggression. It is concluded that there is little unique about aggressive driving and that individuals drive as they live. Aggr. Behav. 32:89–98, 2006. © 2006 Wiley‐Liss, Inc.     

Enhancing Criterion-Related Validity Through Bottom-Up Contextualization of Personality Inventories: The Construction of an Ecological Conscientiousness Scale for Ph.D. Candidates

In this article the concept of “ecological personality scales” is introduced. These are contextualized inventories with a high ecological validity. They are developed in a bottom-up or qualitative way and combine a relatively high trait specificity with a relatively high situational specificity. An ecological Conscientiousness or time management scale for Ph.D. candidates was developed. It significantly predicted Ph.D. performance criteria and showed incremental validity beyond Big Five (Study 1) and narrow trait and frame-of-reference scales (Study 2). These findings suggest that an ecological approach may contribute to further improving the criterion validity of personality measures.     

A longitudinal study of real-life decision making: Choosing a college

Three hundred and twenty two college-bound high school students described aspects of their college decision-making processes. Students listed their criteria as well as the alternatives (i. e., schools) under consideration, rated the importance of each criterion, and rated each alternative with respect to each criterion. They also gave their overall impressions of each alternative. Finally, students rated their comfort with the decision-making process and, at the conclusion of the study, reported on how many schools they had applied to, had or had not been accepted at, were waiting to hear from, or were waitlisted at. Students consider four or five alternatives, and use eight to ten criteria in evaluating them. These figures do not change appreciably over the course of the process, although only about half the criteria or/and slightly more than half of the schools considered at one time are considered again 6 months later, and there are several changes in the kinds of criteria considered at different points in time. There was a marginally significant trend for higher ability and average ability students to consider more criteria, more distinct types of criteria, and more alternatives than do lower ability students. There were no gender differences in this regard. Gender differences and academic ability group differences were apparent, however, in the types of criteria students reported. Participation in multiple sessions in this study had few reliable effects on decision-making performance. Students were given a list of 34 standard criteria at each session, and incorporated some of these into their own lists of criteria during subsequent sessions. However, there was no indication that repeated participation led students to adopt a more analytical strategy than they would have otherwise. Data were compared with three linear models of information integration. Models using data with multiple criteria better fit the students' data than did a model using only the most important criterion. Higher ability students were particularly better able to integrate information according to linear models.     

The Common Beliefs Survey-III and the Situational Self-Statement and Affective State Inventory: Test-Retest Reliability, Internal Consistency, and Further Psychometric Considerations

The Common Beliefs Survey-III (CBS-III), a factored measure of general irrational beliefs, has satisfactory convergent and discriminant validity. The Situational Self-Statement and Affective State Inventory (SSSASI), also with acceptable basic psychometric properties, assesses specific thoughts and feelings in response to vignettes describing frustrating events. Both inventories are potentially useful in assessing dimensions important to REBT, but information on test-retest reliability is needed. The CBS-III and the SSSASI were administered to 101 undergraduate students on two occasions, two or three weeks apart. Satisfactory test-retest reliability coefficients were obtained on all subscales of both inventories, but in the case of the SSSASI reliabilities were generally stronger when the same form of the test was used on both occasions (Form 1 presented the thoughts before the feelings, whereas Form 2 presented the feelings before the thoughts). Further exploration showed that responding to the thoughts before the feelings significantly attenuated scores on the feelings, but responding to the feelings before the thoughts had no effect on scores on the thoughts. This unexpected finding also held true in a previously unexamined data set obtained with a different sample of students several years earlier. Taking careful stock of one's emotional reactions before embarking on cognitive modification interventions seems essential to REBT, and it follows that asking questionnaire respondents to rate their affective states before they rate their self-statements could be desirable conceptually. These results provide empirical support for that strategy.     

Diagnosing agoraphobia in the context of panic disorder: examining the effect of DSM-IV criteria on diagnostic decision-making

A diagnostic and statistical manual (DSM)-IV diagnosis of agoraphobia in the context of panic disorder (PD) is based on three nosologically sufficient criteria: (1) avoidance, (2) use of companions, and (3) endurance of situations despite distress. Therefore, an agoraphobia diagnosis can be made across an extremely broad range of cases including when there are no avoidance behaviors (e.g., the patient endures the situation). It was hypothesized that clinicians do not weight these criteria equally and that the DSMs individual, sufficient criteria lead to poor inter-rater reliability. Clinicians (N=48) rated hypothetical patients with symptom profiles emphasizing each of these three criteria. Consistent with expectation, clinicians differentially weighted these criteria. Avoidance was relatively more apt to produce a diagnosis when only one criterion was emphasized in clinical vignettes. Inter-rater reliability was poor in instances when only one sufficient criterion was highlighted. Knowledge concerning DSM criteria resulted in a greater rate of agoraphobia endorsement, but knowledge did not account for the overall pattern of findings.     

The incremental validity of organizational commitment,organizational trust,and organizational identification

Organizational commitment (OC), organizational trust (OT), and organizational identification (OI) are three types of psychological attachment to an organization. Each of these three variables captures an organization-targeted attitude toward an employment relationship, but it is unclear whether they have incremental validity over each other. To address this question, this study examined the incremental validity of each variable in predicting job involvement, job satisfaction, turnover intentions, and non-self-report measures of task performance and citizenship behavior. It also examined whether perceived organizational support and psychological contract breach, two other organization-targeted attitudinal variables, were related to OC, OT, and OI when the latter were considered jointly. Meta-analytical evidence suggests that OC, OT, and OI have incremental validity over and above one another in their relationships with some, but not all, of the above correlates. This highlights the need for future research to distinguish these three types of psychological attachment to an organization.     

Developing Measures of Basic Job-Relevant English Proficiency for the Prediction of Job Performance and Promotability

The present study is an attempt to devise reliable and valid measures of job-relevant basic English proficiency to be applied in the prediction of job performance and promotability in an entry-level manual labor job dominated by workers whose native language is not English. An audio test and a reading test were developed to assess the ability to understand spoken English and written English respectively using job relevant material. Results showed that both tests demonstrated high predictive validity for the criteria of supervisory assessments of incumbents' English proficiency on the job, overall job performance, and promotability. In predicting ratings of overall job performance and promotability, each test provided incremental validity over the predictability provided by the other test. In addition, the tests also provided incremental validity in predicting promotability over the predictability provided by overall job performance.     

Detection of feigned mental disorders: a meta-analysis of the MMPI-2 and malingering

The validity of test data from multiscale inventories is dependent on self-reports that may be easily distorted by malingering. In examining the Minnesota Multiphasic Personality Inventory-2's (MMPI-2) role in the assessment of feigning, this review provides a conceptual analysis of the detection strategies underlying the MMPI-2 validity scales. The conceptual analysis is augmented by comprehensive meta-analysis of 65 MMPI-2 feigning studies plus 11 MMPI-2 diagnostic studies. For the rare-symptoms strategy, Fp (Cohen's d = 2.02) appears especially effective across diagnostic groups; its cut scores evidence greater consistency than most validity indicators. The data supported the F as an effective scale but questioned the routine use of Fb. Among the specialized scales, Ds appeared especially useful because of its sophisticated strategy, consistent cut score, and minimalfalse-positives. General guidelines are offeredfor specific MMPI-2 validity scales in the assessment of malingering with specific diagnoses.