Incremental validity and the assessment of psychopathology in adults

Results from studies on incremental validity are described for (a) interviews, (b) personality inventories, (c) projective techniques, and (d) brief self-rated and clinician-rated measures. In some of the studies (clinical judgment studies), psychologists were given increasing amounts of information. In other studies (statistical prediction studies), increasing amounts of assessment information were entered into a statistical prediction rule. Although relatively little research has been conducted on incremental validity, results that have been obtained tend to favor the use of interviews, personality inventories (e.g., the Minnesota Multiphasic Personality Inventory--2; J. N. Butcher, W. G. Dahlstrom, J. R., Graham, A. Tellegen, & B. Kaemmer, 1989), and brief self-rated measures (e.g., the Anxiety Sensitivity Index, S. Reiss, R. A. Peterson, D. M. Gursky, & R. M. McNally, 1986). Results are generally less encouraging for projective techniques.     

Integrative Self-Knowledge Scale: correlations and incremental validity of a cross-cultural measure developed in Iran and the United States

The authors used Iranian (N = 723) and American (N = 900) samples to develop an Integrative Self-Knowledge Scale for measuring a temporally integrated understanding of processes within the self. They administered this new instrument, the Mindfulness Scale (K. W. Brown & R. M. Ryan, 2003), the Reflective and Experiential Self-Knowledge Scales (N. Ghorbani, M. N. Bing, P. J. Watson, H. R. Davison, & D. L. Lebreton, 2003), and additional sample-specific measures to 3 separate groups of university students in each society. The Integrative Self-Knowledge Scale displayed internal reliability and measurement equivalence, along with convergent, criterion, discriminant, and incremental validity. This new instrument may be useful in promoting cross-cultural research in positive psychology.     

Cross-language computational investigation of the length effect in reading aloud

The authors examined whether 2 computational models of reading, the dual-route cascaded model (M. Coltheart, K. Rastle, C. Perry, R. Langdon, & J. C. Ziegler, 2001) and the connectionist 2-layer model (M. Zorzi, G. Houghton, & B. Butterworth, 1998), were able to predict the pattern that the length effect found in reading aloud is larger in German than in English (J. C. Ziegler, C. Perry, A. M. Jacobs, & M. Braun, 2001). The results showed that the dual-route cascaded model, which uses a serial mechanism for assembling phonology, successfully predicted this cross-language difference. In contrast, the connectionist model of Zorzi et al. (1998) predicted the opposite: a larger length effect in English than in German. Both the success of one model and the failure of the other highlight fundamental differences between 2 major classes of computational models.     

Incremental validity of the frame-of-reference effect in personality scale scores: a replication and extension

Context-specific personality items provide respondents with a common frame of reference unlike more traditional, noncontextual personality items. The common frame of reference standardizes item interpretation and has been shown to reduce measurement error while increasing validity in comparison to noncontextual items (M. J. Schmit, A. M. Ryan. S. L. Stierwalt. & S. L. Powell, 1995). Although the frame-of-reference effect on personality scales scores has been well investigated (e.g., M. J. Schmit et al., 1995), the ability of this innovation to obtain incremental validity above and beyond the well-established, noncontextual personality scale scores has yet to be examined. The current study replicates and extends work by M. J. Schmit et al. (1995) to determine the incremental validity of the frame-of-reference effect. The results indicate that context-specific personality items do indeed obtain incremental validity above and beyond both noncontextual items and cognitive ability, and in spite of socially desirable responding induced by applicant instructions. The implications of these findings for personnel selection are discussed.     

Let's call the whole thing off: a response to Dawes (2001)

There has been a recent controversy regarding the validity of the Rorschach test. This comment is in response to criticism levied by R. M. Dawes (2001) on the incremental validity of the Ego Impairment Index (EII), a Rorschach-derived measure of cognition, perception, and reasoning. The Dawes articles (1999 and 2001) serve as an example of the recent practice of placing extraordinary challenges on the Rorschach test. Dawes's arguments are examined and parallel examples are provided that demonstrate the bias used to judge the validity of the EII, the Rorschach, and psychological assessment. Still, in the face of criticism, the results (see Dawes, 2001) support the incremental validity of the EII. Thus, the conclusion presented in this Comment is that it is time for us to "call the whole thing off" and end the Rorschach controversy that has occupied so much recent attention and generated so few new ideas.     

Incremental validity of the Ego Impairment Index: a re-examination of Dawes (1999)

This article examined the incremental validity of the Ego Impairment Index (EII), a Rorschach measure of cognition, perception, and reasoning. R. M. Dawes (1999) borrowed 2 previously published data sets to assess the incremental validity of the EII. Dawes determined that in order for the EII to be considered a valid measure, the overall EII score should "outperform" 2 of the variables that compose the index. Using this approach, Dawes reported that the EII had "deficient" (p = .059) incremental validity. In this study, the incremental validity of the EII was re-examined using 1 of the data sets used by R. M. Dawes (1999). Applying the same strategy as Dawes, the author tested the incremental validity of the EII in predicting positive symptoms of schizophrenia, a more appropriate and clinically relevant criterion measure of impairment. The EII significantly predicted positive symptoms (p < .005) even after the 2 EII variables selected by Dawes and the measure of social competence were 1st entered into the equation. The results suggest that the EII adds meaningful information in the assessment of thought disturbance.     

Incremental validity of new clinical assessment measures

The authors address conceptual and methodological foundations of incremental validity in the evaluation of newly developed clinical assessment measures. Incremental validity is defined as the degree to which a measure explains or predicts a phenomenon of interest, relative to other measures. Incremental validity can be evaluated on several dimensions, such as sensitivity to change, diagnostic efficacy, content validity, treatment design and outcome, and convergent validity. Indices of incremental validity can vary depending on the criterion measures, comparison measures, and individual differences in samples. The authors review the rationale for, principles, and methods of incremental validation, including the selection of comparison and criterion measures, and address data analytic strategies and the conditional nature of incremental validity evaluations in the selection of measures. Incremental validity contributes to, but is different from, cost-benefits, which reflect the cost of acquiring the data and the benefits from the data. The impact of an incremental validity index on whether a measure is selected will be moderated by the cost of acquiring the new data, the importance of the measured phenomenon, and the clinical utility of the new data.     

The positive and negative semantic dimensions of relationship satisfaction

Semantic differential items were reconfigured to assess relationship satisfaction across separate positive and negative attitude dimensions. Study 1 (N = 1,656) supported a 2‐factor model for the Positive and Negative Semantic Differential (PN–SMD), as well as its convergent, criterion‐related, and incremental validity over the 16‐item Couples Satisfaction Index (CSI; J. L. Funk & R. D. Rogge, 2007) using known correlates of relationship satisfaction as criteria. Study 2 (N = 89) replicated the convergent, criterion‐related, and incremental validity findings of Study 1 using different criterion measures, the CSI, a bipolar semantic differential measure designed for assessing relationship satisfaction, and an existing 2‐dimensional measure of relationship satisfaction. The authors demonstrated across studies that the PN–SMD captures criterion‐relevant information about ambivalence versus indifference toward the relationship—associations that are only detectable when using a 2‐dimensional satisfaction measure.     

Perry's Legacy: Models of Epistemological Development

This paper compares the different ways that intellectual development has been described by W. Perry (1970) and M. Belenky, B. Clinchy, N. Goldberger, and J. Tarule (1986); M. Baxter Magolda (1992); and P. King and K. Kitchener (1994). The similarities are summarized into a comprehensive, gender-neutral 4-stage model (E. J. West, 1996). The nature of the stages is discussed from both a constructivist-developmental perspective (H. G. Furth, 1981; A. Colby & L. Kohlberg, 1987) and using skill-complexity theory (M. L. Commons, E. J. Trudeau, S. A. Stein, F. A. Richards, & S. R. Krause, 1998; M. L. Commons, D. Danaher-Gilpin, P. M. Miller, & E. A. Goodheart, 2002). Educational implications are proposed.     

Using dynamic risk and protective factors to predict inpatient aggression: Reliability and validity of START assessments

The Short-Term Assessment of Risk and Treatability (START; C. D. Webster, M. L. Martin, J. Brink, T. L. Nicholls, & S. L. Desmarais, 2009; C. D. Webster, M. L. Martin, J. Brink, T. L. Nicholls, & C. Middleton, 2004) is a relatively new structured professional judgment guide for the assessment and management of short-term risks associated with mental, substance use, and personality disorders. The scheme may be distinguished from other violence risk assessment instruments because of its inclusion of 20 dynamic factors that are rated in terms of both vulnerability and strength. This study examined the reliability and validity of START assessments in predicting inpatient aggression. Research assistants completed START assessments for 120 male forensic psychiatric patients through review of hospital files. They also completed Historical-Clinical-Risk Management-20 (HCR-20; C. D. Webster, K. S. Douglas, D. Eaves, & S. D. Hart, 1997) and Hare Psychopathy Checklist: Screening Version (PCL:SV; S. D. Hart, D. N. Cox, & R. D. Hare, 1995) assessments. Outcome data were coded from hospital files for a 12-month follow-up period using the Overt Aggression Scale (OAS; S. C. Yudofsky, J. M. Silver, W. Jackson, J. Endicott, & D. W. Williams, 1986). START assessments evidenced excellent interrater reliability and demonstrated both predictive and incremental validity over the HCR-20 Historical subscale scores and PCL:SV total scores. Overall, results support the reliability and validity of START assessments and use of the structured professional judgment approach more broadly, as well as the value of using dynamic risk and protective factors to assess violence risk. (PsycINFO Database Record (c) 2012 APA, all rights reserved).     

Development and preliminary validation of a Chinese version of the Buss-Perry Aggression Questionnaire in a population of Hong Kong Chinese

I developed a Chinese version of the Aggression Questionnaire (Buss & Perry, 1992) by translating scale items into Chinese and subjecting them to standard validation procedures. I used confirmatory factor analysis via structural equation modeling to compare several measurement models. Models based on Buss and Perry's (1992) original four-factor (29-item) scale failed to replicate in the Chinese sample; however, the construct validity of Bryant and Smith's (2001) abridged version of the Aggression Questionnaire received strong overall support. The new 12-item scale demonstrated good fit to the data and adequate internal reliability. Evidence for criterion validity was provided by the scale's sensitivity to differing levels of aggression in males and females. Convergent and discriminant validity received partial support from the pattern of correlations with a measure of anger rumination. Linguistic and metric equivalence were supported by high correlation coefficients between scores on Chinese and English versions of the scale completed by bilingual Chinese on separate occasions. Consistent replications of these preliminary results across three independent samples suggest that the Chinese version of the Aggression Questionnaire may be useful for clinical assessment and cross-cultural research.     

Revised estimates of dimension and exercise variance components in assessment center postexercise dimension ratings

The authors reanalyzed assessment center (AC) multitrait-multimethod (MTMM) matrices containing correlations among postexercise dimension ratings (PEDRs) reported by F. Lievens and J. M. Conway (2001). Unlike F. Lievens and J. M. Conway, who used a correlated dimension-correlated uniqueness model, we used a different set of confirmatory-factor-analysis-based models (1-dimension-correlated Exercise and 1-dimension-correlated uniqueness models) to estimate dimension and exercise variance components in AC PEDRs. Results of reanalyses suggest that, consistent with previous narrative reviews, exercise variance components dominate over dimension variance components after all. Implications for AC construct validity and possible redirections of research on the validity of ACs are discussed.     

Emotional Intelligence in Personnel Selection: Applicant reactions,criterion, and incremental validity

In two studies, based on four samples and more than 600 participants, the authors examined applicant reactions, criterion and incremental validity, and differential prediction of emotional intelligence (EI) in personnel selection using the Mayer–Salovey–Caruso Emotional Intelligence Test. The first study examined applicant reactions in terms of face validity and fairness evaluation of the EI test. The second study examined across three samples (salespeople, front‐desk public employees, and hospital chief executive officers) the criterion validity, incremental validity, and differential prediction of EI. Results suggest that the EI test is perceived as a fair selection tool, and is predictive of performance. The EI test has incremental validity over cognitive abilities and personality (big five) when predicting subjective and objective performance criteria. Based on these results, the authors encourage further research on the use of EI in selection settings.     

The incremental validity of passive-aggressive personality symptoms rivals or exceeds that of other personality symptoms in suicidal outpatients

We examined the incremental validity of passive-aggressive and other personality symptoms, by determining personality syndromes' ability to account for unique variance in distress and impairment as indexed by several Axis I-related symptoms. Two hundred fifty young adults, referred for suicidal behavior or ideation, completed the Millon Clinical Multiaxial Inventory (Millon, 1983) as well as several other symptom measures. The personality syndromes varied considerably in their ability to account for unique variance in Axis I-related symptoms. Several syndromes, including schizoid, schizotypal, and compulsive symptoms, showed relatively low incremental validity, whereas the incremental validity of passive-aggressive symptoms exceeded all others. The criterion of incremental validity represents an inclusive and persuasive means to evaluate personality disorder validity. On this criterion, passive-aggressive symptoms display high validity, and passive-aggressive personality disorder, in some form, may deserve inclusion on Axis II of future editions of Diagnostic and Statistical Manual of Mental Disorders (American Psychiatric Association).     

Child Care in The Netherlands: Trends in Quality Over the Years 1995-2005

The authors assessed the quality of child care in a representative national sample of 42 child-care centers in the Netherlands and compared it with the quality of care that researchers have found using similar samples in 1995 (M. H. van IJzendoorn, L. W. C. Tavecchio, G. J. J. M. Stams, M. J. E. Verhoeven, & E. J. Reiling, 1998) and 2001 (M. J. J. M. Gevers Deynoot-Schaub & J. M. A. Riksen-Walraven, 2005). In the present study, results showed a low level of overall process quality for the 2005 sample, measured by the Infant/Toddler Environment Rating Scale-Revised (T. Harms, D. Cryer, & R. M. Clifford, 2003) and the Early Childhood Environment Rating Scale-Revised (T. Harms, R. M. Clifford, & D. Cryer, 1998). The present authors found a significant decline in process quality in comparison with the 1995 and 2001 findings. They concluded that, from an international perspective, the Netherlands has lost its leading position in child-care quality compared with that from 10 years ago.     

Assessing Intellectual Development: Three Approaches, One Sequence

In this paper, I compare three developmental assessment systems, employed to score a set of 152 interviews of engineering students: the Perry Scoring System (W. G. Perry, 1970), the Hierarchical Complexity Scoring System (T. L. Dawson, 2004, 1/31/03), and the Lexical Abstraction Assessment System (LAAS; T. L. Dawson & M. Wilson, in press). Overall, the Hierarchical Complexity Scoring System and Perry Scoring System agree with one another within the parameters of interrater agreement commonly reported for either one of the systems, and the Perry system and the LAAS agree with one another about as well as the LAAS and the Hierarchical Complexity Scoring System, upon which the LAAS is based.     

The incremental validity of psychological testing and assessment: conceptual, methodological, and statistical issues

There has been insufficient effort in most areas of applied psychology to evaluate incremental validity. To further this kind of validity research, the authors examined applicable research designs, including those to assess the incremental validity of test instruments, of test-informed clinical inferences, and of newly developed measures. The authors also considered key statistical and measurement issues that can influence incremental validity findings, including the entry order of predictor variables, how to interpret the size of a validity increment, and possible artifactual effects in the criteria selected for incremental validity research. The authors concluded by suggesting steps for building a cumulative research base concerning incremental validity and by describing challenges associated with applying nomothetic research findings to individual clinical cases.     

An inferential confidence interval method of establishing statistical equivalence that corrects Tryon's (2001) reduction factor

Evidence of group matching frequently takes the form of a nonsignificant test of statistical difference. Theoretical hypotheses of no difference are also tested in this way. These practices are flawed in that null hypothesis statistical testing provides evidence against the null hypothesis and failing to reject H-sub-0 is not evidence supportive of it. Tests of statistical equivalence are needed. This article corrects the inferential confidence interval (ICI) reduction factor introduced by W. W. Tryon (2001) and uses it to extend his discussion of statistical equivalence. This method is shown to be algebraically equivalent with D. J. Schuirmann's (1987) use of 2 one-sided t tests, a highly regarded and accepted method of testing for statistical equivalence. The ICI method provides an intuitive graphic method for inferring statistical difference as well as equivalence. Trivial difference occurs when a test of difference and a test of equivalence are both passed. Statistical indeterminacy results when both tests are failed. Hybrid confidence intervals are introduced that impose ICI limits on standard confidence intervals. These intervals are recommended as replacements for error bars because they facilitate inferences.     

Incremental validity of the MMPI-A content scales in a residential treatment facility

The current study explores the incremental validity of the Minnesota Multiphasic Personality Inventory-Adolescent (MMPI-A) content scales. Participants were 335 adolescents (250 boys and 85 girls) between the ages of 13 and 18 who were receiving mental health services from a residential treatment facility. Regression analyses were conducted to identify the amount of additional variance accounted for by the content scales over the clinical scales in predicting scores on a clinician rating form of psychological symptomatology. Several of the MMPI-A content scales demonstrated significant incremental validity above the clinical scales in predicting clinician ratings of adolescents' behavior and personality characteristics. The clinical scales also demonstrated incremental validity in reference to the content scales, indicating then that the two sets of scales provide complementary information. Magnitude of the additional criterion variance predicted was modest.