Incremental validity and the assessment of psychopathology in adults

Results from studies on incremental validity are described for (a) interviews, (b) personality inventories, (c) projective techniques, and (d) brief self-rated and clinician-rated measures. In some of the studies (clinical judgment studies), psychologists were given increasing amounts of information. In other studies (statistical prediction studies), increasing amounts of assessment information were entered into a statistical prediction rule. Although relatively little research has been conducted on incremental validity, results that have been obtained tend to favor the use of interviews, personality inventories (e.g., the Minnesota Multiphasic Personality Inventory--2; J. N. Butcher, W. G. Dahlstrom, J. R., Graham, A. Tellegen, & B. Kaemmer, 1989), and brief self-rated measures (e.g., the Anxiety Sensitivity Index, S. Reiss, R. A. Peterson, D. M. Gursky, & R. M. McNally, 1986). Results are generally less encouraging for projective techniques.     

Incremental validity of new clinical assessment measures

The authors address conceptual and methodological foundations of incremental validity in the evaluation of newly developed clinical assessment measures. Incremental validity is defined as the degree to which a measure explains or predicts a phenomenon of interest, relative to other measures. Incremental validity can be evaluated on several dimensions, such as sensitivity to change, diagnostic efficacy, content validity, treatment design and outcome, and convergent validity. Indices of incremental validity can vary depending on the criterion measures, comparison measures, and individual differences in samples. The authors review the rationale for, principles, and methods of incremental validation, including the selection of comparison and criterion measures, and address data analytic strategies and the conditional nature of incremental validity evaluations in the selection of measures. Incremental validity contributes to, but is different from, cost-benefits, which reflect the cost of acquiring the data and the benefits from the data. The impact of an incremental validity index on whether a measure is selected will be moderated by the cost of acquiring the new data, the importance of the measured phenomenon, and the clinical utility of the new data.     

Issues in the psychological assessment of preschool children

This article discussed two issues relating to assessment of preschool children: (1) whether assessments of children of this age should be attempted, and (2) the issue of what is a “good” assessment. The general impression from a review of the literature and from personal experience is that preschool assessment is desirable and can be useful, but only under conditions which avoid labeling and attempt to circumvent and reduce the problems of reliability and validity, taking into consideration the interacting variables of the child, the task, and the setting.     

Incremental validity principles in test construction

The authors articulate 5 basic principles for enhancing incremental validity, both among elements within a test and between tests, during test construction: (a) careful, precise articulation of each element or facet within the content domain; (b) reliable measurement of each facet through use of multiple, alternate-form items; (c) examination of incremental validity at the facet level rather than the broad construct level; (d) use of items that represent single facets rather than combinations of facets; and (e) empirical examination of whether there is a broad construct or a combination of separate constructs. Using these principles, the authors offer specific suggestions for modifications in 3 classic test construction approaches; (a) criterion keying, (b) inductive test construction, and (c) deductive test construction. Implementation of these suggestions is likely to provide theoretical clarification and improved prediction.     

Incremental validity of the psychological inventory of criminal thinking styles as a predictor of continuous and dichotomous measures of recidivism

Postrelease recidivism data were collected on 137 male inmates released 1 to 72 months after completing the Psychological Inventory of Criminal Thinking Styles (PICTS) in follow-ups lasting 6 to 55 months. When a dichotomous measure of recidivism (0, 1+ arrests) was employed, Entitlement (En) was the only PICTS thinking-style scale to achieve significance after age, prior arrests, time at risk in the community, and the PICTS Superoptimism (So) scale were controlled in a binary logistic regression analysis. Both the Cutoff (Co) and En scales proved significant in predicting a continuous measure of recidivism (range = 0 to 6) after age, prior arrests, time at risk, and the So scale were controlled in a sequential linear regression analysis. Whereas the En scale displayed mixed incremental validity results relative to the Co scale, the Co scale failed to achieve incremental validity relative to the En scale.     

Reliability and concurrent validity of the movement assessment battery for children.

Reliability and concurrent validity of the Movement Assessment Battery for Children were evaluated with a sample of 106 boys and girls, distributed into four age groups: 5-6 yr. (n = 20), 7-8 yr. (n = 20), 9-10 yr. (n = 46), and 11-12 yr. (n = 20). Test-retest reliability of the Movement Assessment Battery for Children, estimated using intraclass correlation coefficients, was high across all age groups, and concurrent validity yielded moderate Pearson correlation coefficients between the Move ment battery and long and short forms of the Bruininks-Oseretsky Test of Motor Proficiency. These results support the use of the Movement battery as a measure of motor ability in children, ages 5 to 12 years.     

Construct validity and case validity in assessment

Clinical assessment relies on both construct validity, which focuses on the accuracy of conclusions about a psychological phenomenon drawn from responses to a measure, and case validity, which focuses on the synthesis of the full range of psychological phenomena pertaining to the concern or question at hand. Whereas construct validity is grounded in understanding causal influences of a distinct phenomenon on responses to various measures and life contexts, case validity encompasses the joint influences of multiple phenomena on individuals' responses. Two sets of distinctions essential to understanding psychological phenomena, hence to understanding construct validity, are (a) implicit and explicit versions of personality constructs and (b) ability and personality as versions of constructs measured by performance tests presenting maximal and typical conditions, respectively. Since both implicit and explicit versions of constructs interface with maximal or typical performance conditions, case validity requires systematic inclusion of these distinctions in assessment protocols.     

Rorschach Depression indices with children and adolescents: concurrent validity findings.

Exner (1983, 1986) developed and recently revised (1990b) a Rorschach Depression Index based on scores from variables in the Comprehensive System. This study evaluated both the original and the revised DEPIs for child and adolescent outpatient (n = 67) and inpatient (n = 99) samples in order to assess the diagnostic utility of these indices. There were no significant relationships between the original form and the revised form of the DEPI and clinical elevations on the Depression scale of the Personality Inventory for Children in the outpatient sample or treatment team diagnostic judgments in the inpatient sample. These findings sound a strong cautionary note for using only Rorschach Depression indices to diagnose depression in children and adolescents.     

Survey of current practices in the psychological assessment of limited-english-proficiency handicapped children

This article discusses current psychological and educational assessment practices used with elementary school limited-English-proficiency (LEP) children, as revealed by a nationwide study of 21 local education agencies (LEAs) that had LEP handicapped children mainstreamed in bilingual education classrooms. Data were obtatained through telephone and personal interviews of bilingual and special education directors, inspection of documents, and site visits. The testing approaches most frequently used with LEP students were the common culture approach (which relies on nonverbal measures) and translated tests. In many of the LEAs, translations were done while testing was in progress. An average of 12 psychological, language, and educational tests were used with each LEP child. The assessment instruments used were not normed on LEP populations. Only a third of the LEAs studied were incorporating multicultural pluralistic approaches in their assessment procedures. The professionals involved in the assessment of LEP children were the same as those used for English-speaking children except that they were bilingual. Most of the LEAs reported serious shortages in bilingual assessment personnel. LEAs have tried to cope with this problem in various ways, including the use of trained or untrained interpreters, the use of central teams or contracted professionals, and the use of nonverbal tests administered by nonbilingual personnel. However, many children remain to be assessed, and LEAs have been forced to establish priorities.     

Focus groups in psychological assessment: enhancing content validity by consulting members of the target population

A review of articles in Psychological Assessment reveals that many researchers develop instruments without the benefit of consultation with members of the target population. To the extent that researchers do consult the target population, most fail to bring consultation in early enough to inform the identification and specification of key constructs. Moreover, this consultation typically takes the form of one-to-one interviews. The authors' goal in this article was to elaborate on the importance of population consultation as part of content validation and to critically evaluate the potential of one method of consultation, focus groups, to inform multiple stages of instrument development. The authors suggest that this method holds promise for enhancing the content validity of instruments and, ultimately, the validity of research findings.     

Toward a process-focused model of test score validity: improving psychological assessment in science and practice

Although definitions of validity have evolved considerably since L. J. Cronbach and P. E. Meehl's classic (1955) review, contemporary validity research continues to emphasize correlational analyses assessing predictor-criterion relationships, with most outcome criteria being self-reports. The present article describes an alternative way of operationalizing validity--the process-focused (PF) model. The PF model conceptualizes validity as the degree to which respondents can be shown to engage in a predictable set of psychological processes during testing, with those processes dictated a priori by the nature of the instrument(s) used and the context in which testing takes place. In contrast to the traditional approach wherein correlational methods are used to quantify the relationship between test score and criterion, the PF model uses experimental methods to manipulate variables that moderate test score-criterion relationships, enabling researchers to draw more definitive conclusions regarding the impact of underlying psychological processes on test scores. By complementing outcome-based validity assessment with a process-driven approach, researchers will not only improve psychology's assessment procedures but also enhance their understanding of test bias and test score misuse by illuminating the intra- and interpersonal factors that lead to differential performance (and differential prediction) in different groups.     

Incremental validity of a measure of emotional intelligence

After the Schutte Self-Report Inventory of Emotional Intelligence (SSRI; Schutte et al., 1998) was found to predict college grade point average, subsequent emotional intelligence (EI)-college adjustment research has used inconsistent measures and widely varying criteria, resulting in confusion about the construct's predictive validity. In this study, we assessed the SSRI's incremental validity for a wide range of adjustment criteria, pitting it against a competing trait measure, the NEO Five-Factor Inventory (NEO-FFI; Costa & McCrae, 1992), and tests of fluid and crystallized intelligence. At a broad bandwidth, the SSRI total score significantly and uniquely predicted variance beyond NEO-FFI domain scores in the UCLA Loneliness Scale, Revised (Russell, Peplau, & Cutrono, 1980) scores. Higher fidelity analyses using previously identified SSRI factors and NEO-FFI item clusters revealed that the SSRI's Optimism/Mood Regulation and Emotion Appraisal factors contributed unique variance to self-reported study habits and social stress, respectively. The potential moderation of incremental validity by gender did not reach significance due to loss of power from splitting the sample, and mediational analyses revealed the SSRI Optimism/Mood Regulation factor was both directly and indirectly related to various criteria. We discuss the small magnitude of incremental validity coefficients and the differential incremental validity of SSRI factor and total scores.     

The incremental validity of psychological testing and assessment: conceptual, methodological, and statistical issues

There has been insufficient effort in most areas of applied psychology to evaluate incremental validity. To further this kind of validity research, the authors examined applicable research designs, including those to assess the incremental validity of test instruments, of test-informed clinical inferences, and of newly developed measures. The authors also considered key statistical and measurement issues that can influence incremental validity findings, including the entry order of predictor variables, how to interpret the size of a validity increment, and possible artifactual effects in the criteria selected for incremental validity research. The authors concluded by suggesting steps for building a cumulative research base concerning incremental validity and by describing challenges associated with applying nomothetic research findings to individual clinical cases.     

Recognizing young children in need of mental health assessment: Development and preliminary validity of the early childhood screening assessment

The Early Childhood Screening Assessment (ECSA) is a primary care screening measure developed to identify very young children (1½–5 years old) who need further emotional or behavioral assessment. The ECSA was developed specifically to meet the logistical constraints of primary care settings. This study assessed the preliminary psychometric properties of the ECSA by comparing it with extant validated measures of young children's emotional and behavioral development, comparing it with a diagnostic interview, and measuring test‐retest reliability. In the study, 309 mothers in two primary care clinics completed the ECSA and the Child Behavior Checklist (CBCL; T. Achenbach & L. Rescorla, 2000). A subset (n = 69) also completed the Diagnostic Interview for the Preschool Age (DIPA; M. Scheeringa & N. Haslett, 2010). ECSA score correlated significantly and strongly with the CBCL Total Problem T score (Spearman's rho = 0.86, p ? .01). The ECSA was 86% sensitive and 83% specific in identifying DIPA diagnoses. Internal consistency of the ECSA was 0.91. Test‐retest reliability at 10 days was excellent (Spearman's rho = 0.81, p ? .01). The ECSA is an easy‐to‐use screening measure that demonstrates strong convergent validity, criterion validity, and test‐retest reliability in the pediatric setting. It shows potential as a feasible and psychometrically strong tool for busy primary care providers to identify preschoolers who need further socioemotional assessment.     

Multivariate base rates for the assessment of executive functioning among children and adolescents

This study provides observed base rates of low executive functioning test scores among healthy children and adolescents, stratifies those base rates by narrow intellectual functioning and age groupings, and provides normative classification ranges to aid the interpretation of performances on the Delis-Kaplan Executive Function System (D-KEFS) in clinical practice and research. Participants included 875 children and adolescents between 8 and 19 years old from the D-KEFS normative sample (48% male; 52% female). Among these participants, 838 had complete data and were included in the current study. The racial/ethnic composition of the sample was: White (73.7%), African American (12.4%), Hispanic (11.1%), and other racial/ethnic backgrounds (2.7%). The Overall Test Battery Mean (OTBM) and the prevalence of low scores at various clinical cut-offs were calculated for the 13 primary scores from the D-KEFS Trail Making Test, Verbal Fluency Test, and Color-Word Interference Test. The OTBM and base rates were also calculated separately for those scores reflecting executive functioning (n = 7) and processing speed (n = 6). Healthy children and adolescents commonly obtained low scores on the D-KEFS tests considered here. Younger age, lower estimated full-scale intelligence quotient, and more test scores interpreted were associated with a greater frequency of low scores. Clinicians and researchers are encouraged to consider these multivariate base rates when assessing and attempting to identify executive functioning impairment among children and adolescents with the D-KEFS.     

Incremental validity of the MMPI-A content scales in the prediction of self-reported symptoms

The incremental contribution of the MMPI-A (Butcher et al., 1992) content scales to the prediction of scores on self-report measures of psychopathology was examined in a sample of 62 adolescents in inpatient treatment and 59 adolescents from the community. All participants completed the MMPI-A and a battery of criterion measures. A series of hierarchical regression analyses was conducted in which the MMPI-A clinical and content scales served as the independent variables and the criterion measures as the dependent variables. The content scales were found to have incremental validity beyond the clinical scales in predicting variance in the criterion measures. Similarly, the clinical scales also demonstrated incremental validity over the content scales in making these predictions. Both sets of scales made independent contributions to the prediction of sample membership (clinical vs. nonclinical). Findings suggest that both the clinical and content scales of the MMPI-A make significant contributions to the assessment of adolescents' psychological functioning.     

Incremental validity of the MMPI-A content scales in a residential treatment facility

The current study explores the incremental validity of the Minnesota Multiphasic Personality Inventory-Adolescent (MMPI-A) content scales. Participants were 335 adolescents (250 boys and 85 girls) between the ages of 13 and 18 who were receiving mental health services from a residential treatment facility. Regression analyses were conducted to identify the amount of additional variance accounted for by the content scales over the clinical scales in predicting scores on a clinician rating form of psychological symptomatology. Several of the MMPI-A content scales demonstrated significant incremental validity above the clinical scales in predicting clinician ratings of adolescents' behavior and personality characteristics. The clinical scales also demonstrated incremental validity in reference to the content scales, indicating then that the two sets of scales provide complementary information. Magnitude of the additional criterion variance predicted was modest.