Introduction to the special section on incremental validity and utility in clinical assessment

This special section focuses on the incremental validity and utility of clinical assessment data. The lack of replicated incremental validity research limits the ability of psychologists to establish their assessment practices on a solid empirical footing. The articles in this section deal with conceptual, methodological, and content issues in the development and use of clinical measures. The authors addressed several aspects of incremental validity research, including (a) the evaluation of the magnitude of validity increments, (b) the use of incremental validity data for test development and validation, (c) the costs of assessment, (d) the use of multi-informant and multimethod assessments, and (e) the treatment utility of assessment. Obstacles limiting the use of incremental validity research to inform and guide clinical practice are also emphasized.     

An assessment of social validity trends in applied behavior analysis

Research articles published during the first 31 years of Journal of Applied Behavior Analysis were assessed for reports of social validity measures. Two measures of social validity (treatment outcome and treatment acceptability) were assessed for each article. In addition, articles were assessed for the incorporation of analog versus naturalistic settings. Overall, treatment outcome and acceptability measures were reported in less than 13% of articles. In addition, studies conducted in naturalistic settings were more likely to report social validity measures than studies conducted in analog settings. Possible reasons for the deficit in the reporting of social validity measures and potential outcomes of this deficit are discussed. Copyright © 1999 John Wiley & Sons, Ltd.     

Utilization and predicted trends of behavioral inventories

Published articles (N=1703) during the period 1975 to 1982 in three behavioral journals (Behavior Therapy, Journal of Behavior Therapy and Experimental Psychiatry, andJournal of Applied Behavior Analysis) were reviewed for the utilization of preintervention and categories of behavioral inventories. Results revealed that (a) the three forms of the Fear Survey Schedule (FSS) and the Social Avoidance and Distress Scale (SADS) were the most utilized traditional behavioral measures, (b) new self-developed inventories were reported with significantly (p<.05) different frequencies across the three journals, (c) numerous published inventories beyond the traditional FSS and SADS were reported, and (d) an expanded use of IQ measures in preintervention behavioral assessment was also found. Trends regarding the nature of behavioral inventories among the three journals and a discussion on advances in behavioral assessments are presented.     

Development and psychometric properties of the Children's Assertive Behavior Scale

The purpose of this paper is to present the psychometric properties and individual items comprising the Children's Assertive Behavior Scale (CABS). The CABS is a behaviorally designed self-report instrument for children which measures general and specific social skills and covers many socially relevant situations which are problematic for children. Acceptable psychometric properties of the 27-item test were obtained in several independent investigations across both geographical regions and grade levels. The CABS showed significant concurrent validity with peer, parent, and teacher measures of social competency. The CABS also discriminated trained versus untrained children participating in social skills versus a placebo discussion group. The instrument is presented, along with recommendations for future applications and research.Both authors share equal responsibility for the development and evaluation of the Children's Assertive Behavior Scale.     

Differentiating anxious,aggressive, and socially competent preschool children: validation of the Social Competence and Behavior Evaluation-30 (parent version)

The present study examined the factor structure, internal consistency, and construct validity of the parent version of the Social Competence and Behavior Evaluation-30 for preschoolers (SCBE-30; LaFreniere, P. J. (1990). Social competence and behavior evaluation-30. Unpublished measure.), an adaptation of the validated teacher version of the same measure (LaFreniere & Dumas, Psychol. Asses. 8 (1996) 369). The parent version of the SCBE-30 is a 30-item Likert rating scale questionnaire designed to assess patterns of anxiety/withdrawal, anger/aggression, and social competence. Principal components analysis was used to identify the factor structure of the parent version of the SCBE-30 (N = 218 preschool children). To assess construct validity, a compliance task was utilized to determine whether children identified as high on anxiety/withdrawal, anger/aggression, or social competence with the parent version of the SCBE-30 (n = 20 for each group) could be distinguished behaviorally on several observational variables. Principal components analysis identified three factors accounting for 44% of the variance. Ten items positively loaded onto each factor and matched conceptual expectations. A between-subjects MANOVA demonstrated significant group differences in observed child behaviors including compliance, noncompliance, subtypes of noncompliance, and aversive behavior. Results of the current study suggested that the parent version of the SCBE-30 demonstrated both internal consistency and construct validity, and findings paralleled many of the results from LaFreniere and Dumas' validation of the teacher version of the SCBE-30.     

The Spanish version of the Wisconsin Personality Disorders Inventory-IV (WISPI-IV): tests of validity and reliability

Personality disorders (PD) are a prevalent class of mental disorders that interfere with functioning and cause subjective distress while increasing the intensity and duration of Axis I clinical syndromes, and therefore assessing PD is important even when PDs are not the focus of treatment. The purpose of these studies was to develop and test a new Spanish version of a self-report measure of PD, the Wisconsin Personality Inventory-IV (WISPI-IV) that would be psychometrically equivalent to the English version while also maintaining the same interpersonal content, which is based on Benjamin's analysis of the PD criteria using her Structural Analysis of Social Behavior (SASB) model (1974). Study 1 participants completed the WISPI-IV twice over a two-week interval. For Study 2, participants from two sites in Spain and one site in Argentina completed Spanish versions of the WISPI-IV and other personality measures. SASB-analysis of the translated items showed high correspondence between the interpersonal content of the English version and the Spanish version demonstrating theoretical validation in relation to other PD measures. The Spanish WISPI-IV showed satisfactory reliability based on test-retest correlations and alphas for internal consistency. Study 2 showed the Spanish WISPI-IV had good convergent validity with the Spanish versions of the IIP and SCID-II and performed similarly to the English versions of these measures. Our goal in translating the WISPI-IV was to extend this measure to Spanish-speakers in language that would be understood by different Hispanic sub-groups, however research team members and subjects had a variety of suggestions for changes in item wording. This reflects the difficulty with creating a "neutral" Spanish version of any assessment given regional differences.     

PERSONALITY MEASURES AS PREDICTORS OF JOB PERFORMANCE: A META-ANALYTIC REVIEW

The purpose of this study was to investigate conflicting findings in previous research on personality and job performance. Meta-analysis was used to (a) assess the overall validity of personality measures as predictors of job performance, (b) investigate the moderating effects of several study characteristics on personality scale validity, and (c) appraise the predictability of job performance as a function of eight distinct categories of personality content, including the "Big Five" personality factors. Based on review of 494 studies, usable results were identified for 97 independent samples (total N = 13,521). Consistent with predictions, studies using confirmatory research strategies produced a corrected mean personality scale validity (.29) that was more than twice as high as that based on studies adopting exploratory strategies (.12). An even higher mean validity (.38) was obtained based on studies using job analysis explicitly in the selection of personality measures. Validities were also found to be higher in longer tenured samples and in published articles versus dissertations. Corrected mean validities for the "Big Five" factors ranged from .16 for Extroversion to .33 for Agreeableness. Weaknesses in the reporting of validation study characteristics are noted, and recommendations for future research in this area are provided. Contrary to conclusions of certain past reviews, the present findings provide some grounds for optimism concerning the use of personality measures in employee selection.     

Student-teacher concordance on dimensions of student social competence: A multitrait-multimethod analysis

Teacher and self-ratings of social competence were obtained for 669 (341 males, 328 females) grades 3–6 students using the Social Behavior Assessment—Revised (SBA-R) and a self-report scale consisting of the identical items reworded in first person. Factor analysis and multitrait-multimethod matrix analysis were used to validate construct interpretations. The previous four-factor structure of the SBA-R was not replicated. However, the three-factor structure that did emerge was better defined. Although student-teacher concordance was moderate for the Self-Control/Social Convention and Academic Responsibility factors, it was minimal for the Social Participation factor. Evidence of discriminant validity was weak. As with other studies using rating scales, method bias was in evidence. Sex differences were found in terms of variance explained by each factor and convergent validity. Discussion focuses on problems associated with construct validation in general and validation of school-related social competence in particular.     

It Was Nice to Be There: Construct Validity Then and Now

I appear to be one of the few persons who attended both the colloquium at Princeton titled Construct Validity in Psychological Measurement held in 1979 and the conference at Newport Beach titled Construct Validity: Issues and Opportunities held in 1990. As a discussant for both meetings, I have compared construct validity as seen by participants at the 1979 meetings with construct validity as seen by participants at the 1990 meeting. Comparisons are drawn according to conceptualizations of construct validity, construct va- lidity as scientific inquiry, and construct validity in assessment technology. A most noteworthy conclusion is that these three aspects no longer apply very well to the articles herein that were originally delivered at the 1990 meeting. Instead these articles show strongly that classification according to scientific inquiry versus applications is not possible. Such a conceptual gap no longer seems to exist for construct validity. Instead, these articles as well as the one by Hogan (1991) all emphasize large scale programmatic efforts characterized by approaches driven primarily by conceptualizations derived from construct validity. A secondary theme in these articles emphasizes analysis according to taxonomies at both the predictor and criterion levels. A primary conclusion from reviewing these articles is that construct validity no longer revolves around issues, but it clearly does connote opportunities for future research and applications.     

Citation Analysis of Skinner's Verbal Behavior: 1984-2004

The present study undertook an updated citation analysis of Skinner's (1957)Verbal Behavior. All articles that cited Verbal Behavior between 1984 and 2004 were recorded and content analyzed into one of five categories; four empirical and one nonempirical. Of the empirical categories, studies that employed a verbal operant from Skinner's analysis were assigned to either basic, applied, or observational categories. Empirical studies that did not employ a verbal operant were categorized as other-empirical. The total number of citations remained stable across the review period and averaged just over 52 per year. Of these, 80% were from nonempirical articles, 13.7% were from other-empirical articles, 4% were from applied articles, 1.4% were from basic articles, and 0.9% were from observational articles. An "obliteration" analysis was also conducted to identify articles that employed Skinner's verbal operant terms but did not cite Verbal Behavior. This analysis identified 44 additional articles, suggesting that a degree of obliteration had occurred in the half century since the publication of Verbal Behavior. In particular, the analysis suggests that the verbal operant of manding has sufficient presence in the applied empirical literature to render citation of Verbal Behavior redundant. Overall, Verbal Behavior continues to make an important contribution to the psychological literature.     

Psychometric Properties of the Social Phobia and Anxiety Inventory, the Social Anxiety Scale for Adolescents, the Fear of Negative Evaluation Scale, and the Social Avoidance and Distress Scale in an Adolescent Spanish-Speaking Sample

The relationship among several social anxiety measures and a semistructured interview in an adolescent Spanish-speaking sample is examined. Construct validity and test-retest reliability were tested. A principal axis factor analysis was also explored. Results revealed good construct validity and alpha coefficients for the assessment instruments such as the Social Phobia and Anxiety Inventory (SPAI), the Social Anxiety Scale for Adolescents (SAS-A), the Fear of Negative Evaluation Scale (FNES) and the Social Avoidance Distress Scale (SADS). Among these, data strongly support the validity of the Social Phobia and Difference measures of the SPAI and Total SAS-A score as assessment measures in the adolescent population even in non-American cultures and languages. Furthermore, results appear to support the presence of a single higher-order dimension, social anxiety, as measured by the instruments used in this study.     

Psychometric and Validity Issues in Machine Learning Approaches to Personality Assessment: A Focus on Social Media Text Mining

In the age of big data, substantial research is now moving toward using digital footprints like social media text data to assess personality. Nevertheless, there are concerns and questions regarding the psychometric and validity evidence of such approaches. We seek to address this issue by focusing on social media text data and (i) conducting a review of psychometric validation efforts in social media text mining (SMTM) for personality assessment and discussing additional work that needs to be done; (ii) considering additional validity issues from the standpoint of reference (i.e. ‘ground truth’) and causality (i.e. how personality determines variations in scores derived from SMTM); and (iii) discussing the unique issues of generalizability when validating SMTM for personality assessment across different social media platforms and populations. In doing so, we explicate the key validity and validation issues that need to be considered as a field to advance SMTM for personality assessment, and, more generally, machine learning personality assessment methods. © 2020 European Association of Personality Psychology     

The Social Phobia Inventory: validity and reliability in an adolescent community sample

Social anxiety disorder and social anxiety are highly prevalent psychological conditions in youth. Because of the known risks associated with the spectrum of social anxiety in youth, early detection with valid and reliable assessment measures is imperative. Self-report measures have become the most widely used method of assessment for child and adolescent social anxiety. Because research to date on self-report measures of social anxiety disorder in adolescents is limited, the primary objective of the present study was to contribute to the literature by examining the validity and reliability of a new self-report measure, the Social Phobia Inventory (SPIN), in a community sample of adolescents. The SPIN is a 17-item measure purported to assess the full spectrum of social anxiety disorder symptomatology. Previously, psychometric research on the SPIN in adult populations has demonstrated its validity and reliability. In the current psychometric examination, results revealed strong support for the temporal stability, internal consistency, and construct validity of the SPIN, suggesting it is an appropriate screening measure for the assessment of social anxiety disorder in adolescents.     

Introduction to the special section on clinical applications of analogue behavioral observation

This Special Section addresses methods, validity, and utility of analogue behavioral observation. Separate contributions to the Special Section cover analogue behavioral observation of marital interaction, child behavior problems, parent-child interaction, and adult social functioning. Additional articles address psychometric foundations of analogue behavioral observations, general issues, and future directions in the development and evaluation of this assessment method. Many published studies were reviewed in detail and issues of validity, clinical assessment utility, and sources of variance in obtained measures are addressed.     

The Social Behavior Scale for Preschool Children: Factorial Structure and Concurrent Validity

The aims of the present study were to examine the factor validity of a new social behavior assessment - the Social Behavior Scale (SBS) - using exploratory (sample 1, n = 169) and confirmatory (sample 2, n = 127) factor analysis, and to test factor invariance across gender in a sample of Italian preschoolers. The concurrent validity of the SBS was examined with measures of peer nominations and observed interactions (sample 3, n = 277). Findings confirmed the three-factor structure of the SBS: Emotional Competence, Social Engagement and Aggressiveness. Multigroup analysis provided evidence of factor invariance across gender. Concerning concurrent validity, results confirmed a great association between teacher-rating SBS and the other concurrent measures.     

Validity and structure of a self-report measure of youth psychopathy

An interest in early identification of youth who may be at risk of psychopathy has generated measures for age-appropriate screening and assessment. This study examines the structural, concurrent, and divergent validity of the Psychopathy Content Scale, a 20-item self-report instrument derived from the Millon Adolescent Clinical Inventory (MACI). Data for 481 youths who had taken the MACI, the Child Behavior Checklist Youth Self-Report (YSR), and the Massachusetts Youth Screening Instrument, Second Version (MAYSI-2), were analyzed. Results showed that the PCS is best described by a two-factor model and that analyses of the two factors offer limited support for convergent and divergent validity. High scores on both factors were associated with high YSR and MAYSI-2 scales, suggesting that the measure identified youth who were distressed on several measures of emotional, psychological, and behavioral disorder.     

A Dialectic on Validity: Where We Have Been and Where We Are Going

It has been suggested that the primary issue in considerations of validity today is the discrepancy between currently accepted theory and the practice of validation. The reason for this discrepancy may be that although considerable discussion has taken place in educational measurement and psychometrics regarding changes in our conceptions of validity, that has not been the case more generally in the social and behavioral sciences. A dialectic on the current status of validity theory for social and behavioral scientists is presented to help bridge this gap.     

THE MEASUREMENT OF VERBAL PREDISPOSITIONS: SCALE DEVELOPMENT AND APPLICATION

A self-report instrument (PVB), designed to measure Predispositions toward Verbal Behavior, was found to have acceptable internal reliability, content validity, moderate construct validity, and strong criterion-related validity. Subjects were found to have a generalized cognitive orientation toward their speech patterning that correlated positively with number of words and duration of talk in both highly-structured interviews and unstructured discussion situations.     

METAANALYSES OF VALIDITY STUDIES PUBLISHED BETWEEN 1964 AND 1982 AND THE INVESTIGATION OF STUDY CHARACTERISTICS

Review and metaanalyses of published validation studies for the years 1964-1982 of Journal of Applied Psychology and Personnel Psychology were undertaken to examine the effect of (1) research design; (2) criterion used; (3) type of selection instrument used; (4) occupational group studies; and (5) predictor-criterion combination on the level of observed validity coefficients. Results indicate that concurrent validation designs produce validity coefficients roughly equivalent to those obtained in predictive validation designs and that both of these designs produce higher validity coefficients than does a predictive design which includes use of the selection instrument. Of the criteria examined, performance rating criteria generally produced lower validity coefficients than did the use of other more "objective" criteria. In comparing the validities of various types of predictors, it was found cognitive ability tests were not superior to other predictors such as assessment centers, work samples, and supervisory/peer evaluations as has been found in previous metaanalytic work. Personality measures were clearly less valid. Compared to previous validity generalization work, much unexplained variance in validity coefficients remained after corrections for differences in sample size. Finally, the studies reviewed were deficient for our purposes with respect to the data reported. Selection ratios, standard deviations, reliabilities, predictor and criterion intercorrelations were rarely and inconsistently reported. There are also many predictor-criterion relationships for which very few validation efforts have been undertaken.     

Dimensions of social competence: Method factors in the assessment of adaptive behavior, social skills, and peer acceptance

Dimensions of social competence were investigated by several measures of adaptive behavior, social skills, and peer acceptance. Principal components analysis of the measures revealed that the method of measurement was clearly the most important factor in the assessment of social competence. The five-factor solution to the principal components analysis was defined by the five methods used to operationalize social competence (i.e., method factors). Correlation analysis showed relatively weak relations among the same social competence domains measured by different methods. The results are discussed in the context of convergent and discriminant validity and the interacting influences of method, setting, and content on the assessment of social competence.