Predictive Criterion-Related Validity of Observer Ratings of Personality and Job-Related Competencies Using Multiple Raters and Multiple Performance Criteria

The purpose of this study was to assess the predictive validity of observer ratings of personality and job-related competencies in a selection setting. Based on ratings from multiple raters of both the predictors and the criteria in a sample of MBA students, results indicated that observer ratings of Conscientiousness, Emotional Stability, leadership, and interpersonal skills predicted work performance, team performance, and academic performance. For work performance and team performance, a composite of the four predictors had incremental predictive validity over general mental ability, even after controlling for how well the rater knew the ratee.     

Male Physical Attractiveness as a Potential Contaminating Variable in Ratings of Heterosocial Skill

Abstract

Behavioral and self-report measures of intrinsic task interest of American university students were compared in relation to momentary mood, long-term motivational orientation, and interest in competing activities. Although the two measures of intrinsic motivation were significantly correlated, significant contrasts were found in their correlations with other factors. Results support an earlier proposal that the behavioral (free-choice time) measure depends on how treatments affect interest in alternative activities, not just interest in the target activity. A model of how the two measures differ is outlined.     

Clinical Validity of Prototype Personality Disorder Ratings in Adolescents

A growing body of research shows that personality pathology in adolescents is clinically distinctive and frequently stable into adulthood. A reliable and useful method for rating personality pathology in adolescent patients has the potential to enhance conceptualization, dissemination, and treatment effectiveness. The aim of this study is to examine the clinical validity of a prototype matching approach (derived from the Shedler Westen Assessment Procedure–Adolescent Version) for quantifying personality pathology in an adolescent inpatient sample. Sixty-six adolescent inpatients and their parents or legal guardians completed forms of the Child Behavior Checklist (CBCL) assessing emotional and behavioral problems. Clinical criterion variables including suicide history, substance use, and fights with peers were also assessed. Patients’ individual and group therapists on the inpatient unit completed personality prototype ratings. Prototype diagnoses demonstrated substantial reliability (median intraclass correlation coefficient =.75) across independent ratings from individual and group therapists. Personality prototype ratings correlated with the CBCL scales and clinical criterion variables in anticipated and meaningful ways. As seen in prior research with adult samples, prototype personality ratings show clinical validity across independent clinician raters previously unfamiliar with the approach, and they are meaningfully related to clinical symptoms, behavioral problems, and adaptive functioning.     

Construct Validity of Parent Ratings of Inhibitory Control

Recent literature has emphasized the need to examine executive functions (EF) in children using multiple sources, including both parent rating and performance-based measures. Computerized Go/No-Go tests, including commercially available continuous performance tests (CPTs), represent one of the most commonly used methods of assessing inhibitory control—a variable central to the executive function construct. We examined the relationship between parent ratings of inhibitory control and CPT performance in two mixed clinical samples. Experiment 1 examined 109 children ages 6–18 using the Behavior Rating Inventory of Executive Function (BRIEF; Gioia, Isquith, Guy, & Kenworthy, 2000) and the Conners' CPT-II (Conners, 2000). In this sample, ratings on the BRIEF Inhibit scale (mean T-score?=?62.3) were significantly higher than the CPT-II commissions score (mean T-score?=?50.7; p < .0001); and the BRIEF and CPT-II scores were not highly correlated (r?=???.12). Experiment 2 examined a sample of 131 children ages 7–18 using the BRIEF and the Tests of Variables of Attention (TOVA; Greenberg, 1996). In this sample, parent ratings on the BRIEF Inhibit scale (mean T-score?=?56.8) were similar to TOVA commissions scores (mean T-score?=?58.6; p?=?.33), although still poorly correlated (r?=??.02). Factor analyses exploring covariance between BRIEF scales CPT-II variables (Experiment 1) and between BRIEF and TOVA (Experiment 2) yielded similar findings. In both experiments, all eight BRIEF scales loaded on a single factor, with no overlap with either the CPT-II or the TOVA. In mixed outpatient clinical samples, the BRIEF appears to measure different elements of inhibitory control than those assessed by computerized continuous performance tests.     

Longitudinal Predictive Validity of a Learning Potential Test

The aim of this article is to provide empirical psychometric evidence of the (longitudinal) predictive validity of a learning potential measure—the Learning Potential Computerised Adaptive Test (LPCAT)—in comparison with standard static tests with school aggregate results as the criterion measure. Participants were 79 boys (mean age 12.44, SD = 0.44) and 72 girls (mean age 11.18, SD = 0.42) attending two private schools. Correlation and regression analyses were used to evaluate the predictive validity of the learning potential and standard test scores for school aggregate academic results as criterion measure. Results indicate that learning potential scores were statistically significant predictors of aggregate academic results and provided results that were comparable to those of the standard test results—providing empirical support for the use of learning potential tests in mainstream educational settings.     

Training Family Therapists in a Multicultural Setting

Authors describe a multicultural training model for family therapists and psychologists at a traditionally conservative Afrikaans speaking university in South Africa. Black psychologists are now receiving high priority in a masters degree program that leads to professional registration licensure because of the needs of the population. The authors emphasize the importance of being able to think multidimensionally and creating a context in which students can do the same in such contexts.     

Outcomes of Client Transfers in a Training Setting

Although transferring clients from one clinician to another is a common practice in most training facilities and community clinics, there is a dearth of empirical studies that explore the impact on client retention and subsequent therapeutic gains. This study explored client retention following transfer to a new clinician in a COAMFTE accredited master level marriage and family therapy program in a Midwestern university over a period of five years. The researchers analyzed the relationship between client, clinician, and therapy process variables and client retentions following transfer. Although client and therapist variables were not significantly correlated with successful transfers, the number of sessions missed prior to transfer and the numbers of co-therapy transfer sessions were. We conclude that at least four co-therapy transfer sessions are appropriate to optimize successful transfers.     

Internal Validity of the Disruptive Behavior Disorder Symptoms: Implications from Parent Ratings for a Dimensional Approach to Symptom Validity

A dimensional approach was used to evaluate the internal validity of the DSM-III-R ADHD-inattention, ADHD-hyperactivity/impulsivity, oppositional defiant disorder (ODD), and conduct disorder (CD) symptoms (i.e., whether a symptom has a stronger correlation with its own dimension than the other three). Parents rated 4,019 children between the ages of 2 and 19 on these symptoms. The results showed that 5 of the 6 inattention symptoms, 3 of the 4 hyperactivity symptoms, 1 of the 4 impulsivity symptoms, 6 of the 9 oppositional defiant disorder symptoms, and 8 of the 11 CD symptoms had significant internal validity. Confirmatory factor analysis (CFA) found support for inattention, hyperactivity/impulsivity, oppositional defiant, and conduct disorder dimensions. Multiple-group CFA also found support for factor pattern and loading invariance across gender. The implications of these results as well as the merits of the dimensional approach to symptom validity are discussed in the context of the DSM-IV changes in ADHD, ODD, and CD.     

Validity of Selection/ Classification Variables for Performance in Three Navy Ratings

This study examined speed of advancement and score on a job knowledge test as job performance criteria in the validation of selection and classification variables for three Navy ratings. Longitudinal samples spanning a 4-year period were drawn for men in radioman, machinist's mate, and hospital corpsman ratings who, within their first enlistment, advanced to E-4 by means of the Navy competitive advancement system. Validity coefficients for Navy enlisted selection and classification variables (Armed Services Vocational Aptitude Battery [ASVAB] and education) were computed for personnel in each rating, subgrouped by entry pay grade. Validity coefficients for the job knowledge test were high by historical standards, and the effects were highest for the subgroups with the highest selection standards. Validity coefficients for the advancement criterion were much lower. Advancement, the criterion that appeared to have the greater face validity, was not as good a measure of job performance as the job knowledge test. The criteria were predicted much better by mental ability than by level of education variables. Personnel who enlisted at E-2 and E-3 pay grades spent substantially more time working at the journeyman performance level during their first enlistment than did personnel who enlisted at E-1.     

Predictive Validity of the Suicide Probability Scale in a Male Inmate Population

This paper describes the predictive power of the Suicide Probability Scale (SPS) within a male inmate population. The SPS scores of 1,047 inmates from 1995 to 1996 were compared against their suicidal behaviors in the following 10 years. The SPS probability score and the scores on its four subscales were found to be significantly higher in the group that engaged in at least one of three types of suicidal behavior: suicide, non-lethal self-harm and serious suicidal intentions. However, the SPS did not discriminate the inmates who went on to commit suicide. The SPS’s capacity to predict suicidal behavior proved limited in terms of sensitivity. In this regard, the SPS’s low and high risk levels correctly identified 36% of inmates who later engaged in suicidal behavior. The specificity was 0.85. These results confirm the SPS’s predictive power with male prisoners. These findings are the first to demonstrate the predictive validity of the SPS.     

Validity of Treatment Target Progress Ratings as Indicators of Youth Improvement

We examined whether clinical progress ratings on the Monthly Treatment and Progress Summary form (MTPS), an idiographic treatment progress measure, were meaningfully related to changes measured by two separate standardized instruments; the Child and Adolescent Functional Assessment Scale (CAFAS) and the Child and Adolescent Level of Care Utilization System (CALOCUS). Validity coefficients were examined at intake and three-, six-, and nine-month follow-up periods. Samples were selected for each measure and follow-up period from the population of youth receiving services through the Hawaii Child and Adolescent Mental Health Division’s (CAMHD) system of care. Significant youth improvement was evident on all three measures across all follow-up intervals. The type of changes measured by the MTPS and CAFAS were more alike over longer follow-up intervals than changes measured by the CALOCUS. The MTPS captures distinct aspects of client change that overlaps somewhat with CAFAS and CALOCUS measures. The MTPS is a brief client-tailored measure that seems to provide valid, sensitive, and nonredundant client specific treatment outcome information that can be collected on a frequent basis within a complex system of care.     

Effects of Training Reflective Listening in a Corporate Setting

A communication skills training of insurance company employees was devised and evaluated. Study 1 compared trained and untrained employees on their Reflective listening (RL; Rogers, 1951) skills before and after a 16-hr training, as they conversed with a confederate who rated them, and their responses were categorized. In a follow-up Study 2, some of the employees audiotaped conversations with customers, and their responses were categorized as well. In a validation Study 3, students rated conversations by employees who used more or less RL. Results showed that training increased RL, and that the skills were subsequently transferred to an authentic setting. Results did not show that trained employees were evaluated differently than untrained. Implications of the results are discussed.     

Exploring the Relationship Between Interrater Correlations and Validity of Peer Ratings

A field study was conducted to investigate the relationship between interrater correlations and validity estimates of peer ratings. Validity coefficients and interrater correlations were calculated for 281 work units in a large law enforcement organization in Israel. The main result was a weak positive linear relationship between these two variables. Furthermore, in some of the analyses conducted, a nonlinear quadratic component in the relationship between these measures was evident. Validity was low only when interrater correlation was very low (r = .4 and less). Above this level, validity was stable and almost did not change as interrater correlation increased. This finding, together with other studies (Borman, 1975 Borman, W. C. 1975. Effects of instructions to avoid halo error on reliability and validity of performance evaluation ratings. Journal of Applied Psychology, 60: 556–560. [Crossref], [Web of Science ®] , [Google Scholar]; Buckner, 1959 Buckner, D. N. 1959. The predictability of ratings as a function of interrater agreement. Journal of Applied Psychology, 43: 60–64.  [Google Scholar]; Freeberg, 1969 Freeberg, N. E. 1969. Relevance of rater-ratee acquaintance in the validity and reliability of ratings. Journal of Applied Psychology, 53: 518–524.  [Google Scholar]; Weekley & Gier, 1989 Weekley, J. A. and Gier, J. A. 1989. Ceilings in the reliability and validity of performance ratings: The case of expert raters. Academy of Management Review, 32: 213–222.  [Google Scholar]), cast doubt on the assertion that interrater correlation in the field of performance rating is a proper measurement of reliability.     

Alliance Ratings as a Part of Trainee Evaluations Within Family Therapy Training

Client-reported therapeutic alliance scores were examined as a measure of improvement in therapeutic skills across master's-level marriage and family therapy (MFT) training. The results indicate significant correlations between alliance ratings and training variables, with the number of cumulative clinical hours identified as a significant predictor of alliance ratings according to a regression analysis. Trainees lacking the expected number of clinical contact hours in each level of training were found to be more likely to receive consistently marginal alliance ratings. Additionally, alliance ratings were especially useful in identifying trainees who were functioning at minimal levels of competency.     

Generalizability Theory as Construct-Related Evidence of the Validity of Job Performance Ratings

Investigations of the construct-related evidence of the validity of performance ratings have been rare, perhaps because researchers are dissuaded by the con- siderable amount of evidence needed to show construct validity (Landy, 1986). It is argued that generalizability (G) theory (Cronbach, Gleser, Nanda, & Rajaratnam, 1972) is well-suited to investigations of construct-related evi- dence of validity because a single generalizability investigation may provide multiple inferences of validity. G theory permits the researcher to partition observed score variance into universe (true) score variance and multiple, distinct estimates of error variance. G theory was illustrated through the anal- ysis of proficiency ratings of 256 Air Force jet engine mechanics. Mechanics were rated on three different rating forms by themselves, peers, and supervi- sors. Interpretation of G study variance components revealed suitable evi- dence of construct validity. Ratings within sources were reliable. Proficiency ratings showed strong convergence over rating forms, though not over rating sources. Raters showed adequate discriminant validity across rating dimen- sions. The expectation of convergence over sources was further questioned.     

Fluency Training a Writing Skill: Editing for Concision

The Psychological Record - The goal of this study was to design and evaluate fluency-based training units to help students eliminate inconcision. Participants first completed a 1.5-hr lesson on...