Exploring the Relationship Between Interrater Correlations and Validity of Peer Ratings

A field study was conducted to investigate the relationship between interrater correlations and validity estimates of peer ratings. Validity coefficients and interrater correlations were calculated for 281 work units in a large law enforcement organization in Israel. The main result was a weak positive linear relationship between these two variables. Furthermore, in some of the analyses conducted, a nonlinear quadratic component in the relationship between these measures was evident. Validity was low only when interrater correlation was very low (r = .4 and less). Above this level, validity was stable and almost did not change as interrater correlation increased. This finding, together with other studies (Borman, 1975 Borman, W. C. 1975. Effects of instructions to avoid halo error on reliability and validity of performance evaluation ratings. Journal of Applied Psychology, 60: 556–560. [Crossref], [Web of Science ®] , [Google Scholar]; Buckner, 1959 Buckner, D. N. 1959. The predictability of ratings as a function of interrater agreement. Journal of Applied Psychology, 43: 60–64.  [Google Scholar]; Freeberg, 1969 Freeberg, N. E. 1969. Relevance of rater-ratee acquaintance in the validity and reliability of ratings. Journal of Applied Psychology, 53: 518–524.  [Google Scholar]; Weekley & Gier, 1989 Weekley, J. A. and Gier, J. A. 1989. Ceilings in the reliability and validity of performance ratings: The case of expert raters. Academy of Management Review, 32: 213–222.  [Google Scholar]), cast doubt on the assertion that interrater correlation in the field of performance rating is a proper measurement of reliability.     

Constructing Confidence Intervals for Effect Size Measures of an Indirect Effect

Confidence intervals for an effect size can provide the information about the magnitude of an effect and its precision as well as the binary decision about the existence of an effect. In this study, the performances of five different methods for constructing confidence intervals for ratio effect size measures of an indirect effect were compared in terms of power, coverage rates, Type I error rates, and widths of confidence intervals. The five methods include the percentile bootstrap method, the bias-corrected and accelerated (BCa) bootstrap method, the delta method, the Fieller method, and the Monte Carlo method. The results were discussed with respect to the adequacy of the distributional assumptions and the nature of ratio quantity. The confidence intervals from the five methods showed similar results for samples of more than 500, whereas, for samples of less than 500, the confidence intervals were sufficiently narrow to convey the information about the population effect sizes only when the effect sizes of regression coefficients defining the indirect effect are large.     

“Anxious” and “reckless” drivers

This paper presents personality profiles of two types of recidivist traffic offenders. One is the “anxious” traffic offender who imputes legitimacy to traffic norms, yet due to innate structural personality defects, he tends to enter into a state of anxiety, when confronted with risky traffic situations or a traffic jam. The other type is the “reckless” driver, who does not impute legitimacy to traffic norms nor to other legal norms. From this personality theory a number of hypotheses were derived and tested by means of multiple regression. Our ability to predict the number of traffic offenses suggests that similar tools may be used by the traffic authorities and traffic courts.     

Development and Validation of the Scientific Reasoning Scale

Scientific findings and innovations play an important role in a range of decisions faced by nonscientists, yet little is known about the skills that nonscientists need in order to read and evaluate scientific evidence. Drawing on research in public understanding of science, cognitive developmental psychology, and behavioral decision research, we develop an individual difference measure of scientific reasoning skills, defined as the skills needed to evaluate scientific findings in terms of the factors that determine their quality. We present the results of three studies assessing its psychometric validity. Our results indicate that the Scientific Reasoning Scale (SRS) is internally consistent and distinct from extant measures of scientific literacy. Participants with higher SRS scores are more likely to have beliefs consistent with the scientific consensus on potentially contentious issues, above and beyond education, political and religious beliefs, and scores on two widely used measures of scientific literacy. Participants with higher SRS scores also had better performance on a task requiring them to analyze scientific information. Our results suggest that the SRS provides a theoretically informed contribution to decoding lay responses to scientific results and controversies. Copyright © 2015 John Wiley & Sons, Ltd.     

Association of social-environmental factors with cognitive function in children with sickle cell disease

The aim of this study was to examine the relationship between cognitive function in pediatric sickle cell disease (SCD) patients and mothers’ reports of social-environmental stress, depressive symptoms, and parenting. A total of 65 children with SCD completed comprehensive neuropsychological testing to assess several domains of cognitive functioning, including general intellectual ability, academic achievement, and executive function. Mothers reported on demographics, social-environmental stress, depressive symptoms, and parenting. As predicted, children with SCD significantly underperformed relative to normative data on measures of cognitive function. Associations between maternal social-environmental stress, maternal depressive symptoms, and parenting were mixed. The results show partial support for the hypothesis that greater stress and depressive symptoms and less positive parenting are associated with poorer cognitive function in children with SCD. Linear regression analyses showed that maternal financial stress was the strongest predictor across all domains of cognitive function. The findings replicate and extend past research, reaffirming that children with SCD are at risk for cognitive impairment across multiple domains. Additionally, social-environmental stress, particularly financial strain, is linked to mothers’ depressive symptoms and parenting behaviors as well as children’s cognitive function. Future studies using direct observations of parenting behaviors are needed. These findings, along with recent research on parenting interventions, may inform the development of concrete, teachable parenting and coping skills to improve cognitive functioning in children with SCD.     

Cognitive behavioral therapy in practice: treatment delivered by trainees at an outpatient clinic is clinically effective

Cognitive–behavioral therapy (CBT) is known to be effective for a number of disorders, and can be delivered effectively by trainees in controlled settings. However, the effectiveness of trainee therapists in general practice compared to that of more experienced therapists is unknown. In this study, the authors used a benchmarking strategy to compare the outcomes of naturalistic CBT delivered by trainee therapists to those of efficacy and effectiveness studies using primarily professional therapists. Patients (N = 249) with mood and anxiety disorders were treated by trainees (primarily by interns and postdocs) using nonstandardized nonmanualized CBT at an outpatient clinic in an urban academic medical center. Changes in anxiety and depression were assessed using effect sizes, reliable and clinically significant change, and benchmarked to efficacy and effectiveness studies. Symptoms of anxiety and depression significantly improved from start to end of treatment. Rates of improvement and recovery compared favorably to those achieved in other studies, with the exception of recovery rates in severe depression. Effect sizes were in the medium to large range, but generally lower than those achieved in other studies. Results suggest that CBT can be delivered effectively by trainees in an outpatient setting.     

Natural Mentoring Processes Deter Externalizing Problems Among Rural African American Emerging Adults: A Prospective Analysis

A 3-wave model linking natural mentoring relationships to externalizing behavior was tested with 345 rural African American emerging adults in their final year of high school. Structural equation models were executed linking multi-informant reports of mentor-emerging adult relationship quality with youths' externalizing behavior 18 months later. Consistent with our primary hypotheses, emerging adults whose relationships with their natural mentors were characterized by instrumental and emotional support and affectively positive interactions reported lower levels of anger, rule-breaking behavior, and aggression. These effects emerged independent of the influences of family support and youth gender. Two intrapersonal processes, a future orientation and self-regulation, emerged as mediators of the influence of natural mentoring relationships. The influence of natural mentors was most pronounced for emerging adults experiencing high levels of life stress.     

The Functional Analytic Psychotherapy Rating Scale: a Replication and Extension

Few theories of psychotherapy give direction to the therapist on a moment-to-moment level or make predictions about how specific therapist techniques change client behavior in session. Functional analytic psychotherapy (FAP, Kohlenberg and Tsai in Functional analytic psychotherapy: a guide for creating intense and curative therapeutic relationships. Plenum, New York 1991) provides this direction and prediction. Specifically, FAP claims that client problem behaviors will be displayed in the therapeutic relationship and that the therapist can improve client in-session behavior through differential, contingent responding. Further, it is assumed that these improvements in session can be generalized to out of session relationships. The FAP rating scale (FAPRS) was developed for the purpose of coding in-session client and therapist behaviors in an effort to test FAP’s purported mechanism of change. The current study seeks to replicate and extend initial FAPRS findings (Callaghan et al. in J Contemp Psychother 33:321–339, 2003) regarding mechanism of change and to address transportability. FAPRS coding data from a single successful case of an individual diagnosed with Borderline Personality Disorder and treated with FAP is presented. Results indicate that the FAPRS system is transportable and are generally supportive of the claim that therapist contingent responding leads to client improvement.     

Mothers’ and Fathers’ Responsive Problem Solving with Early Adolescents: Do Gender,Shyness, and Social Acceptance Make a Difference?

We assessed the extent to which youths’ (n = 231) shyness and social acceptance in preadolescence were associated with parents’ responsive problem solving 1 year later after controlling for initial levels of parents’ problem solving. Teachers (n = 176) completed assessments of youths’ shyness and social acceptance, and parents (n = 231 married pairs) completed assessments of their responsive problem solving with the child. For shy daughters, higher levels of social acceptance from peers predicted more responsive problem solving from fathers. Greater social acceptance predicted marginally less maternal problem solving for sons, but greater social acceptance predicted marginally more maternal problem solving with daughters. Results suggest the salience of child gender, shyness, and social acceptance when considering typical levels of parental involvement with youth. Implications for parents include the importance of successful problem solving discussions with emerging adolescents, especially youth who are shy or excluded by peers.     

Generalist and specialist graduate business degrees: Tangible and intangible value

Completion of a graduate degree in business, be it the general MBA or a specialized Master program, should add value to graduates and provide them with certain competencies to make them better managers and leaders. This article evaluates the impact of several Master level studies in business at a large, urban university in the southwestern US. We analyzed competencies, skills, self-perception, and career progress as well as salary and performance. Participant choice of degree program was either the generalist MBA degree or specialist graduate Master degree. Responses from 318 alumni, most of them now working managers, clearly indicate the value they gained from pursuing a graduate business degree. The most prominent impact of competencies and skills portfolio gained was on both internal and external measures of career success, for both the MBA and specialized programs.