Addressing Issues in the Development and Use of the Composite International Reference Values as Rorschach Norms for Adults

This article describes 3 studies evaluating normative reference data for the Rorschach Comprehensive System (CS; Exner, 2003, 2007), with a particular focus on the viability of the Composite International Reference Values (CIRVs) that were compiled from 21 adult studies by Meyer, Erdberg, and Shaffer (2007). Study 1 documented how the CIRV norms are virtually identical when organized into 3 groups differentiated by the quality of their data collection effort, including an optimal group of 4 samples that relied on multiple experienced examiners and provided ongoing quality control over administration and coding. Analyses also showed that relative to the group of more optimal samples, the group of less optimal samples did not produce more variability in summary scores within or across samples or lower interrater reliability for coding. Study 2 used the existing CS reference norms to generate T scores for the CIRV means and documented how the CS norms make other samples of healthy nonpatients look psychologically impaired in multiple domains. Study 3 documented with examples from 4 different countries how 2 sets of within-country local norms produced notably different results on some variables, which compromises the ability of local norms to be used instead of the CIRVs. Taken together, the 3 studies provide support for the use of CIRVs in clinical practice as norms that are generalizable across samples, settings, languages, and cultures and that account for the natural variability that is present when clinicians and researchers contend with the ambiguity contained in the standard CS reference materials concerning the proper ways to administer and code. We conclude by urging CS users to rely on the CIRVs when making clinical inferences and to adopt alternative methods of ensuring they are following cohesively standardized administration and coding guidelines.     

Rorschach Comprehensive System data for a sample of 409 adult nonpatients from Brazil

This is a normative study with 409 adult nonpatients living in the state of Sao Paulo, Brazil. The Rorschach was administered by a team of nine psychologists; eight had had further training in the Rorschach method by the Brazilian Rorschach Society and one intensively was prepared by the project coordinator. Of the study participants, 200 lived in the state capital (Sao Paulo) and the other 209 were in other large and small cities in the state, including a coastal city and one in the mountains. Previous psychological or psychiatric treatments were criteria for exclusion. Each protocol was coded independently by two examiners, and then agreement of the two codings was checked. Differences between the two codings were discussed in a meeting of the whole team, which was supervised by the project coordinator to guarantee codification quality control. Upon completion of the codings, an analysis of examiner differences was undertaken, the results of which are in the text. Interrater reliability statistics among examiners were calculated, including percentage of agreement and kappa. Reliability statistics among examiners at the response level are presented as are Comprehensive System (CS; 1999, 2003) findings.     

Rorschach Comprehensive System data for a sample of 506 adult nonpatients from Argentina

This study is composed of 506 nonpatient adults, ages 18 to 65, all from Argentina. The data were gathered by five examiners, who tested most of the participants at their work site. Inclusion and exclusion criteria are described and interrater reliability statistics at the response level are presented along with findings for the Rorschach Comprehensive System (CS; Exner, 1994).     

Content Analysis of Counselor and Teacher Responses

The findings of this report were based on content analysis of written responses made by experienced and inexperienced counselors, and experienced and inexperienced teachers. Responses of the 4 groups were elicited with an instrument containing 4 open-end questions, administered after viewing a film. The written responses were coded in 8 categories. An estimate of reliability was determined for the coding in 5 of the categories and found to be uniformly high. Ratios were computed for the written responses of each respondent in the eight categories. The data were treated to determine the significance of differences between the four groups. Counselors were found to use a style that is ambiguous and characterized by more sophisticated vocabulary.     

A Web-based program for coding open-ended response protocols

A Web-based coding application was designed to improve coding efficiency and to provide a systematic means of evaluating responses to open-ended assessments. The system was developed for use by multiple raters to assign open-ended responses to predetermined categories. The application provides a software environment for efficiently supervising the work of coders and evaluating the quality of the coding by (1) systematically presenting open-ended responses to coders, (2) tracking each coder’s categorized responses, and (3) assessing interrater consistency at any time in order to identify coders in need of further training. In addition, the application can be set to automatically assign repeated responses to categories previously identified as appropriate for those responses. To evaluate the efficacy of the coding application and to determine the statistical reliability of coding open-ended data within this application, we examined data from two empirical studies. The results demonstrated substantial interrater agreement on items assigned to various categories across free and controlled association tasks. Overall, this new coding application provides a feasible method of reliably coding open-ended data and makes the task of coding these data more manageable.     

Rorschach Comprehensive System data for two samples of nonpatient children from Italy: 75 aged 5-7 years and 148 aged 8-11 years

This project provides information on how nonpatient children perform on the Rorschach test, administered and scored following Exner's guidelines (1995). Lis, Parolin, Zennaro, and Mazzeschi (2001) previously reported initial data for 70 nonpatient children living in Italy who were administered this instrument by graduate and postgraduate students in a 2-year research course at the Psychotherapy School of the University of Padua between July 1998 and February 2001. The current study is an extension of that work and includes information on an additional 153 participants gathered between November 2002 and December 2006. The total number of participants includes 223 individuals, 75 5-7-year-old children in the first level of elementary school, and 148 8-11-year-old children in the second level of elementary school. Exclusion criteria are described, and interrater reliability statistics at the response level for scoring segments are reported using percent agreement and iota. Rorschach Comprehensive System (CS) findings are presented.     

Rorschach Comprehensive System data for a sample of 249 adult nonpatients from Italy

This project provides information on how Italian adult nonpatients perform on the Rorschach test administered and scored following Rorschach Comprehensive System (CS) guidelines (Exner, 1995). Lis, Zennaro, Calvo, and Salcuni (2001) reported previously initial data for 101 of the nonpatients who were administered this instrument between July 1998 and February 2001 by graduate and postgraduate students in a 2-year research course at the Psychotherapy School of the University of Padua. The current study extends that previous work and includes information on an additional 148 participants gathered between April 2001 and March 2004, for a total of 249 individuals. Exclusion criteria are described and interrater reliability statistics at the response level for scoring segments are reported using both percent of agreement and iota. Comprehensive System findings are presented.     

Two Kinds of Learning

Episodic analysis is the separation of a novel into its component episodes —i.e., into short stories each of which has a unitary theme—in order to provide equivalent units for content analysis. Independent coders show two types of disagreement. Neither is serious for content analysis, and both may be minimized by some suggested ground rules. In a study of 13 stylistically different books, intercoder reliability ranged from 76% to 94%. Variations in accuracy were clearly related to literary style. Relevant information on the coding process and reliability, as well as data on the number and length of episodes in the 13 books, are given in tabular form. Problems connected with reliable coding are discussed and some solutions are suggested.     

The Marital Communication Rating Schedule: An instrument for clinical assessment

The Marital Communication Rating Schedule (MCRaS) is presented as an observationally based clinical rating system for assessing verbal behavior in marital communication. Data from 35 response display discussions lasting from 20 to 30 min each, which took place between 11 married couples, were used to examine aspects of the reliability and validity of the instrument. Three raters made independent ratings of 37 MCRaS categories for each husband and wife for each discussion period. Reliability among the raters was shown to be high when calculated within one scale point. Concurrent validity was assessed by comparing MCRaS ratings for four categories with observationally based validation criteria independently coded and measured. Results indicated that for three categories — negative statements, overgeneralizations, and amount of talk — ratings produced results that were similar to those yielded by laborious coding of audiotapes. For one category, opinions requested, a relationship between the ratings and coded data was not found. The validation results were discussed in terms of possible differences in the basis of ratings for the categories subjected to validation. Although further research is needed, it was concluded that MCRaS has many of the desirable qualities needed in a clinically useful, observationally based rating system.This investigation was conducted in connection with the Sociobehavioral Research Project at The University of Michigan when Joyce Borkin and Claude L. Walter were affiliated with the project.     

Rorschach Comprehensive System data for a sample of 111 adult nonpatients from Romania

In this study Rorschach data from a Romanian sample of 111 respondents was collected and analyzed in a first attempt to establish national norms. The protocols were collected in a 5-year period (2002-2006) by the author. Interrater reliability statistics are presented for a sample of 20 cases, along with scores for the Rorschach Comprehensive System (CS; Exner, 1993). These results can be used for cross-cultural comparisons of the CS.     

Measuring Mental Toughness in Sport: A Psychometric Examination of the Psychological Performance Inventory–A and Its Predecessor

The Rorschach Inkblot Method (RIM) is a performance-based personality assessment instrument used in both clinical and research settings worldwide. This investigation examines response-level, interrater reliability of U.S. graduate students enrolled in the same doctoral program and divided into two unique participant groups: 20 of those with previous Comprehensive System (CS) training background who are now trained in Rorschach Performance Assessment System (R-PAS; initial coders N = 11, blind coders N = 9), and 19 of those trained solely with R-PAS (initial coders N = 10, blind coders N = 9). Trainees’ coding was analyzed based on a sample of 50 protocols (half per group) representing a total of 1,189 responses. Blind coding was used among trainees and coding between the two investigated training groups was analyzed for each variable at the response level and expressed as percent agreement, Gwet's AC, and kappa values. Rates of agreement between the trainees were similar across both groups, and most of the concordance rates can be characterized as excellent. Overall, these results suggest that trainees may learn how to code R-PAS variables with similar reliability levels whether or not they had prior CS knowledge.     

Stability of scores for the Slosson Full-Range Intelligence Test

The test-retest stability of the Slosson Full-Range Intelligence Test by Algozzine, Eaves, Mann, and Vance was investigated with test scores from a sample of 103 students. With a mean interval of 13.7 mo. and different examiners for each of the two test administrations, the test-retest reliability coefficients for the Full-Range IQ, Verbal Reasoning, Abstract Reasoning, Quantitative Reasoning, and Memory were .93, .85, .80, .80, and .83, respectively. Mean differences from the test-retest scores were not statistically significantly different for any of the scales. Results suggest that Slosson scores are stable over time even when different examiners administer the test.     

The interclinician reliability of Rorschach interpretation in four data sets

To examine agreement on Rorschach Comprehensive System (CS; Exner, 2004) interpretations, 55 patient protocols were interpreted by 3 to 8 clinicians across 4 data sets on a representative set of 29 characteristics. Substantial reliability was observed across data sets, although a problematic design produced lower results in one. Unexpectedly, a Q-sort task had slightly lower reliability than a simple rating task. As expected, scales that summarized judgments had higher agreement than judgments to individual interpretive statements, and some clinicians produced more generalizable inferences than others. Interpretations for all clinicians were more strongly associated with patients' psychometric true scores (aggregated judgment M range = .82 to .92) than with the judgments of other clinicians (range = .76 to .89). Compared to meta-analyses of interrater reliability in psychology and medicine, the findings indicate these clinicians could reliably interpret Rorschach CS data.     

Rorschach Comprehensive System data for a sample of 116 preadolescent and 117 adolescent nonpatients from Italy

This project provides information on how preadolescent and adolescent nonpatients perform on the Rorschach test, administered and scored following Exner's guidelines (2003). Lis, Salcuni, Parolin, and Superchi (2001) reported previously initial data for 51 adolescent nonpatients living in Italy who were administered this instrument by graduate and postgraduate students in a 2-year research course at the Psychotherapy School of the University of Padua, between July 1998 and February 2001. The current study is an extension of that work and includes information on additional adolescent nonpatients and preadolescent nonpatients, gathered between April 2001 and December 2006. The study thus includes information on 233 participants, 116 preadolescents (aged 12-14) and 117 adolescents (aged 15-18). Individuals were excluded if they had a major medical illness in the past 6 months, any psychiatric hospitalization, psychological treatment within the past 2 years, any felony conviction, or psychological testing within the past year. Interrater reliability statistics at the response level for scoring segments are reported both with percentage of agreement and iota. Rorschach Comprehensive System (CS) findings are presented.     

Observer-Rated Measures of Defense Mechanisms

Observer-rated measures of defense mechanisms are the direct lineal descendants of Freud's clinical proposition that an observer can reasonably infer defensive operations in an individual of which the individual himself or herself is unaware. Assessment methods of defenses have used both projective testing and clinical interview data. Recent Rorschach test methods have focused on borderline and neurotic-level defenses used to discriminate diagnostic groups and predict other aspects of functioning. A TAT method has demonstrated some evidence for age progression in the development of defensive functioning and for regression in defensive functioning following psychological stress. While interrater reliability is generally good, the methods are limited by the availability of the testing situation and the need to demonstrate generalization to external situations.
Following the early introduction of systematic methods by Haan and Vaillant, a number of methods use clinical interview data to assess specific defenses and overall defensive functioning. These methods identify defenses either as qualitative categories, rank-ordered scores by Q-sort, or actual counts of each instance a defense is used. Summary scores are generally of good reliability, with wider variation for individual defenses. There is wide variation on training required, ease of use, and levels of validation.
A review of some correlates of each method suggests that the instruments have a robust potential for use in fundamental and applied clinical research. While each method has some clear applications, the quantitative clinical interview methods appear best adapted to microanalytic process research on issues of defensive change in treatment.     

Factor analysis of the English version of the Kategoriensystem für Partnerschaftliche Interaktion [Interaction Coding System

The Kategoriensystem für Partnerschaftliche Interaktion (KPI), or Interaction Coding System, is an observational coding system used to study behaviors occurring during dyadic interactions. The KPI offers several summary categories (e.g. positive, negative, neutral) to facilitate data analysis, but empirical evidence supporting these rationally derived groupings is lacking. Reported here are the results of a factor analysis of data obtained from 121 spouses or other close relatives of patients with anxiety disorders who participated in dyadic problem-solving interactions with the patients. A three-factor solution accounting for 50% of the variance emerged. The factors were Negativity, Problem-Solving Focus and Emotional Engagement. Confidence in the reliability of the first two factors is bolstered by their similarity to factors detected in prior research on a different coding system, the Marital Interaction Coding System.     

Reasons for Living versus Reasons for Dying: Examining the Internal Debate of Suicide

The Reasons for Living vs. Reasons for Dying (RFL/RFD) Assessment was used to obtain suicidal outpatients' top five reasons for living and for dying, respectively. Forty-nine suicidal university counseling center patients provided 173 RFL and 145 RFD responses. These responses were organized into eight RFL coding categories and nine RFD coding categories. Two coders trained in the RFL/RFD coding system showed high levels of inter-rater reliability (K_RFL = .81; K_RFD = .80). Chi-square results for RFL and RFD coding categories showed that the coding categories were not equally salient to these suicidal patients.     

Describing suicidality: an investigation of qualitative SSF responses

Given the incidence and seriousness of suicidality in clinical practice, the need for new and better ways to assess suicide risk is clear. While there are many published assessment instruments in the literature, survey data suggest that these measure are not widely used. One possible explanation is that current quantitatively developed assessment instruments may fail to capture something essential about the suicidal patient's experience. The current exploratory study examined a range of open ended qualitative written responses made by suicidal outpatients to five assessment prompts from the Suicide Status Form (SSF)--psychological pain, press, perturbation, hopelessness, and self-hate. Two different samples of suicidal outpatients seeking treatment, including suicidal college students (n = 119) and active duty U.S. Air Force personnel (n = 33), provided a wide range of written responses to the five SSF prompts. A qualitative coding manual was developed through a step-by-step methodology; two naive coders were trained to use the coding system and were able to sort all the patients' written responses into the content categories with very high interrater reliability (Kappa > .80). Certain written qualitative responses of the patients were more frequent than others, both within and across the five SSF constructs. Among a range of specific exploratory findings, one general finding was that two thirds of the 636 obtained written responses could be reliably categorized under four major content headings: relational (22%), role responsibilities (20%), self (15%), and unpleasant internal states (10%). Theoretical, research, and clinical implications of the methodology and data are discussed.