Coming to grips with negative evidence for the Comprehensive System for the Rorschach: a comment on Gacono, Loving, and Bodholdt; Ganellen; and Bornstein

The Comprehensive System (CS; Exner, 1991, 1993) for the Rorschach is currently engulfed in controversy. This comment article responds to 3 articles by Rorschach proponents in this issue of the Journal of Personality Assessment. Contrary to the claims of Gacono, Loving, and Bodholdt (this issue), CS scores do not bear a well-demonstrated relationship to psychopathy, antisocial personality disorder, or conduct disorder. Contrary to the claims of Ganellen (this issue), both the original and the revised CS Depression Index (Exner, 1993) bear little or no relationship to depression diagnoses. Furthermore, the scoring reliability of some CS scores is problematic. Although we agree with Bornstein (this issue) that Rorschach scores generally bear little or no relation to psychiatric diagnoses or self-report questionnaires, we believe this lack of relationship tends to disconfirm hypotheses concerning the validity of the Rorschach. In the spirit of the philosopher Sir Karl Popper, the Rorschach community should not minimize negative evidence or engage in post hoc arguments to immunize the CS against falsification.     

Rorschach Comprehensive System data for a sample of 409 adult nonpatients from Brazil

This is a normative study with 409 adult nonpatients living in the state of Sao Paulo, Brazil. The Rorschach was administered by a team of nine psychologists; eight had had further training in the Rorschach method by the Brazilian Rorschach Society and one intensively was prepared by the project coordinator. Of the study participants, 200 lived in the state capital (Sao Paulo) and the other 209 were in other large and small cities in the state, including a coastal city and one in the mountains. Previous psychological or psychiatric treatments were criteria for exclusion. Each protocol was coded independently by two examiners, and then agreement of the two codings was checked. Differences between the two codings were discussed in a meeting of the whole team, which was supervised by the project coordinator to guarantee codification quality control. Upon completion of the codings, an analysis of examiner differences was undertaken, the results of which are in the text. Interrater reliability statistics among examiners were calculated, including percentage of agreement and kappa. Reliability statistics among examiners at the response level are presented as are Comprehensive System (CS; 1999, 2003) findings.     

Scoring accuracy using the comprehensive system for the Rorschach.

The Comprehensive System (CS; Exner, 1974, 1978) for scoring Rorschach responses is the most widely taught and most widely accepted system in use today. The complexity and labor- intensive nature of the CS makes the issue of scoring accuracy a central concern. Twenty-one graduate psychology students and 12 professionals scored 20 Rorschach responses drawn from normal and clinical protocols. In general. accuracy scores for both students and professionals were below acceptable levels. Accuracy scores were clearly better for the code categories of Location, DQ, Pairs, Popular, and Z than for Determinants, FQ, Content, and Special Scores. Responses from clinical protocols were subject to more error. The results suggest that high levels of scoring errors may exist in the field use of the CS. Training standards may need to be devised to insure scoring competence.     

Rorschach Comprehensive System data for a sample of 111 adult nonpatients from Romania

In this study Rorschach data from a Romanian sample of 111 respondents was collected and analyzed in a first attempt to establish national norms. The protocols were collected in a 5-year period (2002-2006) by the author. Interrater reliability statistics are presented for a sample of 20 cases, along with scores for the Rorschach Comprehensive System (CS; Exner, 1993). These results can be used for cross-cultural comparisons of the CS.     

Rorschach Comprehensive System data for a sample of 249 adult nonpatients from Italy

This project provides information on how Italian adult nonpatients perform on the Rorschach test administered and scored following Rorschach Comprehensive System (CS) guidelines (Exner, 1995). Lis, Zennaro, Calvo, and Salcuni (2001) reported previously initial data for 101 of the nonpatients who were administered this instrument between July 1998 and February 2001 by graduate and postgraduate students in a 2-year research course at the Psychotherapy School of the University of Padua. The current study extends that previous work and includes information on an additional 148 participants gathered between April 2001 and March 2004, for a total of 249 individuals. Exclusion criteria are described and interrater reliability statistics at the response level for scoring segments are reported using both percent of agreement and iota. Comprehensive System findings are presented.     

Rorschach Comprehensive System data for a sample of 150 adult nonpatients from Israel

This study is composed of 150 adult nonpatients, all from Israel, who were administered the Rorschach. Inclusion/exclusion criteria are described. Senior graduate students from the clinical psychology program at Bar-Ilan University administered the Rorschach and were provided with extensive supervision and ongoing quality control. Interrater reliability statistics at the response level are presented along with scores for the Rorschach Comprehensive System (CS; Exner, 2005).     

Rorschach Comprehensive System data for a sample of 517 adults from Spain (Barcelona)

This study is composed of 517 adult voluntary participants, from Barcelona and its surroundings, all of whom are from Spain, Barcelona, and Catalonia. The study builds on a previously published report for a smaller sample (N = 250; see Alvarez, Baeza, Campo, Garcia, Guardia, et al., 1993) and is the result of collaboration between students and professors at the Barcelona Rorschach School (Catalan Rorschach Society). Our aim is to offer local data that should help to understand possible differences with existing reference data for the Rorschach Comprehensive System (CS; Exner, 2005), which is the system that has been taught at this school since 1978. Inclusion and exclusion criteria are described and interrater reliability statistics at the response level are presented, along with scores for the CS, followed by a discussion and very brief comparison with other international reference samples.     

How do I score thee? Let me count the ways. Or some different methods of categorizing Rorschach responses

The controversy over the merit of Exner's (1986) Comprehensive System (CS) has unfortunately led to a confusion of the CS with all methods of scoring responses to inkblots. Six other widely used Rorschach scoring methods and representative examples of the research they have generated are described. Objective tests of personality ask participants to acknowledge explicit motives, whereas projective tests sample implicit needs participants may not recognize. Projective methods provide unique means of studying personality dynamics. The CS, whatever its merits and limitations, is but 1 of a number of systems of categorizing Rorschach responses.     

Rorschach Intercoder Reliability for Protocol-Level Comprehensive System Variables in an International Sample

This study examines the intercoder reliability of Rorschach Comprehensive System (CS; Exner, 2001) protocol-level variables. A large international sample was combined to obtain intercoder agreement for 489 Rorschach protocols coded using the CS. Intercoder agreement was calculated using an Iota coefficient, a statistical coefficient similar to kappa that is corrected for chance. Iota values for the variables analyzed ranged from .31 to 1.00, with 2 in the poor range of agreement, 4 in the fair range, 25 in the good range, and 116 in the excellent range of agreement. Discrepancies between variables are discussed.     

Rorschach Comprehensive System data for a sample of 98 adult nonpatients from Greece

Ninety-eight adult nonpatients from Greece were administered the Rorschach according to the Rorschach Comprehensive Systems (CS; Exner, 2003) procedures. These participants were of Greek heritage, were older than 21 years of age, and resided in Greece. Participants were residents of an urban area, Athens, as well as residents of provincial areas including smaller cities and villages. The examiner was a native Greek and of Greek heritage and administered the protocols in Greek. Exclusion criteria are described. Interrater reliability statistics at the response level are presented along with scores for CS variables.     

Rorschach intercoder reliability for protocol-level comprehensive system variables in an international sample

This study examines the intercoder reliability of Rorschach Comprehensive System (CS; Exner, 2001) protocol-level variables. A large international sample was combined to obtain intercoder agreement for 489 Rorschach protocols coded using the CS. Intercoder agreement was calculated using an Iota coefficient, a statistical coefficient similar to kappa that is corrected for chance. Iota values for the variables analyzed ranged from .31 to 1.00, with 2 in the poor range of agreement, 4 in the fair range, 25 in the good range, and 116 in the excellent range of agreement. Discrepancies between variables are discussed.     

Vers des normes francophones pour le Rorschach en système intégré : premiers résultats sur un échantillon de 146 adultes

We present the main features of the European French-Speaking Normative Study for the Rorschach comprehensive system (CS) in adults. After reviewing previous attempts at establishing norms and diverse methodological considerations, we justify the choices made for the present project based on a clear definition of the sample, careful control of recruitment and quality of data collection procedures. Results of phase 1 (N =146) are presented and analyzed. Main variables from the CS are described and some key variables are compared to samples currently used as references (from the US). Differences are observable, although it is too early at this point to draw any conclusions that could influence interpretation. Such conclusions will be formulated when data collection is completed (final objective N =450).     

Rorschach Comprehensive System data for two samples of nonpatient children from Italy: 75 aged 5-7 years and 148 aged 8-11 years

This project provides information on how nonpatient children perform on the Rorschach test, administered and scored following Exner's guidelines (1995). Lis, Parolin, Zennaro, and Mazzeschi (2001) previously reported initial data for 70 nonpatient children living in Italy who were administered this instrument by graduate and postgraduate students in a 2-year research course at the Psychotherapy School of the University of Padua between July 1998 and February 2001. The current study is an extension of that work and includes information on an additional 153 participants gathered between November 2002 and December 2006. The total number of participants includes 223 individuals, 75 5-7-year-old children in the first level of elementary school, and 148 8-11-year-old children in the second level of elementary school. Exclusion criteria are described, and interrater reliability statistics at the response level for scoring segments are reported using percent agreement and iota. Rorschach Comprehensive System (CS) findings are presented.     

Rorschach Comprehensive System data for a sample of 240 adult nonpatients from Japan

This article presents Rorschach Comprehensive System (CS; Exner, 1995) results for 240 adult nonpatients living in the central area of Tokyo, Japan. Inclusion/exclusion criteria are described. Interrater reliability statistics at the response level are presented along with scores for CS variables.     

Rorschach Comprehensive System data for a sample of 41 adult nonpatients from Israel

This study presents data concerning Rorschach protocols of 41 Jewish Israeli students. The sample includes 20 males and 21 females, ranging in age from 19 to 35, who were selected from a larger sample of 50 undergraduates by applying psychiatric exclusion criteria. The results concerning the main Rorschach Comprehensive System's (CS; Exner, 2003) measures of distress (D and Adj D scores, S-Constellation, and DEPI) revealed substantial elevation. These results are analyzed in relation to the constant threat of terror, which the participants are exposed to within the context of the ongoing political conflict in the Middle East. The analysis is supported by data concerning a new Rorschach index, the Reality-Fantasy scale (RFS; Tibon, Handelzalts, & Weinberger, 2005), which applies the CS to psychoanalytic conceptualization of transitional space.     

Rorschach Comprehensive System data for a sample of 283 adult nonpatients from the United States

This article presents an update on adult nonpatients living in central California previously described by Shaffer, Erdberg, and Haroian (1999). In this study an additional 160 nonpatients were administered the Rorschach for a total sample size of 283. Graduate students enrolled in a 2-year Rorschach research seminar administered the Rorschach (Exner, 1995), WAIS-R (Weschler, 1981), and MMPI-2 (Butcher, Dahlstrom, Graham, Tellegen, & Kaemmer, 1989), and were provided with extensive supervision and ongoing quality control. All protocols were recoded by the first two authors and a psychologist with extensive Rorschach experience. Exclusion criteria are described. Interrater reliability statistics at the response level are presented along with scores for Rorschach Comprehensive System (CS), WAIS-R, and MMPI-2 variables.     

Introduction to the JPA special supplement on International Reference Samples for the Rorschach Comprehensive System

This Introduction provides an overview to the JPA Special Supplement on International Reference Samples for the Rorschach Comprehensive System (CS; Exner, 2001). It contains a history of this project and a table including all of the lead authors, their country and the type and size of their sample. Suggestions for conducting this type of research are offered, including information on normative vs. nonpatient samples, inclusion/exclusion criteria, the use of collateral instruments, and concerns relative to representativeness. Reliability and coding concerns are addressed, and information on the Popular response in Japan are reviewed. Finally, trends within and across the data are highlighted.     

The impact of administration and inquiry on Rorschach Comprehensive System protocols in a national reference sample

We investigated the impact of administration and inquiry skills on Rorschach Comprehensive System (CS; Exner, 1974, 1991, 1993) protocols collected for the Italian adult nonpatient reference sample. The same research team collected CS protocols on two occasions. The initial reference sample (N = 212; Lis, Rossi, & Priha, 1998) was collected under the supervision of experienced psychologists who carefully studied CS administration and scoring procedures (Exner, 1986, 1990, 1993). The second sample (N = 101; Lis, Zennaro, Calvo, & Salcuni, 2001) was collected after the team obtained additional and sustained CS training from Rorschach workshops certified instructors. Both samples were scored, reliably but they showed large differences on many codes, with protocols from the second sample being richer and more complex than the first. The results indicate that administration skills can have a dramatic impact on CS protocols and may contribute to variations in samples collected by different investigators. Training standards should be devised to insure uniform administration procedures are followed when collecting CS protocols.