Addressing Issues in the Development and Use of the Composite International Reference Values as Rorschach Norms for Adults

This article describes 3 studies evaluating normative reference data for the Rorschach Comprehensive System (CS; Exner, 2003, 2007), with a particular focus on the viability of the Composite International Reference Values (CIRVs) that were compiled from 21 adult studies by Meyer, Erdberg, and Shaffer (2007). Study 1 documented how the CIRV norms are virtually identical when organized into 3 groups differentiated by the quality of their data collection effort, including an optimal group of 4 samples that relied on multiple experienced examiners and provided ongoing quality control over administration and coding. Analyses also showed that relative to the group of more optimal samples, the group of less optimal samples did not produce more variability in summary scores within or across samples or lower interrater reliability for coding. Study 2 used the existing CS reference norms to generate T scores for the CIRV means and documented how the CS norms make other samples of healthy nonpatients look psychologically impaired in multiple domains. Study 3 documented with examples from 4 different countries how 2 sets of within-country local norms produced notably different results on some variables, which compromises the ability of local norms to be used instead of the CIRVs. Taken together, the 3 studies provide support for the use of CIRVs in clinical practice as norms that are generalizable across samples, settings, languages, and cultures and that account for the natural variability that is present when clinicians and researchers contend with the ambiguity contained in the standard CS reference materials concerning the proper ways to administer and code. We conclude by urging CS users to rely on the CIRVs when making clinical inferences and to adopt alternative methods of ensuring they are following cohesively standardized administration and coding guidelines.     

An examination of interrater reliability for scoring the Rorschach Comprehensive System in eight data sets

In this article, we describe interrater reliability for the Comprehensive System (CS; Exner. 1993) in 8 relatively large samples, including (a) students, (b) experienced re- searchers, (c) clinicians, (d) clinicians and then researchers, (e) a composite clinical sample (i.e., a to d), and 3 samples in which randomly generated erroneous scores were substituted for (f) 10%, (g) 20%, or (h) 30% of the original responses. Across samples, 133 to 143 statistically stable CS scores had excellent reliability, with median intraclass correlations of.85, .96, .97, .95, .93, .95, .89, and .82, respectively. We also demonstrate reliability findings from this study closely match the results derived from a synthesis of prior research, CS summary scores are more reliable than scores assigned to individual responses, small samples are more likely to generate unstable and lower reliability estimates, and Meyer's (1997a) procedures for estimating response segment reliability were accurate. The CS can be scored reliably, but because scoring is the result of coder skills clinicians must conscientiously monitor their accuracy.     

Beyond “Objective” and “Projective”: A Logical System for Classifying Psychological Tests: Comment on Meyer and Kurtz (2006)

Using a diverse sample of 4,786 protocols obtained with the Rorschach Performance Assessment System (Meyer, Viglione, Mihura, Erard, &; Erdberg, 2011), we provide a contemporary overview of how people organize Rorschach's inkblots into identifiable regions while formulating responses. After examining how frequently each location was used across all cards in this sample, we examined the consistency of their use by computing parallel information in 17 samples (N = 4,701) obtained using the Comprehensive System (CS; Exner, 2003), including clinical, nonclinical, and adult, child, and adolescent data. Even though the CS data could only record a single location for each response, the average correlation of location use across samples was .96. The results also document continuous dimensionality in use rather than any discontinuities that would demarcate a boundary between common and uncommon locations. Implications of this notable reproducibility and dimensionality are discussed for future conceptualization of location typicality, including location coding considerations and possibilities for improved measures of perceptual fit.     

The impact of administration and inquiry on Rorschach Comprehensive System protocols in a national reference sample

We investigated the impact of administration and inquiry skills on Rorschach Comprehensive System (CS; Exner, 1974, 1991, 1993) protocols collected for the Italian adult nonpatient reference sample. The same research team collected CS protocols on two occasions. The initial reference sample (N = 212; Lis, Rossi, & Priha, 1998) was collected under the supervision of experienced psychologists who carefully studied CS administration and scoring procedures (Exner, 1986, 1990, 1993). The second sample (N = 101; Lis, Zennaro, Calvo, & Salcuni, 2001) was collected after the team obtained additional and sustained CS training from Rorschach workshops certified instructors. Both samples were scored, reliably but they showed large differences on many codes, with protocols from the second sample being richer and more complex than the first. The results indicate that administration skills can have a dramatic impact on CS protocols and may contribute to variations in samples collected by different investigators. Training standards should be devised to insure uniform administration procedures are followed when collecting CS protocols.     

Rorschach Comprehensive System data for a sample of 100 adult nonpatients from the Belgian French-speaking community

This article presents normative data for the Rorschach Comprehensive System (CS) on 100 adult nonpatients from the Belgian French-speaking community. We compare our Belgian sample to a French one (Sultan et al., 2004), data of Exner's (Exner & Erdberg, 2005) and a Californian sample (Shaffer, Erdberg, & Haroian, 1999). Each sample's specific characteristics are explored, as well as findings provided by sample-specific comparisons. Exner's data, ultimate normative reference for the CS, differ significantly from the three other samples for many variables. Cultural factors, examiner's level of experience, as well as participants' selection and recruitment, presumably flow into calculated differences between the four samples. Our findings indicate the necessity for constituting normative data for different communities, as well as the indispensability of examiners' familiarity with the CS, in order to provide reliable clinical information.     

Rorschach Comprehensive System data for a sample of 517 adults from Spain (Barcelona)

This study is composed of 517 adult voluntary participants, from Barcelona and its surroundings, all of whom are from Spain, Barcelona, and Catalonia. The study builds on a previously published report for a smaller sample (N = 250; see Alvarez, Baeza, Campo, Garcia, Guardia, et al., 1993) and is the result of collaboration between students and professors at the Barcelona Rorschach School (Catalan Rorschach Society). Our aim is to offer local data that should help to understand possible differences with existing reference data for the Rorschach Comprehensive System (CS; Exner, 2005), which is the system that has been taught at this school since 1978. Inclusion and exclusion criteria are described and interrater reliability statistics at the response level are presented, along with scores for the CS, followed by a discussion and very brief comparison with other international reference samples.     

Crossing the North Sea seems to make DCD disappear: Cross-validation of Movement Assessment Battery for Children-2 norms

The Movement Assessment Battery for Children has been revised as the Movement ABC-2 (Henderson, Sugden, & Barnett, 2007). In Europe, the 15th percentile score on this test is recommended for one of the DSM-IV diagnostic criteria for Developmental Coordination Disorder (DCD). A representative sample of Dutch and Flemish children was tested to cross-validate the UK standard scores, including the 15th percentile score. First, the mean, SD and percentile scores of Dutch children were compared to those of UK normative samples. Item standard scores of Dutch speaking children deviated from the UK reference values suggesting necessary adjustments. Except for very young children, the Dutch-speaking samples performed better. Second, based on the mean and SD and clinical relevant cut-off scores (5th and 15th percentile), norms were adjusted for the Dutch population. For diagnostic use, researchers and clinicians should use the reference norms that are valid for the group of children they are testing. The results indicate that there possibly is an effect of testing procedure in other countries that validated the UK norms and/or cultural influence on the age norms of the Movement ABC-2. It is suggested to formulate criterion-based norms for age groups in addition to statistical norms.     

The interclinician reliability of Rorschach interpretation in four data sets

To examine agreement on Rorschach Comprehensive System (CS; Exner, 2004) interpretations, 55 patient protocols were interpreted by 3 to 8 clinicians across 4 data sets on a representative set of 29 characteristics. Substantial reliability was observed across data sets, although a problematic design produced lower results in one. Unexpectedly, a Q-sort task had slightly lower reliability than a simple rating task. As expected, scales that summarized judgments had higher agreement than judgments to individual interpretive statements, and some clinicians produced more generalizable inferences than others. Interpretations for all clinicians were more strongly associated with patients' psychometric true scores (aggregated judgment M range = .82 to .92) than with the judgments of other clinicians (range = .76 to .89). Compared to meta-analyses of interrater reliability in psychology and medicine, the findings indicate these clinicians could reliably interpret Rorschach CS data.     

The five‐factor personality inventory: cross‐cultural generalizability across 13 countries

In the present study, we investigated the structural invariance of the Five‐Factor Personality Inventory (FFPI) across a variety of cultures. Self‐report data sets from ten European and three non‐European countries were available, representing the Germanic (Belgium, England, Germany, the Netherlands, USA), Romance (Italy, Spain), and Slavic branches (Croatia, Czech Republic, Slovakia) of the Indo‐European languages, as well as the Semito‐Hamitic (Israel) and Altaic (Hungary, Japan) language families. Each data set was subjected to principal component analysis, followed by varimax rotation and orthogonal Procrustes rotation to optimal agreement with (i) the Dutch normative structure and (ii) an American large‐sample structure. Three criteria (scree test, internal consistency reliabilities of the varimax‐rotated components, and parallel analysis) were used to establish the number of factors to be retained for rotation. Clear five‐factor structures were found in all samples except in the smallest one (USA, N = 97). Internal consistency reliabilities of the five components were generally good and high congruence was found between each sample structure and both reference structures. More than 80% of the items were equally stable within each country. Based on the results, an international FFPI reference structure is proposed. This reference structure can facilitate standardized communications about Big Five scores across research programmes. Copyright © 2003 John Wiley & Sons, Ltd.     

Measurement Invariance of the Day Reconstruction Method: Results from the COURAGE in Europe Project

Given the growing interest in the study of subjective well-being as a measure of social progress, instruments that produce valid and reliable scores and that can be used within and across countries are needed. The aim of the present study was to analyze the measurement equivalence of the Day Reconstruction Method in its brief version, using nationally representative samples from Finland, Poland, and Spain obtained within the COURAGE in Europe project. The goodness-of-fit of a two-correlated-factors model and the reliability of the scores obtained were assessed. Cross-country invariance was tested employing a multiple group confirmatory factor analysis, through sequential constraint imposition. In each country, measurement invariance was tested across time frames (morning, afternoon and evening) and days of the week (weekday and weekend). The results found support for the hypothesis of a two-correlated-factors (positive and negative affect) structure; the reliability of the positive, the negative and the net affect scores showed appropriate values. A high equivalence across the three national samples was found: all items except one showed strong measurement invariance indicating that respondents from Finland, Poland, and Spain attribute the same meaning to the latent construct under study, and the levels of the underlying items are equal in all three countries. Similar results were found for the measurement equivalence across time frames and days of the week. Our findings support the assumption of comparability across the different samples considered; in general, higher positive affect and lower negative affect were found in Finland, in the evening and at the weekend.     

A comparison of normative data for the Trail Making Test from several countries: equivalence of norms and considerations for interpretation

The Trail Making Test may not be equivalent across cultures, i.e., differences in the scores across different cultures may not reveal real differences in the ability of the subjects on the construct being measured. In order to assess this hypothesis, normative samples from ten different countries were compared. Age decade subgroups across samples were ranked based on mean time taken to complete each part of the task. Large Z scores differences were found between these samples when comparing the first with the second, and the last in the rank. These differences were significant even when age and education were comparable across samples. Following Van de Vijver & Tanzer (1997), several possible sources of bias were identified. Incomparability of samples and administration differences were the most likely factors accounting for differences. Because of the lack of validity studies in the countries considered, no firm conclusions could be obtained regarding construct bias. Although the TMT may be measuring visual scanning, psychomotor speed and mental flexibility, normative data from different countries and cultures are not equivalent which might lead to serious diagnostic errors.     

Bootstrap assessment of the reliability of maxima in surface maps of brain activity of individual subjects derived with electrophysiological and optical methods

Surface maps of brain activity can be obtained with electrophysiological and optical recordings. However, there are no established methods for determining the reliability of maps of brain activity across subject groups or across tasks within the same subject. In this paper, we use bootstrapping to establish the reliability of the locations of maxima in maps of surface brain activity of individual subjects obtained with ERP and optical (EROS) recordings and report sample analyses for two data sets. Bootstrapping is a nonparametric method for estimating statistical accuracy from the data in a single sample. The distribution of the statistic of interest is estimated by constructing “bootstrap samples” from a pool of all available cases (with replacement). Many “bootstrap replications” are obtained by calculating the statistic of interest for each sample. In the case of brain activity, many (e.g., 10,000) amplitude distributions can be derived from the data of an individual subject. Frequency counts are then computed for each recording location to establish how many times that location corresponds to a maximum. The value obtained in this fashion represents an estimate of the reliability of the observation.     

Supportive Organizations,Work–Family Enrichment,and Job Burnout in Low and High Humane Orientation Cultures

The present study draws on the work–family and cross-national management literature to examine the relationships between Family-Supportive Organizational Perceptions (FSOP), work–family enrichment, and job burnout across five countries with different cultural backgrounds: Malaysia, New Zealand, France, Italy, and Spain. Using a combined sample of 980 employees, we find support for a partial mediation model in which FSOP is positively associated with work–family enrichment, which in turn is negatively related to job burnout. Given our focus on support, we test the moderating role of the cultural value humane orientation, that is, the extent to which a society values altruism, kindness, and compassion. The five countries in our sample offer variation in their country-level scores as determined by the GLOBE study (House et al., 2004). We found that individuals from cultures that scored higher in “as is” humane orientation (i.e., scores for actual practices) experienced lower job burnout when FSOP increased. This pattern was reversed when considering “should be” humane orientation (i.e., scores for ideal values). The implications for the work–family and the cross-national management literature, and for practice, are discussed.     

A four-country investigation of factors facilitating student internet use.

Education and national culture may influence Internet use. The present study explored factors that might facilitate college students' use of the Internet across samples in four countries: India, Mauritius, Reunion Island (an overseas Department of France), and the United States. Questionnaire data was collected on students' extent and frequency of Internet use. Also, structured items asked about the factors that would help them to use the Internet more. Data was analyzed using dummy-coded multiple regression so as to compare responses across the country groups. The U.S. sample was coded as the reference group so that the regression coefficients in the equation provided a comparison between the U.S. sample and each other country represented. Further country comparisons were made using t-tests for differences between the regression coefficients. Results suggested that there was more Internet use in the U.S. sample compared to the others. The Non-Western samples also reported greater desire for Internet access and non-English web sites. Training on how to use the Internet and course requirements for Internet use were reported as helpful by the India and Mauritius groups to a greater extent than by those in Reunion and the United States. Implications of the present exploratory results are discussed for applications of theoretical models across cultures.     

WAIS-III percentile scores by education and sex for U.S. and Canadian populations

Tables are presented for the Wechsler Adult Intelligence Scale-Third Edition (WAIS-III) IQ and index scores by education level for both the U.S. and Canadian normative samples. This allows clinicians to provide more accurate identification of relative strengths or weaknesses, compared to expectations from an individual's background, rather than the general population. Because sex differences are notable on the Processing Speed Index, data for this measure are presented separately. The similarities and differences between the two national samples are noted, with particular reference to the relatively weaker demographic effects found in the Canadian sample.     

Effect sizes and cut-off points: a meta-analytical review of burnout in latin American countries

Burnout is a highly prevalent globalized health issue that causes significant physical and psychological health problems. In Latin America research on this topic has increased in recent years, however there are no studies comparing results across countries, nor normative reference cut-offs. The present meta-analysis examines the intensity of burnout (emotional exhaustion, cynicism and personal accomplishment) in 58 adult nonclinical samples from 8 countries (Argentina, Brazil, Chile, Colombia, Ecuador, Mexico, Peru and Venezuela). We found low intensity of burnout but there are significant differences between countries in emotional exhaustion explained by occupation and language. Social and human service professionals (police officers, social workers, public administration staff) are more exhausted than health professionals (physicians, nurses) or teachers. The samples with Portuguese language score higher in emotional exhaustion than Spanish, supporting the theory of cultural relativism. Demographics (sex, age) and study variables (sample size, instrument), were not found significant to predict burnout. The effect size and confidence intervals found are proposed as a useful baseline for research and medical diagnosis of burnout in Latin American countries.     

A general scheme for the analytic decomposition of objective test scores: Illustrative demonstrations using the Rod-and-Frame test and the Müller-Lyer illusion

Objective tests of personality typically include a number of items or trials; the total score on the test is the sum of the subject's “correct” responses across all such trials. Normally, the trials are varied systematically across various facets of the test design, so that the total score represents a composite measure of accuracy averaged across these test facets. However, since only one score is computed for each subject, some potentially important kinds of individual differences—namely all those associated with each particular variation in the test design—are treated solely as measurement unreliability. Such a psychometric stance may serve to obscure more differentiated types of individual differences, with the result that composite scores from trials based on one type of experimental design may not be highly related to such scores from trials using a somewhat different design. The present paper presents a general procedure for scoring objective tests more analytically. To illustrate this general rationale, and to demonstrate its potential utility, data have been reanalyzed from two previous studies, one using the Rod-and-Frame test, the other the Müller-Lyer illusion. In both cases, the traditional global accuracy score did not correlate significantly with other theoretically related variables, while a number of component scores were quite highly related.     

Composite and preferences Scales of Morningness: reliability and factor invariance in adult and university samples

The creation and adaptation of scales or inventories assessing specific circadian typologies has been a predominant focus within the field of chronopsychology. The present study addressed the psychometric properties of two scales of morningness-eveningness: the Morningness Composite Scale (CS; Smith, Reilly, & Midkiff, 1989) and the Early/Late Preferences Scale (PS; Smith, Folkard, Schmieder, Parra, Spelten, & Almirall, 1993). Internal consistency and factor invariance of the CS and PS were analyzed in two samples: a group of 203 university students (age range = 19-30) and a group of 125 working adults (age range = 31-65). Results indicated satisfactory internal consistency for both full scales with each age group and confirmed the factor invariance across age for the two CS factors and one of the PS factors. A higher tendency in morningness on both scales was noted in the adult sample.