Internet-based questionnaire assessment: appropriate use in clinical contexts

Use of internet-mediated questionnaire assessment is growing in behavioural telehealth provision. Literature on web-based psychological testing indicates that--while tests used online should be validated for online use--such measures can be reliable and valid. However, use of normative data (especially data from paper-and-pencil samples) presents problems. Studies have shown score distributions may differ across testing media, even when comparisons are made between equivalent samples. Data from 3 different projects illustrate such (substantive) differences, and the errors that would result from using established norms to interpret internet-mediated assessments. Given representativeness issues with "internet norms", it is recommended that if online tests are used for clinical purposes, this should currently be done in a manner that does not rely on normative data.     

Can the Discretionary Nature of Certain Criteria Lead to Differential Prediction Across Cultural Groups?

We examined the conjecture that relations between constructs across cultures may be susceptible to cultural moderation where the performance of the criterion construct is discretionary. This hypothesis was investigated using the relationship between personality and three performance constructs, with samples from the United States and New Zealand, two ideologically distinct cultures with respect to achievement orientation. All hypotheses were supported by results of hierarchical moderated regression analyses using bias free measures, suggesting that considering whether construct behaviors are discretionary is important when considering the merit of generalizing research findings across cultures.     

Addressing Issues in the Development and Use of the Composite International Reference Values as Rorschach Norms for Adults

This article describes 3 studies evaluating normative reference data for the Rorschach Comprehensive System (CS; Exner, 2003, 2007), with a particular focus on the viability of the Composite International Reference Values (CIRVs) that were compiled from 21 adult studies by Meyer, Erdberg, and Shaffer (2007). Study 1 documented how the CIRV norms are virtually identical when organized into 3 groups differentiated by the quality of their data collection effort, including an optimal group of 4 samples that relied on multiple experienced examiners and provided ongoing quality control over administration and coding. Analyses also showed that relative to the group of more optimal samples, the group of less optimal samples did not produce more variability in summary scores within or across samples or lower interrater reliability for coding. Study 2 used the existing CS reference norms to generate T scores for the CIRV means and documented how the CS norms make other samples of healthy nonpatients look psychologically impaired in multiple domains. Study 3 documented with examples from 4 different countries how 2 sets of within-country local norms produced notably different results on some variables, which compromises the ability of local norms to be used instead of the CIRVs. Taken together, the 3 studies provide support for the use of CIRVs in clinical practice as norms that are generalizable across samples, settings, languages, and cultures and that account for the natural variability that is present when clinicians and researchers contend with the ambiguity contained in the standard CS reference materials concerning the proper ways to administer and code. We conclude by urging CS users to rely on the CIRVs when making clinical inferences and to adopt alternative methods of ensuring they are following cohesively standardized administration and coding guidelines.     

Bias and equivalence in cross-cultural assessment: an overview

In every cross-cultural study, the question as to whether test scores obtained in different cultural populations can be interpreted in the same way across these populations has to be dealt with. Bias and equivalence have become the common terms to refer to the issue. Taxonomy of both bias and equivalence is presented. Bias can be engendered by the theoretical construct (construct bias), the method such as the form of test administration (method bias), and the item content (item bias). Equivalence refers to the measurement level at which scores can be compared across cultures. Three levels of equivalence are possible: the same construct is measured in each cultural group but the functional form of the relationship between scores obtained in various groups is unknown (structural equivalence), scores have the same measurement unit across populations but have different origins (measurement unit equivalence), and scores have the same measurement unit and origin in all populations (full scale equivalence). The most frequently encountered sources of bias and their remedies are described.     

Zuckerman's revised alternative five-factor model: validation of the Zuckerman-Kuhlman-Aluja Personality Questionnaire in four French-speaking countries

The aim of this study was to analyze the replicability of Zuckerman's revised Alternative Five-factor model in a French-speaking context by validating the Zuckerman-Kuhlman-Aluja Personality Questionnaire (ZKA-PQ) simultaneously in 4 French-speaking countries. The total sample was made up of 1,497 subjects from Belgium, Canada, France, and Switzerland. The internal consistencies for all countries were generally similar to those found for the normative U.S. and Spanish samples. A factor analysis confirmed that the normative structure replicated well and was stable within this French-speaking context. Moreover, multigroup confirmatory factor analyses have shown that the ZKA-PQ reaches scalar invariance across these 4 countries. Mean scores were slightly different for women and men, with women scoring higher on Neuroticism but lower on Sensation Seeking. Globally, mean score differences across countries were small. Overall, the ZKA-PQ seems an interesting alternative to assess both lower and higher order personality traits for applied or research purposes.     

Zuckerman's Revised Alternative Five-Factor Model: Validation of the Zuckerman-Kuhlman-Aluja Personality Questionnaire in Four French-Speaking Countries

The aim of this study was to analyze the replicability of Zuckerman's revised Alternative Five-factor model in a French-speaking context by validating the Zuckerman-Kuhlman-Aluja Personality Questionnaire (ZKA–PQ) simultaneously in 4 French-speaking countries. The total sample was made up of 1,497 subjects from Belgium, Canada, France, and Switzerland. The internal consistencies for all countries were generally similar to those found for the normative U.S. and Spanish samples. A factor analysis confirmed that the normative structure replicated well and was stable within this French-speaking context. Moreover, multigroup confirmatory factor analyses have shown that the ZKA–PQ reaches scalar invariance across these 4 countries. Mean scores were slightly different for women and men, with women scoring higher on Neuroticism but lower on Sensation Seeking. Globally, mean score differences across countries were small. Overall, the ZKA–PQ seems an interesting alternative to assess both lower and higher order personality traits for applied or research purposes.     

Measurement equivalence/invariance of the abusive supervision measure across workers from Taiwan and the United States

Growing international research interest in negative-leadership behaviors prompts the need to examine whether measures of ineffective leadership developed in the United States are equivalent across countries outside the United States. B. J. Tepper's (2000) abusive supervision measure has been used widely inside and outside the United States and merits research attention on its construct equivalence across different cultural settings. The authors conducted a series of multigroup confirmatory factor analyses to investigate the measurement equivalence of this measure across Taiwan (N = 256) and the United States (N = 389). Configural invariance was established, suggesting that both U.S. and Taiwanese samples perceive abusive supervision as a single-factor concept. Furthermore, the establishment of partial metric invariance and partial scalar invariance suggests that the abusive supervision measure is applicable to crosscultural comparisons in latent means, construct variance, construct covariances, and unstandardized path coefficients with the caution that workers from different cultures calibrate their responses differently when answering some items.     

Measurement Equivalence/Invariance of the Abusive Supervision Measure Across Workers From Taiwan and the United States

Growing international research interest in negative-leadership behaviors prompts the need to examine whether measures of ineffective leadership developed in the United States are equivalent across countries outside the United States. B. J. Tepper's (2000) abusive supervision measure has been used widely inside and outside the United States and merits research attention on its construct equivalence across different cultural settings. The authors conducted a series of multigroup confirmatory factor analyses to investigate the measurement equivalence of this measure across Taiwan (N = 256) and the United States (N = 389). Configural invariance was established, suggesting that both U.S. and Taiwanese samples perceive abusive supervision as a single-factor concept. Furthermore, the establishment of partial metric invariance and partial scalar invariance suggests that the abusive supervision measure is applicable to crosscultural comparisons in latent means, construct variance, construct covariances, and unstandardized path coefficients with the caution that workers from different cultures calibrate their responses differently when answering some items.     

Ipsative and Normative Scales in Adjectival Measurement of Personality: Problems of Bias and Discrepancy

887 respondents completed ipsative and normative versions of the PAL-TOPAS personality questionnaire. Data were analysed to test for (1) systematic bias in scores associated with the two response formats and (2) predictors of the magnitude of the discrepancy in the individual's ipsative and normative scores. Discrepancy was assessed for both item responses and scale scores. Sources of biases investigated included ipsative scaling artifact, extremeness of scores on the normative scales and response variability. Results showed that systematic bias in scale scores and magnitude of discrepancy were predicted by different factors. One source of systematic bias was associated with ipsative scaling artifact: the ipsative scales measure both the scale itself and rejection of other alternatives. A second source of systematic bias was acquiescence in response to normative items. A confirmatory factor analysis showed that a good but imperfect fit to the data may be obtained by constructing a structural model of the inter-relationship between normative and ipsative scores which accommodates both sources of bias. The strongest influence on discrepancy in scale scores was extremeness of normative scoring, associated with a bias towards either general acceptance or rejection of trait adjectives. It is concluded that both normative and ipsative response formats have limitations, and it may often be desirable to assess both.     

Leiter-3 generalizability in Scandinavia

The Leiter-3 is a nonverbal assessment that evaluates cognitive abilities and has been adapted for use in Scandinavia. Generalizability of United States-based normative scoring for use with the Scandinavian population was evaluated. Leiter-3 scores from a sample of Scandinavian students were compared with scores obtained from the Leiter-3 standardization sample, controlling for confounding variables, across ages, using mixed-methods analysis. A Scandinavian-population-based sample was created from Leiter-3 standardization data and norms were constructed and were used to generate standardized scores from the sample data. Results suggest that overall the Scandinavian test-takers score higher than American test-takers, but that differences between groups were minimized when controlling for factors that may influence cognitive performance. Creating Scandinavian based scores was not effective at reducing gaps in performance, suggesting that differences in performance between the different populations may be attributable to factors other than those typically controlled for when constructing standardized tests. Implications of these results and recommendations for Leiter-3 adaptation are reviewed.     

The cross-cultural generalizability of Zuckerman's alternative five-factor model of personality

The aim of this study was to analyze the cross-cultural generalizability of the Alternative five-factor model (AFFM; Zuckerman, Kuhlman, & Camac, 1988). The total sample was made up of 9,152 subjects from six countries: China, Germany, Italy, Spain, Switzerland, and the United States. The internal consistencies for all countries were generally similar to those found for the normative American sample. Factor analyses within cultures showed that the normative American structure was replicated in all cultures; however, the congruence coefficients were slightly lower in China and Italy. A similar analysis at the facet level confirmed the high cross-cultural replicability of the AFFM. Mean-level comparisons did not always show the hypothesized effects. The mean score differences across countries were very small.     

THE BENDER GESTALT VISUAL MOTOR TEST IN PAPUA NEW GUINEA

Although the Bender Gestalt Visual Motor Test has been used across cultures, evidence indicates that performance may be related to differences in experience including exposure to Western acculturation. Thus in Papua New Guinea (PNG), with relatively recent exposure to Western systems of education, it might be expected, particularly in view of available cognitive research data, that performance would lag somewhat by comparison with norms from American or other Western samples. Thus normative data for PNG is required before the V.M.T. may be systematically used for diagnostic work. The present paper describes the V.M.T. performance of 245 children from the Eastern Highlands of Papua New Guinea, and 74 Australian children living in the same area. V.M.T. scores were compared also with American data. Consistent with the research on cognitive development in the literature, V.M.T. scores for the PNG children seemed to be about three to four years delayed, though the same kind of decrease in error scores with chronological age was evident.     

Bilingual comparison of Mandarin and English cognitive bias tasks

Most research into cognitive biases has used Western samples, despite potential East-West socio-cultural differences. One reason is the lack of appropriate measures for non-Westerners. This study is about cross-linguistic equivalence which needs to be established before assessing cross-cultural differences in future research. We developed parallel Mandarin and English measures of interpretation bias and attention bias using back-translation and decentering procedures. We assessed task equivalence by administering both sets of measures to 47 bilingual Mandarin-English speakers. Interpretation bias measurement was similar and reliable across language versions, confirming suitability of the Mandarin versions for future cross-cultural research. By contrast, scores on attention bias tasks did not intercorrelate reliably, suggesting that nonverbal stimuli such as pictures or facial expressions of emotion might present better prospects for cross-cultural comparison. The development of the first set of equivalent measures of interpretation bias in an Eastern language paves the way for future research investigating East-West differences in biased cognition.     

Profiles of Everyday Executive Function in Acquired and Developmental Disorders

The current study focused on the early development of inhibitory control in 5- to 7-year-old children attending kindergarten in two Eastern-European countries, Romania and Russia. These two countries share many aspects of child-rearing and educational practices, previously documented to influence the development of inhibitory control. Using the Lurian-based developmental approach offered by the Developmental Neuropsychological Assessment battery, the study aimed to contribute to cross-cultural developmental neuropsychology by exploring (a) early interrelationships between subcomponents of inhibitory control (response suppression and attention control) and generative fluency (verbal and figural) in these two cultures, as well as (b) the predictive value of external factors (culture and maternal education) and individual differences (age, gender, nonverbal intelligence, trait anxiety) on inhibitory control and fluency outcomes in children from both countries. First, findings in both culture samples suggest that even at this young age, the construct of inhibitory control cannot be considered a unitary entity. Second, differences in maternal education were not predictive of either inhibitory control or fluency scores. However, children’s attention control performance varied as a function of culture, and the direction of these cultural effects differed by whether the target outcome involved performance accuracy versus efficiency as an output. Findings also confirmed the previously documented intensive developmental improvement in preschoolers’ inhibitory control during this period, influencing measures of response suppression and particularly attention control. Finally, the results further stress the importance of individual differences effects in trait anxiety on attention control efficiency across cultures.     

网络测验和纸笔测验的测量不变性研究：以生活满意度量表为例

以生活满意度量表为例,运用实证性因素分析,考察在中国文化下网络测验和传统纸笔测验之间的测量不变性。结果显示,网络测验和纸笔测验之间存在弱不变性,即网络测验和纸笔测验有着相同的测量单位;但网络测验和纸笔测验只存在部分的强不变性和部分的严格不变性,测验实施环境对结果的影响不可忽视。该研究表明,恰当设计的网络测验是可靠的,同时还提示,当一个测验在不同情境下运用时,检验测量不变性十分必要     

Parent Ratings of the Oppositional Defiant Disorder Symptoms: Item Response Theory Analyses of Cross-National and Cross-Racial Invariance

Item response theory (IRT) based differential item functioning (DIF) was used to examine the construct and normative invariance of the DSM-IV oppositional defiant disorder (ODD) symptoms for ratings across Malaysian and Australian children, and Malaysian Malay and Malaysian Chinese children. To accomplish these goals, parents completed the Disruptive Behavior Rating Scale, which includes the eight DSM-IV ODD symptoms. Although the comparisons involving Malaysian and Australian children indicated DIF for five symptoms, only the symptom for “touchy” showed notable DIF. This was also the only symptom that showed DIF for the comparisons involving Malay and Chinese children. There were also minimal differences in the latent mean scores across Australian and Malaysian children and also Malay and Chinese children. These results indicate good support for the construct and normative invariance of the ODD symptoms for the samples compared.     

Self-reported competences and problems in Spanish adolescents: a normative study of the YSR

The aim of the study was to provide a standardization of self-reported competences and emotional/behavioural problems in Spanish adolescents, using the Achenbach's Youth Self-Report (YSR), and to compare our results to those from other countries. The YSR was completed by 2822 adolescents aged 11-18 years, recruited from secondary schools in two regions of Spain. There were significant differences in Total behaviour problems and in most problem scales in boys and girls, scoring boys higher on externalizing syndromes, whereas girls score higher on internalizing syndromes . Males' mean scores were higher than females' scores for social competence. Age effects also showed significant differences with respect to internalizing problems, thought problems, social problems and depression. Cut-off points based on the 90th and 98th percentile developed from this sample are presented for the competences and the problem scales. To test the overall range of variations across cultures, comparisons were made between broad band and narrow band measures of the YSR in Spanish adolescents and those from other countries. Gender differences in the YSR were similar to those found across different cultures; there is a consistent trend to increase behavioural problems with age; and social competence seem to be a relevant protective factor against behavioural problems.     

Simultaneous administration of the Rosenberg Self-Esteem Scale in 53 nations: exploring the universal and culture-specific features of global self-esteem

The Rosenberg Self-Esteem Scale (RSES) was translated into 28 languages and administered to 16,998 participants across 53 nations. The RSES factor structure was largely invariant across nations. RSES scores correlated with neuroticism, extraversion, and romantic attachment styles within nearly all nations, providing additional support for cross-cultural equivalence of the RSES. All nations scored above the theoretical midpoint of the RSES, indicating generally positive self-evaluation may be culturally universal. Individual differences in self-esteem were variable across cultures, with a neutral response bias prevalent in more collectivist cultures. Self-competence and self-liking subscales of the RSES varied with cultural individualism. Although positively and negatively worded items of the RSES were correlated within cultures and were uniformly related to external personality variables, differences between aggregates of positive and negative items were smaller in developed nations. Because negatively worded items were interpreted differently across nations, direct cross-cultural comparisons using the RSES may have limited value.     

International comparison of Cloninger’s temperament dimensions

Cloninger’s Tridimensional Personality Questionnaire (TPQ) and Temperament and Character Inventory (TCI) have been developed to measure the following temperament dimensions: novelty seeking (NS), harm avoidance (HA), reward dependence (RD) and persistence (P). We used our previous Finnish normative study of the TPQ and TCI (Miettunen et al., 2004) to estimate correction coefficients to convert TPQ scales to comparable TCI scales. Our aim was to compare these corrected temperament dimension scores across 20 countries adjusting for study sample differences in age and gender. In all, some variations were found in these temperament scores between countries. Differences were especially apparent between the Asian and the Western countries. By far the lowest mean score of the RD was in Japan (11.2 vs. 15.4 for other countries; effect size Cohen’s d = −5.74; z-test p < 0.001) and the highest mean score in P was in USA (5.5 vs. 4.4; d = 4.24, p = 0.001). Some of the findings could be explained by sample differences (e.g. age and education); while some may reflect real differences in the ways which personality is related to cultural factors, such as individualism/collectivism. These differences should be considered when interpreting studies with data on TPQ/TCI from different countries.     

Cross-cultural confirmatory factor analysis of the CES-D in Spanish and Mexican dementia caregivers

The Center for Epidemiologic Studies-Depression Scale (CES-D) is the most frequently used scale for measuring depressive symptomatology in caregiving research. The aim of this study is to test its construct structure and measurement equivalence between caregivers from two Spanish-speaking countries. Face-to-face interviews were carried out with 595 female dementia caregivers from Madrid, Spain, and from Coahuila, Mexico. The structure of the CES-D was analyzed using exploratory and confirmatory factor analysis (EFA and CFA, respectively). Measurement invariance across samples was analyzed comparing a baseline model with a more restrictive model. Significant differences between means were found for 7 items. The results of the EFA clearly supported a four-factor solution. The CFA for the whole sample with the four factors revealed high and statistically significant loading coefficients for all items (except item number 4). When equality constraints were imposed to test for the invariance between countries, the change in chi-square was significant, indicating that complete invariance could not be assumed. Significant between-countries differences were found for three of the four latent factor mean scores. Although the results provide general support for the original four-factor structure, caution should be exercised on reporting comparisons of depression scores between Spanish-speaking countries.