VALIDITY GENERALIZATION: A REVIEW AND CRITIQUE OF THE CORRELATION MODEL

This paper presents a review and critique of the validity generalization research based on the correlation model. Emphasis was placed on integrating the applied validity generalization studies as well as the computer simulation studies which have tested the accuracy of the various validity generalization procedures in estimating the mean and variance of true validity coefficients. In general, this review indicated that the validity generalization procedures are fairly accurate in estimating the mean and variance of true validities, that the preponderance (e.g., 90%) of corrected validity coefficients for a single predictor-criterion relationship, based on a series of cumulative studies, when placed in distributional form are positive, and that a substantial proportion of observed validity coefficient variation can be attributed to statistical artifacts with sampling error accounting for the majority of the artifactual variance. Moreover, potential problems with validity generalization/meta analysis based on criterion-related validity coefficients were discussed. It was concluded that validity general-ization/meta-analysis is an important development in summarizing cumulative research, however, caution should be exercised in utilizing and interpreting the findings of such analyses. Directions for future research aimed at establishing trait-performance relationships were discussed.     

Multitrait-multimethod model of adolescent deviance,drug use,academic, and sexual behaviors

A multitrait-multimethod design was used to test Jessor and Jessor's (1977) problem behavior theory with a sample of adolescents, their parents, and peers. Behaviors included general deviance, drug use, low academic motivation, and precocious sexual behavior. While research has supported this theory, the reliance on self-report data for the majority of this work may threaten the validity of these findings. Results from hierarchical confirmatory factor analyses indicated that the convergent and discriminant validity of the seven behavior constructs was high and method effects were low. A two-factor second-order model, representing General Drug Use and Other Problem Behavior, accounted for a larger proportion of variance in the lower order factors compared to a single-factor model.This research was supported by Grant DA 03706 from the National Institute on Drug Abuse.     

Technical Knowledge,Prosocial Knowledge,and Clinical Performance of Indian Medical Students

We cross‐culturally replicated and extended findings reported by Kell, Motowidlo, Martin, Stotts, and Moreno that technical knowledge and prosocial knowledge have independent effects on performance. In a sample of 196 Indian medical students, we found that prosocial knowledge explains variance in students' clinical performance beyond the variance explained by technical knowledge and technical knowledge explains variance in clinical performance beyond the variance explained by prosocial knowledge. Contrary to findings that American medical students' prosocial inclinations, as reflected in measures of empathy, seem to decline over the course of their medical training (e.g., Hojat, Vergare, Maxwell, Brainard, Herrine, and Isenberg), we found that Indian medical students' prosocial knowledge steadily increased from their third to fifth years of medical study.     

Using generalizability theory to estimate the relative effect of class size and number of items on the dependability of student ratings of instruction

Previous research on the effect of class size on student ratings of instruction has primarily investigated the effect of class size on the favorableness of these ratings rather than its effect on their reliability (dependability). A few studies have used "generalizability theory" to demonstrate the relative effect of class size on the dependability of student ratings of instruction. The purpose of the present study was to test the validity of the findings of these studies in a different cultural setting using a different student ratings questionnaire. Using a random-effect analysis of variance to estimate the variance components for a design in which students were nested within classes and crossed with items, it was found that the variance component for class size was appreciably larger than that for items. At least 20 students were needed to obtain a generalizability coefficient for relative decisions of .70 or more. Increasing the number of students has a greater effect on generalizability coefficients than increasing the number of items.     

Meta-analysis of the relationship between congruence and well-being measures

This paper summarizes the results of a meta-analysis on findings of 41 different studies, yielding 77 correlation coefficients between measures of personality-environment congruence and well-being. Results show congruence-achievement and congruence-stability correlations of .06 and .15, respectively, with negligible residual variance. The mean congruence-satisfaction correlation was .21, and after two further breakdowns—by environmental reference and congruence measuring method—mean congruence-satisfaction correlations exceeding .35 were found, with almost all total nonrandom variance among findings of different congruence studies explained.     

Bullying and victimization in elementary schools: a comparison of bullies, victims, bully/victims, and uninvolved preadolescents

Research on bullying and victimization largely rests on univariate analyses and on reports from a single informant. Researchers may thus know too little about the simultaneous effects of various independent and dependent variables, and their research may be biased by shared method variance. The database for this Dutch study was large (N = 1,065) and rich enough to allow multivariate analysis and multi-source information. In addition, the effect of familial vulnerability for internalizing and externalizing disorders was studied. Gender, aggressiveness, isolation, and dislikability were most strongly related to bullying and victimization. Among the many findings that deviated from or enhanced the univariate knowledge base were that not only victims and bully/victims but bullies as well were disliked and that parenting was unrelated to bullying and victimization once other factors were controlled.     

Passion, craving, and affect in online gaming: Predicting how gamers feel when playing and when prevented from playing

According to the Dualistic Model of Passion, two forms of passion can motivate a behavior: harmonious passion and obsessive passion. Across various life activities, studies have found that the two forms of passion show different relationships with affect, linking harmonious passion to positive affect and obsessive passion to negative affect. To investigate if this pattern also holds for online gaming, the present study investigated 160 gamers involved in playing massively multiplayer online role-playing games (MMOs) and examined positive and negative affect (a) when playing and (b) when prevented from playing. In addition, the effects of general affect and craving for playing MMOs were controlled for. Results were as expected from the Dualistic Model of Passion: harmonious passion for online gaming predicted positive affect when playing whereas obsessive passion predicted negative affect when playing and when prevented from playing. Moreover, these effects remained unchanged when general affect and craving were controlled for. With this, the present research shows that individual differences in passion for online gaming explain unique variance in gaming-related emotions. Moreover, the present findings suggests that craving is a variable that future research on positive and negative affect in online gaming should pay closer attention to.     

测评中的共同方法偏差

共同方法变异是指由于测量方法而非所测构想造成的变异。共同方法变异会对测评和测评间的相关产生严重影响,甚至会使研究导致错误结论。本文介绍了学界对共同方法偏差的研究,内容包括：(1)共同方法偏差在多大程度上和以何种方式影响测评研究结果;(2)共同方法偏差的潜在来源及其程序控制。     

羞愧对亲社会行为影响的三水平元分析

羞愧是一种典型的道德情绪,其对亲社会行为的作用在既往研究中并不一致。本研究首次采用三水平元分析技术整合相关实证研究,检验羞愧对亲社会行为的影响及调节变量在二者关系中的作用。通过文献检索和筛选,共计纳入26篇文献, 85个效应量,总样本量为5823人。主效应检验发现,羞愧组比控制组表现出更多的亲社会行为,羞愧能够促进亲社会行为的产生。调节效应检验发现,暴露情境比掩蔽情境更能诱发羞愧对亲社会行为的促进作用,亲社会行为的产生情境(暴露情境或掩蔽情境)调节作用显著,但年龄、文化背景、羞愧诱发方法、羞愧类型及亲社会行为类型等变量的调节作用不显著。本研究使用三水平元分析方法保证了纳入文献信息的完整性,从而就羞愧对亲社会行为的影响及调节变量在两者关系中的作用得出更为全面可靠的研究结论,有助于拓展人们对于羞愧与亲社会行为关系及调节机制的认识。后续研究可深入探讨认知因素和个体特征在羞愧对亲社会行为影响中的作用,考察羞愧与其他道德情绪对亲社会行为影响的差异。     

Personality, intelligence and personal space among violent and non-violent delinquents

McGurk, Davis and Grehan (1981) have suggested that personality variables may have a moderating effect on the violence—personal space relationship originally reported by Kinzel (1970). Similarly, Wright (1974) has reported enhanced personal space in inmates classified as of low intelligence. The present study was undertaken in order to further investigate possible confounding effects of personality and intelligence differences on the relationship between aggressive behaviour and spatial use. A direct approach stop-distance and a semi-blind measure of personal space were obtained for a group of violent young offenders and a non-violent control group. Verbal and Non-verbal Intelligence scores and personality scores were also collected. Analysis of variance showed no significant differences in personal space size between the violent and non-violent group on any one measure. However, a significant interaction effect was found between violence, P and Non-verbal Intelligence such that violent inmates exhibited augmented personal space in relation to non-violent inmates only in the cases of those scoring both low on Non-verbal Intelligence and high on P. These findings contradict earlier research reporting a difference in spatial use between violent and non-violent Ss. It is concluded that the lack of a standardized well-controlled technique for measuring personal space has resulted in ambiguous findings in this area of research.     

Competencies,Personality Traits,and Organizational Rewards of Middle Managers: A Motive-Based Approach

Previous literature suggests that performance ratings are saturated with rater-related idiosyncratic variance. Given that modern psychometric theories relegate this source of variance to measurement error, it has not been the subject of much previous research. Of importance, identifying and estimating the variance components underlying idiosyncratic rater variance will inform our understanding of the nature of this variance. In a sample of managerial performance ratings we report on components of variance and find that the idiosyncratic rater variance component is about one third rater main effects variance, one third Rater × Ratee interaction effects variance, and one third upper-bound Rater × Ratee × Dimension interaction effects variance. Further, results indicate that variance components are moderated by the acquaintanceship time between the rater and the ratee.     

Residual Normality Assumption and the Estimation of Multiple Membership Random Effects Models

While conventional hierarchical linear modeling is applicable to purely hierarchical data, a multiple membership random effects model (MMrem) is appropriate for nonpurely nested data wherein some lower-level units manifest mobility across higher-level units. Although a few recent studies have investigated the influence of cluster-level residual nonnormality on hierarchical linear modeling estimation for purely hierarchical data, no research has examined the statistical performance of an MMrem given residual non-normality. The purpose of the present study was to extend prior research on the influence of residual non-normality from purely nested data structures to multiple membership data structures. Employing a Monte Carlo simulation study, this research inquiry examined two-level MMrem parameter estimate biases and inferential errors. Simulation factors included the level-two residual distribution, sample sizes, intracluster correlation coefficient, and mobility rate. Results showed that estimates of fixed effect parameters and the level-one variance component were robust to level-two residual non-normality. The level-two variance component, however, was sensitive to level-two residual non-normality and sample size. Coverage rates of the 95% credible intervals deviated from the nominal value assumed when level-two residuals were non-normal. These findings can be useful in the application of an MMrem to account for the contextual effects of multiple higher-level units.     

Humor, laughter, and physical health: methodological issues and research findings.

All published research examining effects of humor and laughter on physical health is reviewed. Potential causal mechanisms and methodological issues are discussed. Laboratory experiments have shown some effects of exposure to comedy on several components of immunity, although the findings are inconsistent and most of the studies have methodological problems. There is also some evidence of analgesic effects of exposure to comedy, although similar findings are obtained with negative emotions. Few significant correlations have been found between trait measures of humor and immunity, pain tolerance, or self-reported illness symptoms. There is also little evidence of stress-moderating effects of humor on physical health variables and no evidence of increased longevity with greater humor. More rigorous and theoretically informed research is needed before firm conclusions can be drawn about possible health benefits of humor and laughter.     

Joint factors in self-reports and ratings: Neuroticism,extraversion and openness to experience

Although most dimensional theories of personality assume that the same traits can be assessed in either ratings or self-reports, joint factor analyses of data from these two methods have seldom provided clear evidence in support of this position. Previous analyses have found a preponderance of within-method factors, or have had to transform variables or use unorthodox rotational procedures in order to control the effects of method variance. The present study argues that recent conceptual and technical advances should now make it possible to show joint factors at the second-order level using standard factor techniques. The NEO Inventory and NEO Rating Form, which measure 18 traits in the domains of Neuroticism, Extraversion and Openness to Experience, were administered to a sample of 281 men and women. Varimax rotation of three principal components clearly showed the hypothesized structure within and across self-reports and spouse ratings. Convergent and discriminant validity of the joint factors with the EPI scales was also shown. The results suggest that the effects of method variance can be minimized if well-qualified raters use psychometrically adequate instruments to provide ratings of clearly conceptualized traits. In addition, they provide strong evidence for the validity of the proposed three-domain model of personality.     

The Relaxation Inventory: Self-Report Scales of Relaxation Training Effects

The development of a self-report measure to assess the effects of relaxation training was examined. A rigorous statistical method of scale construction consisting of a modification of the scale discrimination technique was employed, resulting in a 45-item questionnaire representing three orthogonally derived scales. The three scales, Physiological Tension, Physical Assessment, and Cognitive Tension, demonstrated adequate internal consistency with KR20 reliability coefficients of .89, .95, and .81, respectively. In a second study of predictive validity, 40 individuals were randomly assigned to one of four conditions: relaxation training, tension inducement, pre-postcontrol, or postcontrol. Univariate analysis of variance indicated significant findings for each of the three dimensions of the inventory. The Physiological Tension Scale detected significant increases in tension following tension inducement, whereas the Physical Assessment Scale and Cognitive Tension Scale detected increases in relaxation following relaxation training. Recommendations were made for future research on the inventory.     

The Relaxation Inventory: self-report scales of relaxation training effects

The development of a self-report measure to assess the effects of relaxation training was examined. A rigorous statistical method of scale construction consisting of a modification of the scale discrimination technique was employed, resulting in a 45-item questionnaire representing three orthogonally derived scales. The three scales, Physiological Tension, Physical Assessment, and Cognitive Tension, demonstrated adequate internal consistency with KR20 reliability coefficients of .89, .95, and .81, respectively. In a second study of predictive validity, 40 individuals were randomly assigned to one of four conditions: relaxation training, tension inducement, pre-postcontrol, or postcontrol. Univariate analysis of variance indicated significant findings for each of the three dimensions of the inventory. The Physiological Tension Scale detected significant increases in tension following tension inducement, whereas the Physical Assessment Scale and Cognitive Tension Scale detected increases in relaxation following relaxation training. Recommendations were made for future research on the inventory.     

Depression, self-esteem, and complex attributions for life problems

Past research has suggested that mild and moderate depression are associated with increased attributional processing and a tendency to make complex attributions involving two or more causes. The present research tested the hypothesis that depression and low self-esteem are associated with a tendency to make attributions to multiple causes when faced with life problems. The results were used to demonstrate that the tendency to make multiple attributions for specific life problems accounts for unique variance in depression and self-esteem scores, even after removing variance due to general attributional style. The findings are discussed with reference to the need for multidimensional models of attribution in depression and attributional retraining efforts to include an emphasis on individual differences in the number of multiple attributions made by people.     

Covariate-free and Covariate-dependent Reliability

Classical test theory reliability coefficients are said to be population specific. Reliability generalization, a meta-analysis method, is the main procedure for evaluating the stability of reliability coefficients across populations. A new approach is developed to evaluate the degree of invariance of reliability coefficients to population characteristics. Factor or common variance of a reliability measure is partitioned into parts that are, and are not, influenced by control variables, resulting in a partition of reliability into a covariate-dependent and a covariate-free part. The approach can be implemented in a single sample and can be applied to a variety of reliability coefficients.     

The Situational Specificity of Validities: Correcting for Statistical Artifacts Does Not Always Reduce the Trans-Situational Variability of Correlation Coefficients

Validity generalization research suggests that much of the variability in validity coefficients is due to statistical artifacts, such as sampling error. Correcting for these artifacts usually reduces the trans-situational variability in validity coefficients, but the reverse can happen. Conditions under which corrections for statistical artifacts can increase the variability in validity coefficients are described, examples of meta-analyses reporting larger corrected than observed variance in validity are cited, and implications for the situational specificity of validity coefficients are discussed. In general, the true variability in validity coefficients based on large samples is often larger than the observed variability in test validities.     

Linking mechanisms of work–family conflict and segmentation

Despite the abundance of work and family research, few studies have compared the linking mechanisms specified in theoretical models of work–family conflict and segmentation. Accordingly, the current study provides a greater degree of empirical clarity concerning the interplay of work and family by directly examining the indirect effects of work–family conflict linkages and the direct effects of segmentation linkages on work and family satisfaction outcomes. Results, based on 209 samples and 994 effect sizes, indicate that indirect effect work–family conflict models explain 2.20% and 6.20% of the variance in job and family satisfaction outcomes, whereas direct effect segmentation models explain 54.10% and 48.50% of the variance in job and family satisfaction outcomes. Theoretical and practical implications of these findings are discussed.