Evidence for genetic influences on personality from self-reports and informant ratings.

Self-report data on Extraversion (E) and Neuroticism (N), together with ratings by the co-twin, were obtained from a sample of 826 adult female twin pairs ascertained through a population-based twin register. Data were analyzed using a model that allowed for the contributions to personality ratings of the rater's personality (rater bias) as well as of the personality of the person being rated. For E, but not for N, significant rater bias was found, with extraverted respondents tending to underestimate, and introverted respondents tending to overestimate, the Extraversion of their co-twins. Good agreement between self-reports and ratings by the respondent's co-twin was found for both E and N. Substantial genetic influences were found for both personality traits, confirming findings from genetic studies of personality that have relief only on self-reports of respondents.     

EXPLORING RELATIONS BETWEEN TYPICAL AND MAXIMUM PERFORMANCE RATINGS AND THE FIVE FACTOR MODEL OF PERSONALITY

The study tests the distinction between typical and maximum criteria with ratings of transformational leadership performance, and examines whether the criterion-related validities of the five factor model differ for the two types of criteria. Using an East Asian military sample ( n = 1,259) where multiple ratings of typical and maximum performance were obtained from different sources, we used structural equation modeling to test the typical/maximum performance distinction. Results found that typical and maximum performance are different latent constructs and that this distinction is present even after considering rating method factors (i.e., rater source, time). The importance of this distinction is shown by the fact that validities for the personality constructs were not equally predictive of both criteria: Openness was most predictive of maximum performance, Neuroticism was most predictive of typical performance, and Extroversion was predictive of both. By distinguishing typical from maximum performance constructs, relationships between personality and transformational leadership were found to be stronger than previous research suggested.     

What is acceptable for women may not be for men: The effect of family conflicts with work on job‐performance ratings

We conducted a laboratory study examining the effect of a family conflict with work on performance appraisal ratings given to men and women. Overall, the experience of a family conflict was associated with lower performance ratings, and ratee sex moderated this relationship. Men who experienced a family conflict received lower overall performance ratings and lower reward recommendations than men who did not, whereas ratings of women were unaffected by the experience of a family conflict. The sex bias was not evident when performance was evaluated on the more specific dimension of planning. Neither rater gender nor work‐family role attitudes moderated the sex bias. Implications for future research and practice are discussed.     

A Meta-Analysis of the Relationship Between Rater Liking and Performance Ratings

This meta-analysis reviewed the magnitude and moderators of the relationship between rater liking and performance ratings. The results revealed substantial overlap between rater liking and performance ratings (ρ = .77). Although this relationship is often interpreted as indicative of bias, we review studies that indicate that to some extent the relationship between liking and performance ratings potentially reflects “true” differences in ratee performance. Moderator analyses indicated that the relationship between liking and performance ratings was weaker for ratings of organizational citizenship behaviors, ratings made by peer raters, ratings in nonsales jobs, and ratings made for development; however, the relationship was strong across moderator levels, underscoring the robustness of this relationship. Implications for the interpretation of performance ratings are discussed.     

DOES PERFORMANCE IMPROVE FOLLOWING MULTISOURCE FEEDBACK? A THEORETICAL MODEL,META‐ANALYSIS,AND REVIEW OF EMPIRICAL FINDINGS

We review evidence showing that multisource feedback ratings are related to other measures of leadership effectiveness and that different rater sources conceptualize performance in a similar manner. We then describe a meta-analysis of 24 longitudinal studies showing that improvement in direct report, peer, and supervisor ratings over time is generally small. We present a theoretical framework and review empirical evidence suggesting performance improvement should be more likely for some feedback recipients than others. Specifically, improvement is most likely to occur when feedback indicates that change is necessary, recipients have a positive feedback orientation, perceive a need to change their behavior, react positively to the feedback, believe change is feasible, set appropriate goals to regulate their behavior, and take actions that lead to skill and performance improvement.     

Reconsidering the Equivalence of Multisource Performance Ratings: Evidence for the Importance and Meaning of Rater Factors

Purpose

The study specified an alternate model to examine the measurement invariance of multisource performance ratings (MSPRs) to systematically investigate the theoretical meaning of common method variance in the form of rater effects. As opposed to testing invariance based on a multigroup design with raters aggregated within sources, this study specified both performance dimension and idiosyncratic rater factors.

Design/Methodology/Approach

Data was obtained from 5,278 managers from a wide range of organizations and hierarchical levels, who were rated on the BENCHMARKS^® MSPR instrument.

Findings

Our results diverged from prior research such that MSPRs were found to lack invariance for raters from different levels. However, same level raters provided equivalent ratings in terms of both the performance dimension loadings and rater factor loadings.

Implications

The results illustrate the importance of modeling rater factors when investigating invariance and suggest that rater factors reflect substantively meaningful variance, not bias.

Originality/Value

The current study applies an alternative model to examine invariance of MSPRs that allowed us to answer three questions that would not be possible with more traditional multigroup designs. First, the model allowed us to examine the impact of paramaterizing idiosyncratic rater factors on inferences of cross-rater invariance. Next, including multiple raters from each organizational level in the MSPR model allowed us to tease apart the degree of invariance in raters from the same source, relative to raters from different sources. Finally, our study allowed for inferences with respect to the invariance of idiosyncratic rater factors.     

Rater bias in the EASI temperament scales: a twin study

Under trait theory, ratings may be modeled as a function of the temperament of the child and the bias of the rater. Two linear structural equation models are described, one for mutual self- and partner ratings, and one for multiple ratings of related individuals. Application of the first model to EASI temperament data collected from spouses rating each other shows moderate agreement between raters and little rating bias. Spouse pairs agree moderately when rating their twin children, but there is significantly rater bias, with greater bias for monozygotic than for dizygotic twins. MLE's of heritability are approximately .5 for all temperament scales with no common environmental variance. Results are discussed with reference to trait validity, the person-situation debate, halo effects, and stereotyping. Questionnaire development using ratings on family members permits increased rater agreement and reduced rater bias.     

Effects of rater conscientiousness on evaluations of task and contextual performance of older and younger co-workers

The possibility for age discrimination and stereotypes to affect performance evaluations is rising. Although careful evaluations might be expected from conscientious raters, little is known about whether they might show more or less bias towards certain age groups. Therefore, in our study using a time-lagged design, we investigated the effects of rater conscientiousness on the performance evaluations of younger and older actual co-worker (N = 242). We found that raters who were more conscientious provided higher ratings for older workers than for younger workers on task performance and organizational citizenship behaviours. Specifically, we tested the model of mediated moderation, in which the relation between rater conscientiousness and ratee age predicts ratee-perceived conscientiousness, which in turn predicts performance ratings. The model was significant for older ratees, but not for younger ratees. We discuss our results in terms of the “similar to me” effects and implications for organizational practices.     

Rater Model Using Signal Detection Theory for Latent Differential Rater Functioning

Differential rater functioning (DRF) occurs when raters show evidence of exercising differential severity or leniency when scoring examinees within different subgroups. Previous studies of DRF have examined rater bias using manifest variables (e.g., use of covariates) to determine the subgroups. These manifest variables include gender and the ethnicity of the examinee. For example, a rater may score males more severely. Ideally, each rater’s severity should be invariant across subgroups. This study examines DRF in the context of latent subgroups that classify possible sources of DRF based on raters’ scoring behavior rather than manifest factors. An extension of the latent class signal detection theory (LC-SDT) model for identifying DRF is proposed and examined using real-world data and simulations. Results from real-world data show that the signal detection approach leads to an effective method to identify latent DRF. Simulations with varying sample sizes and conditions of rater precision were shown to recover parameters at an adequate level, supporting its use to identify latent DRF in large-scale data. These findings suggest that the DRF extension of the LC-SDT can be a useful model to examine characteristics of raters and add information that can aid rater training.     

新世纪20年国内心理统计方法研究回顾

新世纪头20年, 国内心理学11本专业期刊一共发表了213篇统计方法研究论文。研究范围主要包括以下10类(按论文篇数排序)：结构方程模型、测验信度、中介效应、效应量与检验力、纵向研究、调节效应、探索性因子分析、潜在类别模型、共同方法偏差和多层线性模型。对各类做了简单的回顾与梳理。结果发现, 国内心理统计方法研究的广度和深度都不断增加, 研究热点在相互融合中共同发展; 但综述类论文比例较大, 原创性研究论文比例有待提高, 研究力量也有待加强。     

RATER SOURCE EFFECTS ARE ALIVE AND WELL AFTER ALL

Recent research has questioned the importance of rater perspective effects on multisource performance ratings (MSPRs). Although making a valuable contribution, we hypothesize that this research has obscured evidence for systematic rater source effects as a result of misspecified models of the structure of multisource performance ratings and inappropriate analytic methods. Accordingly, this study provides a reexamination of the impact of rater source on multisource performance ratings by presenting a set of confirmatory factor analyses of two large samples of multisource performance rating data in which source effects are modeled in the form of second-order factors. Hierarchical confirmatory factor analysis of both samples revealed that the structure of multisource performance ratings can be characterized by general performance, dimensional performance, idiosyncratic rater, and source factors, and that source factors explain (much) more variance in multisource performance ratings whereas general performance explains (much) less variance than was previously believed. These results reinforce the value of collecting performance data from raters occupying different organizational levels and have important implications for research and practice.     

An autoregressive growth model for longitudinal item analysis

A first-order autoregressive growth model is proposed for longitudinal binary item analysis where responses to the same items are conditionally dependent across time given the latent traits. Specifically, the item response probability for a given item at a given time depends on the latent trait as well as the response to the same item at the previous time, or the lagged response. An initial conditions problem arises because there is no lagged response at the initial time period. We handle this problem by adapting solutions proposed for dynamic models in panel data econometrics. Asymptotic and finite sample power for the autoregressive parameters are investigated. The consequences of ignoring local dependence and the initial conditions problem are also examined for data simulated from a first-order autoregressive growth model. The proposed methods are applied to longitudinal data on Korean students’ self-esteem.     

Optimism and Positivity Biases in Performance Appraisal Ratings: Empirical Evidence from Professional Soccer

Using unique assessment data for players from a German Bundesliga club’s youth academy, we tested four core hypotheses on how player ratings and rater or ratee-related characteristics reflect the (prospective) optimism bias and (retrospective) positivity bias. The results indicate not only that the ratings of predicted and remembered performance are indeed higher than the talents’ actual performance throughout a season, but that these differences depend positively on the rater’s organizational experience and negatively on the amount of ratee data available. They also suggest that (prospective) anticipation is even more positively biased than (retrospective) recollection of player performances, underscoring the asymmetry between looking forward and looking backward.     

Rater bias in psychological research: when is it a problem and what can we do about it?

Rater bias is a substantial source of error in psychological research. Bias distorts observed effect sizes beyond the expected level of attenuation due to intrarater error, and the impact of bias is not accurately estimated using conventional methods of correction for attenuation. Using a model based on multivariate generalizability theory, this article illustrates how bias affects research results. The model identifies 4 types of bias that may affect findings in research using observer ratings, including the biases traditionally termed leniency and halo errors. The impact of bias depends on which of 4 classes of rating design is used, and formulas are derived for correcting observed effect sizes for attenuation (due to bias variance) and inflation (due to bias covariance) in each of these classes. The rater bias model suggests procedures for researchers seeking to minimize adverse impact of bias on study findings.     

Competencies,Personality Traits,and Organizational Rewards of Middle Managers: A Motive-Based Approach

Previous literature suggests that performance ratings are saturated with rater-related idiosyncratic variance. Given that modern psychometric theories relegate this source of variance to measurement error, it has not been the subject of much previous research. Of importance, identifying and estimating the variance components underlying idiosyncratic rater variance will inform our understanding of the nature of this variance. In a sample of managerial performance ratings we report on components of variance and find that the idiosyncratic rater variance component is about one third rater main effects variance, one third Rater × Ratee interaction effects variance, and one third upper-bound Rater × Ratee × Dimension interaction effects variance. Further, results indicate that variance components are moderated by the acquaintanceship time between the rater and the ratee.     

Rater biases in genetically informative research designs: comment on Bartels, Boomsma, Hudziak, van Beijsterveldt, and van den Oord (2007)

Rater biases are of interest to behavior genetic researchers, who often use ratings data as a basis for studying heritability. Inclusion of multiple raters for each sibling pair (M. Bartels, D. I. Boomsma, J. J. Hudziak, T. C. E. M. van Beijsterveldt, & E. J. C. G. van den Oord, see record 2007-18729-006) is a promising strategy for controlling bias variance and may yield information about sources of bias in heritability studies. D. A. Kenny's (2004) PERSON model is presented as a framework for understanding determinants of rating reliability and validity. Empirical findings on rater bias in other contexts provide a starting point for addressing the impact of rater-unique perceptions in heritability studies. However, heritability studies use distinctive rating designs that may accentuate some sources of bias, such as rater communication and contrast effects, which warrant further study.     

Validity of observer ratings of the five-factor model of personality traits: a meta-analysis

Conclusions reached in previous research about the magnitude and nature of personality-performance linkages have been based almost exclusively on self-report measures of personality. The purpose of this study is to address this void in the literature by conducting a meta-analysis of the relationship between observer ratings of the five-factor model (FFM) personality traits and overall job performance. Our results show that the operational validities of FFM traits based on observer ratings are higher than those based on self-report ratings. In addition, the results show that when based on observer ratings, all FFM traits are significant predictors of overall performance. Further, observer ratings of FFM traits show meaningful incremental validity over self-reports of corresponding FFM traits in predicting overall performance, but the reverse is not true. We conclude that the validity of FFM traits in predicting overall performance is higher than previously believed, and our results underscore the importance of disentangling the validity of personality traits from the method of measurement of the traits.     

An investigation of the generalizability and dependability of Direct Behavior Rating Single Item Scales (DBR-SIS) to measure academic engagement and disruptive behavior of middle school students

A total of 4 raters, including 2 teachers and 2 research assistants, used Direct Behavior Rating Single Item Scales (DBR-SIS) to measure the academic engagement and disruptive behavior of 7 middle school students across multiple occasions. Generalizability study results for the full model revealed modest to large magnitudes of variance associated with persons (students), occasions of measurement (day), and associated interactions. However, an unexpectedly low proportion of the variance in DBR data was attributable to the facet of rater, as well as a negligible variance component for the facet of rating occasion nested within day (10-min interval within a class period). Results of a reduced model and subsequent decision studies specific to individual rater and rater type (research assistant and teacher) suggested degree of reliability-like estimates differed substantially depending on rater. Overall, findings supported previous recommendations that in the absence of estimates of rater reliability and firm recommendations regarding rater training, ratings obtained from DBR-SIS, and subsequent analyses, be conducted within rater. Additionally, results suggested that when selecting a teacher rater, the person most likely to substantially interact with target students during the specified observation period may be the best choice.     

多变量追踪研究的模型整合与拓展：考察往复式影响与增长趋势

追踪研究当中, 交叉滞后模型可以探究多变量之间往复式影响, 潜增长模型可以探究个体增长趋势。对两类模型进行整合, 例如同时关注往复式影响与个体增长趋势, 同时可以定义测量误差、随机截距等变异成分, 衍生出随机截距交叉滞后模型、特质-状态-误差模型、自回归潜增长模型、结构化残差潜增长模型等。以交叉滞后模型和潜增长模型分别作为基础模型, 从个体间/个体内变异分解的角度对上述各类模型梳理, 整合出此类模型的分析框架, 并拓展建立“因子结构化潜增长模型(factor latent curve model with structured reciprocals)”作为统合框架。通过实证研究(早期儿童的追踪研究-幼儿园版, ECLS-K), 建立21049名儿童的阅读和数学能力的往复式影响与增长趋势。研究发现, 分离了稳定特质的模型拟合最优。研究也对模型建模思路和模型选择提供了建议。