Reconsidering the Equivalence of Multisource Performance Ratings: Evidence for the Importance and Meaning of Rater Factors

Purpose

The study specified an alternate model to examine the measurement invariance of multisource performance ratings (MSPRs) to systematically investigate the theoretical meaning of common method variance in the form of rater effects. As opposed to testing invariance based on a multigroup design with raters aggregated within sources, this study specified both performance dimension and idiosyncratic rater factors.

Design/Methodology/Approach

Data was obtained from 5,278 managers from a wide range of organizations and hierarchical levels, who were rated on the BENCHMARKS^® MSPR instrument.

Findings

Our results diverged from prior research such that MSPRs were found to lack invariance for raters from different levels. However, same level raters provided equivalent ratings in terms of both the performance dimension loadings and rater factor loadings.

Implications

The results illustrate the importance of modeling rater factors when investigating invariance and suggest that rater factors reflect substantively meaningful variance, not bias.

Originality/Value

The current study applies an alternative model to examine invariance of MSPRs that allowed us to answer three questions that would not be possible with more traditional multigroup designs. First, the model allowed us to examine the impact of paramaterizing idiosyncratic rater factors on inferences of cross-rater invariance. Next, including multiple raters from each organizational level in the MSPR model allowed us to tease apart the degree of invariance in raters from the same source, relative to raters from different sources. Finally, our study allowed for inferences with respect to the invariance of idiosyncratic rater factors.     

Linking Feedback Quality and Goal Orientation to Feedback Seeking and Job Performance

Contemporary feedback researchers have adopted theoretical perspectives in which personal characteristics interact with aspects of the feedback environment to influence feedback-related perceptions, feedback seeking, and job performance. To test these assertions, this study incorporates implicit person theory, uncertainty reduction theory, and Korman's theory of work motivation (2001) to develop a model that links a critical aspect of the feedback environment (feedback quality) and goal orientation to perceptions of feedback utility, feedback seeking, role clarity, and task performance and organizational citizenship behaviors. Results from supervisor–subordinate dyadic data (N?=?202) obtained from employed students largely support the hypothesized model. Implications are discussed.     

Challenges in Transforming Military Training: Research and Application of Advanced Simulation and Training Technologies and Methods

This study addresses the need to validate estimates of cross-job retraining times against measures of actual retraining success. Specialty-level cross-job retraining time estimates were computed for 43 U.S. Air Force (USAF) enlisted specialties. The estimates were based on profiles of mean subject- matter-expert ratings of months to proficiency across 26 task categories. Individual-level criteria indicating previous job skill utilization, current skills and abilities, and current job performance were obtained from a separate study conducted 10 years earlier. Results were supportive: Correlations between retraining time estimates and performance criteria were negative and statistically significant. Future research needs and potential applications of retraining time estimates are discussed.     

Psychological impact of remote combat/graphic media exposure among US Air Force intelligence personnel

Since 2001 there has been a significant increase in the use of intelligence, surveillance, and reconnaissance (ISR) analysis for tactical, operational, and strategic decision makers engaged in global operations. To meet this demand, US Air Force intelligence personnel participate in remote combat and graphic media exploitation operations (e.g., review of still imagery, video, and audio), the long-term psychological effects of which are not well understood. Research to date has focused primarily on outcomes related to how intelligence personnel work, versus the specifics of what they do. Military psychologists embedded in ISR units conducted studies to address this gap. Intelligence analysts participated in focus groups and surveys assessing the frequency of exposure, previous exposures to other potentially traumatic events, symptoms of PTSD, moral injury, and other psychosocial experiences. Results showed that exposure levels, albeit virtual, rivaled or exceeded those reported by a sample of special operations forces. Results also showed that specific types of exposures (e.g., witnessing US military casualties, civilian casualties, atrocities committed by the enemy) are related to increased posttraumatic stress and other sequelae that may not adequately be captured by standard posttraumatic stress disorder screening measures. The results contribute to the existing literature on posttraumatic stress, shed new light on the emerging construct of moral injury, and highlight challenges presented by remote combat and graphic media exploitation operations to force health sustainment and performance optimization. The authors provide directions for future research and recommendations for ongoing assessment, monitoring, and selection and training of ISR personnel.     

Killing Versus Witnessing Trauma: Implications for the Development of PTSD in Combat Medics

Killing in combat uniquely predicts elevated PTSD symptomatology among military veterans. This study investigated the effects of combat killing in a sample of 345 U.S. Army combat medics who had recently returned from operational deployments to Iraq or Afghanistan. Combat medics provide frontline medical care before, during, and after battles but also fight alongside other soldiers when under attack. Attempting to kill in combat was a significant predictor of PTSD symptomatology even after accounting for passively witnessing trauma in fellow soldiers. Medics may be well prepared to cope with the passive experiencing and witnessing of war-zone trauma, but may benefit from training to cope with the negative consequences of taking actions to kill.     

Not Much More Than g? An Examination of the Impact of Intelligence on NFL Performance

The purpose of this study was to determine the efficiency and equity of general mental ability (GMA) in a nontraditional employment setting—professional football. The National Football League (NFL) uses a measure of GMA, the Wonderlic Personnel Test, to evaluate potential draftees in an assessment-style environment. A total of 762 NFL players, represented from three draft classes, were included in our sample. In terms of efficiency, results indicated that GMA was unrelated to (a) future NFL performance, (b) selection decisions during the NFL Draft, and (c) the number of games started in the NFL. In regards to equity, differential prediction analyses by race suggested only the existence of intercept bias. The implications of these findings to the NFL and the selection literature are further discussed.     

Marriage encounter casualties: A preliminary investigation

Abstract

The purpose of this exploratory study was to investigate the relationship between participation in Marriage Encounter, the largest marriage enrichment program in the world, and subsequent marital or family distress. Analyzing 13 case reports from seven marital therapists, the authors suggest that Marriage Encounter weekends can cause marital or family deterioration for some couples through increased marital conflict, avoidance of constructive problem solving, or marital enmeshment at the expense of children. It appears that the most harmful aspect of the Marriage Encounter experience is its induction of intense couple-centered communication, leading to emotional overload in some couples. The authors offer several recommendations intended to lower the risks associated with participation in Marriage Encounter.     

The ethics of selection quotas in South African sport1

This article explores and unpacks the public debate on the ethics of applying selection quotas to South African international sport sides to achieve transformation, with special attention to cricket and rugby, the Proteas and Springboks respectively. I claim that for quotas to be morally called for, the racial transformation they are in service of must be morally required. Following an earlier article of mine on the subject of transformation in South African sport, I briefly reject two manifestations of the goal of having the sides’ racial demographics match those of the country as a whole, and proceed to defend and develop two different transformational requirements: the ending of ongoing racial bias and the provision of equality of opportunity. I argue that while there may be a place for alternative forms of affirmative selection, the case for rigid quotas is marginal at best. While I do not argue for the claim here, it neither follows from this, nor do I believe, that the use of quotas is morally wrong. I argue only that the case for them is not particularly strong and depends, particularly, on the extent to which ongoing racial bias in selection persists.     

Guilt in response to blame from others

In three studies, participants rated appraisals and emotions experienced when someone else blamed them for something that was not their fault. Several participants spontaneously reported experiencing guilt in each study. Using event-contingent diaries, Study 1 found only weak correlations between rated self-blame and reported guilt when participants were blamed unreasonably. Using directed retrospective recall, Studies 2 and 3 found that guilt was higher in blamed than unblamed conditions when self-blame was low, and that the desire to apologise remained a significant predictor of guilt after controlling for all relevant appraisal dimensions. Taken together, these findings suggest that self-blame-related appraisals are not necessary conditions for the experience of guilt, and support an interpersonal analysis that sees this emotion as a strategy for repairing relationships after perceived (but not always genuine) slights.     

Recognising subtle emotional expressions: The role of facial movements

Two studies investigated the importance of dynamic temporal characteristic information in facilitating the recognition of subtle expressions of emotion. In Experiment 1 there were three conditions, dynamic moving sequences that showed the expression emerging from neutral to a subtle emotion, a dynamic presentation containing nine static stills from the dynamic moving sequences (ran together to encapsulate a moving sequence) and a First–Last condition containing only the first (neutral) and last (subtle emotion) stills. The results showed recognition was significantly better for the dynamic moving sequences than both the Dynamic-9 and First–Last conditions. Experiments 2a and 2b then changed the dynamics of the moving sequences by speeding up, slowing down or disrupting the rhythm of the motion sequences. These manipulations significantly reduced recognition, and it was concluded that in addition to the perception of change, recognition is facilitated by the characteristic muscular movements associated with the portrayal of each emotion.