Reconsidering the Equivalence of Multisource Performance Ratings: Evidence for the Importance and Meaning of Rater Factors

Purpose

The study specified an alternate model to examine the measurement invariance of multisource performance ratings (MSPRs) to systematically investigate the theoretical meaning of common method variance in the form of rater effects. As opposed to testing invariance based on a multigroup design with raters aggregated within sources, this study specified both performance dimension and idiosyncratic rater factors.

Design/Methodology/Approach

Data was obtained from 5,278 managers from a wide range of organizations and hierarchical levels, who were rated on the BENCHMARKS^® MSPR instrument.

Findings

Our results diverged from prior research such that MSPRs were found to lack invariance for raters from different levels. However, same level raters provided equivalent ratings in terms of both the performance dimension loadings and rater factor loadings.

Implications

The results illustrate the importance of modeling rater factors when investigating invariance and suggest that rater factors reflect substantively meaningful variance, not bias.

Originality/Value

The current study applies an alternative model to examine invariance of MSPRs that allowed us to answer three questions that would not be possible with more traditional multigroup designs. First, the model allowed us to examine the impact of paramaterizing idiosyncratic rater factors on inferences of cross-rater invariance. Next, including multiple raters from each organizational level in the MSPR model allowed us to tease apart the degree of invariance in raters from the same source, relative to raters from different sources. Finally, our study allowed for inferences with respect to the invariance of idiosyncratic rater factors.     

Not Much More Than g? An Examination of the Impact of Intelligence on NFL Performance

The purpose of this study was to determine the efficiency and equity of general mental ability (GMA) in a nontraditional employment setting—professional football. The National Football League (NFL) uses a measure of GMA, the Wonderlic Personnel Test, to evaluate potential draftees in an assessment-style environment. A total of 762 NFL players, represented from three draft classes, were included in our sample. In terms of efficiency, results indicated that GMA was unrelated to (a) future NFL performance, (b) selection decisions during the NFL Draft, and (c) the number of games started in the NFL. In regards to equity, differential prediction analyses by race suggested only the existence of intercept bias. The implications of these findings to the NFL and the selection literature are further discussed.     

Just Think About It: Mindfulness, Sexism, and Prejudice Toward Feminists

This study examined the relation between mindfulness, sexist motivations and beliefs, and prejudice toward women who violate traditional gender roles—namely feminists. In a preliminary study, 672 (251 men, 421 women) undergraduates from a United States Midwestern university completed a measure of mindfulness and warmth toward feminists and consistent with hypotheses, more mindfulness was associated with more warmth toward feminists. Extending this initial finding to the main study, 653 (273 men, 380 women) undergraduates from a U.S. Midwestern university completed measures of mindfulness, motivation to respond without sexism, ambivalent sexism, and warmth toward feminists. Consistent with hypotheses, compared to women participants, men participants were lower on internal motivation to respond without sexism, higher on hostile and benevolent sexism and less warm toward feminists. Also consistent with hypotheses, for men participants, more mindfulness was associated with higher internal motivation to respond without sexism, less benevolent sexism, and more warmth toward feminists. In contrast, for women participants, more mindfulness was only associated with less hostile sexism. Finally, a path analysis revealed that the positive relation between mindfulness and warmth toward feminists for men participants was partially mediated by more internal motivation to respond without sexism (i.e., a significant indirect effect), but not by less sexist beliefs. Implications for mindfulness, sexism, and prejudice more generally are discussed.     

Parent–child interaction therapy (PCIT) in young children with autism spectrum disorder

Abstract

Parent–child interaction therapy (PCIT) is an evidence-based treatment for typically developing children with disruptive behavior. We conducted a randomized-controlled trial of PCIT versus wait-list control (WLC) with 23 children with ASD (3–7?years) and disruptive behavior. Over 16 treatment sessions, PCIT significantly predicted reductions in disruptive behavior over WLC and explained a significant variation in scores on the ECBI Intensity subscale. Additionally, parent skills improved significantly compared to WLC. However, no statistically significant group differences were found on child compliance rates, autism severity, or parental stress. Results support PCIT as an evidence-based treatment for disruptive behavior in ASD.

• Highlights

• Parent skills were significantly improved for those receiving PCIT

• Intensity of disruptive behaviors decreased significantly for those receiving PCIT

• Parental stress and autism severity did not significantly decrease with PCIT

     

Searching for the sequential line-up advantage: A distinctiveness explanation

Signal Detection models as well as the Two-High-Threshold model (2HTM) have been used successfully as measurement models in recognition tasks to disentangle memory performance and response biases. A popular method in recognition memory is to elicit confidence judgements about the presumed old/new status of an item, allowing for the easy construction of ROCs. Since the 2HTM assumes fewer latent memory states than response options are available in confidence ratings, the 2HTM has to be extended by a mapping function which models individual rating scale usage. Unpublished data from 2 experiments in Bröder and Schütz (2009) validate the core memory parameters of the model, and 3 new experiments show that the response mapping parameters are selectively affected by manipulations intended to affect rating scale use, and this is independent of overall old/new bias. Comparisons with SDT show that both models behave similarly, a case that highlights the notion that both modelling approaches can be valuable (and complementary) elements in a researcher's toolbox.     

The role of eye fixations in concentration and amplification effects during multiple object tracking

When tracking spatially extended objects in a multiple object tracking task, attention is preferentially directed to the centres of those objects (attentional concentration), and this effect becomes more pronounced as object length increases (attentional amplification). However, it is unclear whether these effects depend on differences in attentional allocation or differences in eye fixations. We addressed this question by measuring eye fixations in a dual-task paradigm that required participants to track spatially extended objects, while simultaneously detecting probes that appeared at the centres or near the endpoints of objects. Consistent with previous research, we observed concentration and amplification effects: Probes at the centres of objects were detected more readily than those near their endpoints, and this difference increased with object length. Critically, attentional concentration was observed when probes were equated for distance from fixation during free viewing, and concentration and amplification were observed without eye movements during strict fixation. We conclude that these effects reflect the prioritization of covert attention to particular spatial regions within extended objects, and we discuss the role of eye fixations during multiple object tracking.     

Multilevel Models for Examining Individual Differences in Within-Person Variation and Covariation Over Time

Heterogeneity of variance may be more than a statistical nuisance—it may be of direct interest as a result of individual differences. In studies of short-term fluctuation, individual differences may relate to the magnitude of within-person variation as well as to level of an outcome or its covariation with other processes. Although models for heterogeneous variances have been utilized in group contexts (i.e., dispersion models), they are not usually applied in examinations of intraindividual variation. This work illustrates how an extension of the multilevel model for heterogeneous variances can be used to examine individual differences in level, between- and within-person covariation, and magnitude of within-person variation of daily positive and negative mood in persons with dementia.     

Technology and Older Adults: Factors Affecting the Adoption of Automatic Teller Machines

Although technology can benefit service providers, caregivers, and the elderly, its application in an aging society can bring special challenges. This study looked at older adults' adoption of one technology that is highly prevalent in modern society—the automatic teller machine (ATM). The findings indicated that users and nonusers differed in mechanical reasoning skills and in attitudes toward ATM technology. Older adults with higher mechanical reasoning skills were more likely to be ATM users. Nonusers had more negative attitudes toward ATMs, and, among non-users, those who had tried an ATM had more positive attitudes than those who had never tried one. The findings of this study are discussed in terms of factors that may affect the adoption of other technologies by older adults. Suggestions for increasing the acceptance of technologies by the elderly are also addressed.     

Simultaneous Semipermeable Groups for Mothers and Their Early Latency-Age Boys

Abstract

Bonding is an ongoing aspect of intersubjective experience, a type of mental relationship to oneself and others. As a behavioral mode, bonding utilizes verbal and nonverbal communication primarily to establish a basic feeling of connection between individuals. Bonding is the floor, a cognitive-affective state that precedes and prepares the way for the complex processes underlying identification. The concept relates to Bion’s early theory of basic assumptions, and takes on new and different meanings according to his later ideas regarding social communication. Bonding is also a therapeutic posture or technique, an element in the establishment of group cohesion, the positive transference, and therapeutic and group alliance. Case examples illustrate how the therapist and other group members bring a complex of bonding needs and anxieties that must be negotiated to establish and maintain bonding, which is a necessary condition for successful therapeutic work.     

Heidegger and the Problem of Idealism

This paper contains a discussion of Quine's thesis of indeterminacy of translation within the more general thesis that using and understanding a language are to be conceived of as a creative and interpretative‐constructional activity. Indeterminacy is considered to be ineliminable. Three scenarios are distinguished concerning, first, the reasons for indeterminacy, second, the kinds of indeterminacy and, third, different levels of a general notion of recursive interpretation. Translational hypotheses are seen as interpretational constructs. The indeterminacy thesis turns out to be a consequence of the externalizing of language, meaning, and epistemology. By means of a three‐leveled interpretation model one can substantiate the crucial aspects, first, that indeterminacy is not an indeterminacy of facts of the matter and, second, that there is a significant difference between indeterminacy and underdetermination. In addition, the relationship between indeterminacy, interpretation, and charity is elucidated. Indeterminacy is seen not as an obstacle to but as a condition for communication. Charity and empathy in dialogue are conditional upon indeterminacy. All three components reveal the interpretative‐constructional character of the inseparable connection of meaning and experience.