Measurement Equivalence and Multisource Ratings for Non-managerial Positions: Recommendations for Research and Practice

ABSTRACT: The present investigation applies a comprehensive sequence of confirmatory factor analysis tests (Vandenberg and Lance, Organisational Research Methods, 3, 4–69, 2000) to the examination of the measurement equivalence of self, peer, and supervisor ratings of non-managerial targets across several performance dimensions. Results indicate a high degree of measurement equivalence across rater sources and performance dimensions. The paper illustrates how this procedure can identify very specific areas of non-equivalence and how the complexity of a multisource feedback system may be represented using such procedures. Implications of these results and recommendations for both research and practice are offered.     

Internationalizing 360-Degree Feedback: Are Subordinate Ratings Comparable?

ABSTRACT: This study addressed whether 360-degree feedback ratings made by subordinates are equivalent across national cultures (Great Britain, Hong Kong, Japan, U.S.). Employees ratings of their leaders were collected via a companys existing 360-degree feedback survey. A two-factor baseline model was developed using exploratory factor analysis and tested across cultures utilizing confirmatory factor analysis techniques. Results revealed differences in factor form, suggesting that the constructs underlying the survey and their relationship to the survey items differed across cultures. These results emphasize the need for multinational companies to use caution when transporting human resources practices, such as 360-degree feedback, to international locations.The author would like to express appreciation to Alice Stuhlmacher for her valuable comments on earlier versions of this article. This research was conducted as part of the authors doctoral dissertation. Portions of this article were presented at the 17th Annual Conference of the Society for Industrial and Organizational Psychology, Toronto, Ontario, Canada, April 2002.     

Managerial Experience and the Measurement Equivalence of Performance Ratings

ABSTRACT: Establishing the measurement equivalence of instruments is a prerequisite to making meaningful comparisons between individuals or within individuals over time. Whereas previous research has investigated the effects of rater characteristics on the measurement equivalence of performance ratings, the current study investigated a ratee characteristic—ratee job experience. Using confirmatory factor analysis and item response theory methods with replication, the measurement equivalence of supervisor ratings of 7,200 managers with differing levels of managerial experience was assessed. Overall, results indicated a high degree of measurement equivalence suggesting that meaningful comparison may be made across ratees with different levels of job experience.     

Associative symmetry in the pigeon after successive matching-to-sample training

If an organism is explicitly taught an A→B association, then might it also spontaneously learn the symmetrical B→A association? Little evidence attests to such “associative symmetry” in nonhuman animals. We report for the first time a clear case of associative symmetry in the pigeon. Experiment 1 used a successive go/no go matching‐to‐sample procedure, which showed all of the training and testing stimuli in one location and intermixed arbitrary and identity matching trials. We found symmetrical responding that was as robust during testing (B→A) as during training (A→B). In Experiment 2, we trained different pigeons using only arbitrary matching trials before symmetry testing. No symmetrical responding was found. In Experiment 3, we trained other pigeons with only arbitrary matching trials and then tested for symmetry. When these pigeons, too, did not exhibit symmetrical responding, we retrained them with intermixed identity and arbitrary matching trials. Less robust symmetrical responding was obtained here than in Experiment 1. Collectively, these results suggest that identity matching may have to be learned concurrently with arbitrary matching from the outset of training for symmetry to emerge.     

Stability of functional equivalence and stimulus equivalence: effects of baseline reversals

Functional equivalence and stimulus equivalence classes were established, reversed, and tested for stability with college students. Functional stimulus classes were established using a task in which students were trained to say nonsense words in the presence of arbitrarily assigned sets of symbols. Computer-controlled speech-recognition technology was used to record and analyze students' vocal responses for accuracy. After the establishment of stimulus classes was demonstrated with a transfer-of-function test, the effects of reversing selected baseline simple discriminations were assessed during an additional transfer-of-function test and a follow-up test that occurred several weeks later. With the same students, stimulus equivalence classes were established and demonstrated with computerized matching-to-sample procedures. The effects of reversing selected baseline conditional discriminations also were assessed during a postreversal equivalence test and a follow-up test. Both functional stimulus classes and stimulus equivalence were sensitive to contingency reversals, but the reversals with stimulus equivalence closses affected stimulus class organization whereas reversals with functional stimulus classes did not. Follow-up performances were largely consistent with the original baseline contingencies. The similarities and differences between stimulus equivalence and functional equivalence are related to the specific contingencies that select responding in the presence of the stimuli that form the classes.     

Contextual control of stimulus generalization and stimulus equivalence in hierarchical categorization

The purpose of this study was to determine whether hierarchical categorization would result from a combination of contextually controlled conditional discrimination training, stimulus generalization, and stimulus equivalence. First, differential selection responses to a specific stimulus feature were brought under contextual control. This contextual control was hierarchical in that stimuli at the top of the hierarchy all evoked one response, whereas those at the bottom each evoked different responses. The evocative functions of these stimuli generalized in predictable ways along a dimension of physical similarity. Then, these functions were indirectly acquired by a set of nonsense syllables that were related via transitivity relations to the originally trained stimuli. These nonsense syllables effectively served as names for the different stimulus classes within each level of the hierarchy.     

Derived relational responding as generalized operant behavior

The major aim of the present study was to demonstrate that derived relational responding may be viewed as a form of generalized operant behavior. In Experiment 1, 4 subjects were divided into two conditions (2 in each condition). Using a two-comparison matching-to-sample procedure, all subjects were trained and tested for the formation of two combinatorially entailed relations. Subjects were trained and tested across multiple stimulus sets. Each set was composed of novel stimuli. Both Conditions 1 and 2 involved explicit performance-contingent feedback presented at the end of each block of test trials (i.e., delayed feedback). In Condition 1, feedback was accurate (consistent with the experimenter-designated relations) following exposure to the initial stimulus sets. When subjects' responding reached a predefined mastery criterion, the feedback then switched to inaccurate (not consistent with the experimenter-designated relations) until responding once again reached a predefined criterion. Condition 2 was similar to Condition 1, except that exposure to the initial stimulus sets was followed by inaccurate feedback and once the criterion was reached feedback switched to accurate. Once relational responding emerged and stabilized, response patterns on novel stimulus sets were controlled by the feedback delivered for previous stimulus sets. Experiment 2 replicated Experiment 1, except that during Conditions 3 and 4 four comparison stimuli were employed during training and testing. Experiment 3 was similar to Condition 1 of Experiment 1, except that after the mastery criterion was reached for class-consistent responding, feedback alternated from accurate to inaccurate across each successive stimulus set. Experiment 4 involved two types of feedback, one type following tests for mutual entailment and the other type following tests for combinatorial entailment. Results from this experiment demonstrated that mutual and combinatorial entailment may be controlled independently by accurate and inaccurate feedback. Overall, the data support the suggestion, made by relational frame theory, that derived relational responding is a form of generalized operant behavior.