MEASUREMENT ERROR IN RESEARCH ON HUMAN RESOURCES and FIRM PERFORMANCE: HOW MUCH ERROR IS THERE AND HOW DOES IT INFLUENCE EFFECT SIZE ESTIMATES?

Studies of the relationship between human resource (HR) practices and firm performance typically use a single respondent to assess firm level HR practices or HR effectiveness. However, previous research in other substantive areas suggests that rater differences are a potentially important source of measurement error. We demonstrate analytically the potential consequences of both random and systematic measurement error in research on HR and firm performance. However, our main focus is on random error and we show how generalizability theory can be applied to obtain better estimates of reliability by simultaneously recognizing multiple sources (e.g., items, raters) of random measurement error. These more inclusive reliability estimates, in turn, offer the possibility of more precisely quantifying substantive relationships in the HR and firm performance literature. In our sample, reliabilities (as estimated by generalizability coefficients) for single-rater assessments of HR variables were generally below .50. This degree of measurement error, if present in substantive studies on HR and firm performance, could lead to considerable bias, given that an unstandardized regression coefficient is corrected for measurement error in the independent variable by dividing by its reliability coefficient (not its square root). We also found only limited convergent validity between HR and line managers ratings of a second type of HR measure, HR effectiveness. In general, our findings suggest that future researchers need to devote greater attention to measurement error and construct validity issues. Our study provides an example of how generalizability theory can be useful in this pursuit.     

Encoding in anticipation of various tests of retention

Abstract.— Subjects learned a pictorial material in anticipation of either free recall (FR), serial recall (SR), or recognition tests. A design containing all possible combinations of anticipated test and test actually given was used. SR and recognition performance was best when subjects anticipated these tests, respectively, whereas FR performance was best when an SR test was anticipated. Anticipation of recognition tended to interfere with SR performance, and vice versa. The results indicate that subjects encode pictorial material differently in anticipation of different retention tests; that this serves to facilitate or to impair performance on the anticipated and/or other retention tests in a predictable manner; and that subjects tend to use different information from the stimuli to pass recognition tests and to pass recall (FR or SR) tests.     

Effects of context variation, intracategory variation, and spacing on the free recall of pictures

Abstract.— Two experiments examined the effects of encoding variability on the free recall of pictures. In Expt. I, drawn objects were presented 3 times in different pictorial contexts (RV) or in the same contexts (RR). In Expt. II, repeated stimuli were 3 different specimens of the same object category (VR) or the same object presented 3 times (RR). Spacing of repetitions was varied from zero to 15, and a mixed-list design was used. Both RV and VR items tended to be recalled better than and prior to RR items. Recall of RR and of VR items increased with increasing spacing, but recall of RV items showed no spacing effect. In a second recall test, preceded by 4 min of "thinking about" the pictures, the only remaining effect of spacing or of context variation was a low recall of massed RR items. VR items were still recalled better than and prior to RR items. It is suggested that spacing, and variation of the external context of pictures, facilitate retrieval when the time and effort spent on recall is limited, by increasing the numberlvariety of potential recall cues stored together with item information.     

A LONGITUDINAL EXAMINATION OF RATER AND RATEE EFFECTS IN PERFORMANCE RATINGS

The consistency and loci of leniency, halo, and range restriction effects in performance ratings were investigated in a longitudinal study. Ratings were provided by approximately 90 supervisors in a metropolitan police department, who rated approximately 350 police-rank subordinates on five occasions over a three and one-half year period. Rating effects were computed separately as rater-and ratee-based statistics, and intercorrelated among the five rating periods. The nature of the data set made it possible to hold either raters or ratees constant for each analysis, thus permitting inferences regarding the sources of reliable variance in effects as due to raters or ratees. It was concluded that reliable variance in mean ratings is partly attributable to ratees, but mainly introduced by raters. Reliable halo variance is attributable to raters, and range restriction is a product of stable group performance variability within intact ratee groups. Implications of these results for future rating process research are discussed.     

On the development of encoding in anticipation of various tests of retention

Abstract.— Seven, 9, 12–13, and 18–21-year-old subjects learned a pictorial material in anticipation of free recall (FR), serial recall (SR), or recognition tests. Type of anticipated test and test actually given were factorially combined, retention being tested after 2 min or after 2 weeks. Recall performance improved with age but recognition performance did not improve after CA 9. Differential anticipation had little effect on the performance of the youngest children. For older subjects the effects were clear-cut and similar at all age levels and at both retention intervals. SR and recognition performance was best when subjects anticipated these tests, respectively; anticipation of recognition interfered with recall performance, and vice versa. The main age-related improvement occurred between age levels 7 and 9.     

Distribution of probability assessments for almanac and future event questions

This study examines the distribution and calibration of probability assessments given to general knowledge questions and question concerning future events. Two experiments revealed that: (1) People use certainty responses less frequently in response to questions concerning then-future events than to general knowledge questions even when the then-future event questions are easier than the general knowledge questions. (2) Indonesian students previously thought to have little grasp of probabilistic thinking, are able to give realistic probability assessments for then-future events. Cultural and task influences on our findings are discussed in relation to a procedural model of the processes involved in answering a question. We conclude that, as most applications of decision analysis involve future uncertainty, research in probability assessment should concentrate on questions concerning future events rather than on general knowledge questions.     

THE RELATIONSHIP BETWEEN HR PRACTICES AND FIRM PERFORMANCE: EXAMINING CAUSAL ORDER

Significant research attention has been devoted to examining the relationship between HR practices and firm performance, and research support has assumed HR as the causal variable. Using data from 45 business units (with 62 data points), this study examines how measures of HR practices correlate with past, concurrent, and future operational performance measures. The results indicate that correlations with performance measures at all 3 times are both high and invariant, and that controlling for past or concurrent performance virtually eliminates the correlation of HR practices with future performance. Implications are discussed.