Statistical properties of the rWG(J) index of agreement

L. R. James, R. G. Demaree, and G. Wolf (1984) introduced rWG(J) to estimate interrater agreement for a group. This index is calculated by comparing an observed group variance with an expected random variance. As researchers have gained experience using this index, several questions have arisen. What are the consequences of replacing values beyond the unit interval by 0? What is the dependence of rWG(J) on the group size? The authors' simulations show that a positive bias is caused by the truncation, but for large population values of rWG(J) it is negligible. Also, in this case, the group size has no effect on the expected value of rWG(J). For inference on rWG(J), researchers can exploit the availability of computers to simulate data from the hypothesized distribution and then compare the simulation results for rWG(J) with the actual values. In addition, it is shown how the bootstrap method can be used for comparing the indices of 2 groups.     

The Relative Importance and Interaction of Contextual and Methodological Predictors of Mean rWG for Work Climate

A variety of collective phenomena are understood to exist to the extent that workers agree on their perceptions of the phenomena, such as perceptions of their organization’s climate or perceptions of their team’s mental model. Researchers conducting group-level studies of such phenomena measure individuals’ perceptions via surveys and then aggregate data to the group level if the mean within-group agreement for a sample of groups is sufficiently high. Despite this widespread practice, we know little about the factors potentially affecting mean within-group agreement. Here, focusing on work climate, we report an investigation of a number of expected contextual (social interaction) and methodological predictors of mean r_WG, a common statistic for judging within-group agreement in applied psychology and management research. We used the novel approach of meta-CART, which allowed us to assess the relative importance and possible interactions of the predictor variables. Notably, mean r_WG values are driven by both contextual (average number of individuals per group and cultural individualism-collectivism) and methodological factors (the number of items in a scale and scale reliability). Our findings are largely consistent with expectations concerning how social interaction affects within-group agreement and psychometric arguments regarding why adding more items to a scale will not necessarily increase the magnitude of an index based on a Spearman-Brown “stepped-up correction.” We discuss the key insights from our results, which are relevant to the study of multilevel phenomena relying on the aggregation of individual-level data and informative for how meta-analytic researchers can simultaneously examine multiple moderator variables.

     

Further Insight and Additional Inference Methods for Polynomial Regression Applied to the Analysis of Congruence

In their seminal paper, Edwards and Parry (1993) presented the polynomial regression as a better alternative to applying difference score in the study of congruence. While this method is increasingly applied in congruence research, its complexity relative to other methods for assessing congruence (e.g., difference score methods) was one of the main claims against its use. The objective of this work is to gain additional insight into the use of polynomial regression in the area of social and behavioral sciences. First, we elaborate on the meaning and the inference based on the coefficients obtained by applying polynomial regression and explain the mathematical terms that are used to describe response surfaces. Then, we introduce additional inference methods and illustrate their application on a real life example from the area of supportive exchanges, using data collected by the Smithers Institute at Cornell University (supported by the National Institute on Alcohol Abuse and Alcoholism). Our work is aimed to provide a guide to researchers who apply polynomial regression in order to assess the effect of congruence between two constructs and enable better understanding and interpretation of the results obtained.     

Firefighters, critical incidents, and drinking to cope: the adequacy of unit-level performance resources as a source of vulnerability and protection

The authors investigated the moderating role of unit-level performance resources on the distress-mediated relationship between the intensity of involvement in workplace critical incidents and problematic drinking behavior (i.e., drinking to cope). Building on recent developments in hierarchical linear modeling, the authors tested a cross-level, moderated-mediation model using data from 1,481 firefighters in 144 companies. The findings indicate that (a) there is a significant, distress-mediated association between intensity of involvement in such incidents and drinking to cope, which varies by company (i.e., unit), and (b) the adequacy of unit-level performance resources explains much of this cross-unit variance and attenuates both individual-level mediation stages (i.e., intensity of involvement in critical incidents 3 distress, and distress 3 drinking to cope). Implications regarding the role of unit resources adequacy as a vulnerability factor in stressor-strain relations are discussed.     

Further Insight and Additional Inference Methods for Polynomial Regression Applied to the Analysis of Congruence

In their seminal paper, Edwards and Parry (1993) Edwards, J. R. and Parry, M. E. 1993. On the use of polynomial regression equations as an alternative to difference scores in organizational research. Academy of Management Journal, 36: 1577–1613. [Crossref], [Web of Science ®] , [Google Scholar] presented the polynomial regression as a better alternative to applying difference score in the study of congruence. Although this method is increasingly applied in congruence research, its complexity relative to other methods for assessing congruence (e.g., difference score methods) was one of the main claims against its use. The objective of this work is to gain additional insight into the use of polynomial regression in the area of social and behavioral sciences. First, we elaborate on the meaning and the inference based on the coefficients obtained by applying polynomial regression and explain the mathematical terms that are used to describe response surfaces. Then, we introduce additional inference methods and illustrate their application on a real-life example from the area of supportive exchanges using data collected by the Smithers Institute at Cornell University (supported by the National Institute on Alcohol Abuse and Alcoholism). Our work is aimed to provide a guide to researchers who apply polynomial regression in order to assess the effect of congruence between two constructs and enable better understanding and interpretation of the results obtained.