THEORY TESTING: COMBINING PSYCHOMETRIC META-ANALYSIS AND STRUCTURAL EQUATIONS MODELING

This paper presents an overview of a useful approach for theory testing in the social sciences that combines the principles of psychometric meta-analysis and structural equations modeling. In this approach to theory testing, the estimated true score correlations between the constructs of interest are established through the application of meta-analysis (Hunter & Schmidt, 1990), and structural equations modeling is then applied to the matrix of estimated true score correlations. The potential advantages and limitations of this approach are presented. The approach enables researchers to test complex theories involving several constructs that cannot all be measured in a single study. Decision points are identified, the options available to a researcher are enumerated, and the potential problems as well as the prospects of each are discussed.     

IN SUPPORT OF PERSONALITY ASSESSMENT IN ORGANIZATIONAL SETTINGS

Personality constructs have been demonstrated to be useful for explaining and predicting attitudes, behaviors, performance, and outcomes in organizational settings. Many professionally developed measures of personality constructs display useful levels of criterion-related validity for job performance and its facets. In this response to Morgeson et al. (2007) , we comprehensively summarize previously published meta-analyses on (a) the optimal and unit-weighted multiple correlations between the Big Five personality dimensions and behaviors in organizations, including job performance; (b) generalizable bivariate relationships of Conscientiousness and its facets (e.g., achievement orientation, dependability, cautiousness) with job performance constructs; (c) the validity of compound personality measures; and (d) the incremental validity of personality measures over cognitive ability. Hundreds of primary studies and dozens of meta-analyses conducted and published since the mid 1980s indicate strong support for using personality measures in staffing decisions. Moreover, there is little evidence that response distortion among job applicants ruins the psychometric properties, including criterion-related validity, of personality measures. We also provide a brief evaluation of the merits of alternatives that have been offered in place of traditional self-report personality measures for organizational decision making. Given the cumulative data, writing off the whole domain of individual differences in personality or all self-report measures of personality from personnel selection and organizational decision making is counterproductive for the science and practice of I-O psychology.     

RELIABILITY IS NOT VALIDITY AND VALIDITY IS NOT RELIABILITY

Interrater correlations do provide an index of reliability of job performance ratings. We show that the arguments presented by Murphy and DeShon (2000) lead to the radical conclusion that traditional measurement models–both classical theory and generalizability theory models–can be used neither with job performance ratings nor with other measures used in I-O and other areas of psychology and the social sciences. We show that this untenable conclusion is based on confusion of validity issues and questions with reliability issues and questions. It is also based on the incorrect belief that classical measurement models are capable of addressing only random response measurement error and cannot address other forms of measurement error. We also show that the solution Murphy and DeShon offer to the problem of measurement error in ratings, as they define this problem, cannot work. Properly understood, the position taken by Murphy and DeShon leaves us with the nihilistic conclusion that no appropriate measurement models are possible in psychological research, thus making meaningful research impossible.     

SATISFACTION, CITIZENSHIP BEHAVIORS, AND PERFORMANCE IN WORK UNITS: A META-ANALYSIS OF COLLECTIVE CONSTRUCT RELATIONS

This paper offers theoretical development clarifying the structure and function of collective job satisfaction and uses meta-analytic methods ( k  = 73) to examine the satisfaction–performance relationship when both constructs are construed at the work unit level. Overall, our results suggest that the relationship between unit-level job satisfaction and unit-level performance is significant (ρ= .34). Specifically, significant relationships were found between unit-level job satisfaction and unit-level criteria, including productivity, customer satisfaction, withdrawal, and organizational citizenship behaviors (OCB). Furthermore, the satisfaction-performance relationship was moderated by the strength of unit consensus, performance criteria, industry type, and whether the sample was U.S. based. Although these moderators were identified, collective satisfaction positively predicted performance across all levels of moderators. In addition, results indicate that unit-level OCB has a moderately strong relationship with unit-level performance. Only limited support was found for the notion that OCB is a route through which satisfaction has an impact on performance. We elaborate on these findings and attempt to provide a more clear direction for future research in this area.     

HOW DEFINITIVE ARE CONCLUSIONS BASED ON SURVEY DATA: ESTIMATING ROBUSTNESS TO NONRESPONSE

The problem of nonresponse poses a threat to the external validity of survey conclusions. In this article, procedures are developed to estimate the robustness of survey conclusions to nonresponse by drawing an analogy between studies unavailable for meta-analysis and nonrespondents in a survey. We extended the procedure to assess the effects of systematic differences between respondents and nonrespondents on t- and F -values used to test group differences. Steps to construct robustness tables that are independent of sample size, survey length, and scale width are outlined to determine how definitive conclusions are from survey data.