THE EFFECTS OF UNREPRESENTED STUDIES ON THE ROBUSTNESS OF VALIDITY GENERALIZATION RESULTS

Researchers conducting meta-analyses such as validity generalization can never be certain that their review contains all studies relevant to the research domain. Indeed, several authors in the past have noted ways in which research reviews may be systematically biased. A few techniques have emerged for addressing the issue of "missing studies" including the use of Rosenthal's (1979) file-drawer equation. Noting that Rosenthal's technique is inappropriate when applied to validity generalization findings, this paper develops a new method for assessing the vulnerability of validity generalization results to unrepresented or missing studies. The results of this new procedure are compared to the results of file-drawer analyses for 103 findings from validity generalization studies. We illustrate that this procedure more appropriately estimates the robustness of validity generalization results.