| Abstract: | In a previous study, Schmidt, Hunter, Croll and McKenzie (1983) demonstrated that estimates of the validity of cognitive tests made by highly trained and experienced judges are more accurate than empirical estimates obtained from small-sample validity studies. The present study examined whether less experienced judges could also produce accurate estimates. Twenty-eight recent Ph.D.'s in I/O Psychology estimated observed validities for the same 54 job-test combinations used by Schmidt et al. (1983). The estimates of these judges contained about twice as much random error as the experts' estimates. Systematic error of the less experienced judges was also greater than that of the experts (.0732 vs .019). The systematic errors of the two sets of judges were in opposite directions: less experienced judges overestimated validities, on average, while experts underestimated them. The results show that the estimates of less experienced judges contain less information than those of experts, but also that averages of estimates of several less experienced judges are as accurate as those obtained from small-sample empirical studies. |
