Reliability (Generalizability) of Job Performance Measurements: Navy Machinist Mates

The Armed Services have undertaken a Joint Services Project to develop prototype methods for measuring job performance and, if feasible, to link enlistment standards to on-the-job performance. As one step toward meeting the first purpose of the project, our study assessed the reliability of scores on four measures of job performance of machinist mates in the Navy: a hands-on performance test, a paper-and-pencil job knowledge test, job task perform- ance ratings, and global ratings. Generalizability theory was used to estimate the reliability of the measures and to improve the design of each. The generalizability analyses investigated the consistency of performance scores over observers, tasks (equipment operation, casualty control, preventative maintenance), and types of raters (supervisor, peer, and self). Recommenda- tions about the optimal designs of studies for decision making (e.g., pre- dicting hands-on performance scores from the other measures of job perform- ance) were based on the findings.