Assessing the validity of computer-based test interpretations: rating reliability and individual differences among raters

Three studies were conducted to assess the role of individual differences among raters participating in validation studies for computer-based test interpretations (CBTIs) and to assess the reliability of ratings. Studies centered on two CBTIs for the Sixteen Personality Factor Questionnaire (16PF). The first study involved 54 students who rated the accuracy of the Human Resource Development Report (HRDR) in an experimental context. The second study involved 73 students who rated the HRDR in a nonexperimental context. The third study involved 28 students who rated the Narrative Score Report (NSR) in a nonexperimental context. Results taken together indicate that (a) shrewdness may influence ratings of CBTI accuracy in an experimental context; (b) self-sufficiency, emotional stability, dominance, or abstract thought may influence ratings in a nonexperimental context, depending on the specific report involved: (c) test-retest reliabilities for accuracy ratings were .74 for HRDR and .81 for NSR; and (d) test-retest reliabilities for usefulness ratings were .75 for HRDR and .54 for NSR.