Order Effects on Situational Judgment Test Items: A case of construct‐irrelevant difficulty

Situational judgment tests (SJTs) pose unique cognitive demands on test takers in that, when presented in written form, they require a great deal of reading and cognitive effort. Because of this cognitive demand, responses to test items toward the end of the test may be influenced by an order effect produced by responding to a large quantity of previous test items. This construct‐irrelevant order effect may increase measurement error and threaten the validity of SJT scores. To test this phenomenon, data were obtained from 1,089 applicants who had completed a lengthy SJT as part of a selection process for an hourly safety and surveillance job at a large international corporation. Results showed that local item dependence, item difficulty, and the rate of omitted responses all increased when items were placed toward the end. The order effect alone was not strong enough to influence subgroup mean score differences in the second half of the test. However, this effect did vary by race: African‐Americans were most strongly affected by the order effect, followed by Caucasians, in their number of omitted responses. Implications and future research of this effect for SJTs and similar types of assessments are discussed.