Calibration processes in a serial talent test

ObjectivesIn the beginning of serial evaluations, raters assess performances without knowledge about following performances. We assume that judges must observe a certain number of performances to calibrate their judgment scale, leading to systematic biases (avoidance of extreme judgments) in the beginning of judgment series. The present experiment investigates how many performance observations are necessary to calibrate internal judgment scales, leading to consistent judgments.DesignA between-group design was used.MethodVideos of a talent-test were presented in different orders. Every performance was presented in early, medium, and late positions. Thirty participants rated these series and we assessed the effect of performance position on performance evaluations.ResultsWe found calibration biases within the first nine performances; after that, evaluations remained stable and consistent.ConclusionCalibration processes are completed after a specific number of judgments; this implies interesting interventions for consistent judgments independent of the position in a series.