The sampling distributions of Gaussian ROC statistics

For a discrimination experiment, a plot of the hit rate against the false-alarm rate--the ROC curve--summarizes performance across a range of confidence levels. In many content areas, ROCs are well described by a normal-distribution model and the z-transformed hit and false-alarm rates are approximately linearly related. We examined the sampling distributions of three parameters of this model when applied to a ratings procedure: the area under the ROC (Az), the normalized difference between the means of the underlying signal and noise distributions (da), and the slope of the ROC on z-coordinates (s). Statistical bias (the degree to which the mean of the sampling distribution differed from the true value) was trivial for Az, small but noticeable for da, and substantial for s. Variability of the sampling distributions decreased with the number of trials and was also affected by the number of response categories available to the participant and by the overall sensitivity level. Figures in the article and tables available on line can be used to construct confidence intervals around ROC statistics and to test statistical hypotheses.