Analyzing the accuracy of probability judgments for multiple events: An extension of the covariance decomposition

The probability score (PS) can be used to measure the overall accuracy of probability judgments for a single event, e.g., “Rain falls,” or “This patient has cancer.” It has been shown previously how a “covariance decomposition” of the mean of PS over many occasions indexes several distinct aspects of judgment performance (J. F. Yates, Organizational Behavior and Human Performance, 30, 132–156 (1982)). There are many situations in which probability judgments are reported for sample space partitions containing more than one event and its complement, e.g., medical situations in which a patient might suffer from Disease X, Disease Y, or Disease Z, or testing situations in which the correct answer to an item might be any one of alternatives (a) through (e). The probability score for multiple events (PSM) serves as a measure of the overall accuracy of probability judgments for the events in partitions of any size. The present article describes and interprets an extension of the covariance decomposition to the mean of PSM. The decomposition is illustrated with data from two contexts, medicine and education.