A signal detection framework for the evaluation of probabilistic forecasts

In this paper I formulate an approach for evaluating probabilistic forecasts in terms of signal detection theory. Signal detection theory provides a powerful perspective for this type of problem, and a rich empirical background including methodological tools as well as an extensive body of research in many domains. I propose procedures which emphasize the maximization of expected utility for the decision maker who uses the forecasts. Further, I suggest approaches to obtaining indices of calibration and resolution within this framework. I also present arguments that the proposed indices will exhibit the same basic properties as do decompositions of Brier's (1950, Monthly Weather Review, 78, 1–3) mean probability score. However, the properties may be reflected in different ways, and hence, the present methods may lead to different conclusions about forecasting ability. Finally, I argue that the use of an expected utility loss function makes this approach more appropriate for practical applications as well as for theoretical research than other procedures with more arbitrary loss functions.     

The effects of feedback consistency and feedback favorability on self-perceived task competence and perceived feedback accuracy

The present study, employing a 2 × 2 true-experimental design and regression analyses assessed the main and interactive effects of feedback consistency (consistent vs inconsistent), and feedback favorability (acceptable vs superior), for feedback given at two time periods on measures of perceived feedback accuracy and self-perceived task competence. Among the study's findings were that (a) the perceived accuracy of feedback at the second period was a function of the favorability of feedback received at the first time period, (b) the self-perceived task competence levels of subjects were jointly determined by the favorability of feedback received at both time periods, and (c) the perceived accuracy of feedback was a function of its consistency. Organizational implications of the study's results are offered.     

The retention of nouns,adjectives, and verbs of sentences

This research compared the retention of nouns, adjectives and verbs of sentences. A model of the structure underlying the retention of sentences and in which deep structure constituent propositions are hierarchically related, was used to make predictions concerning the retention of these word classes. The model was not supported by the data and it was determined that latencies to recognize the words of a previously presented sentence were the same for all classes of words. This result indicated that for recognition (but not necessarily for other memory tasks), each word of a sentence is seperately or equally accessible (as long as inter-word imagery is the same). However, a further experiment showed that, although word class is not a variable, the image-arousing capacity of a word is. Latencies and error rates were much lower for high-imagery words than for low-imagery ones. The findings on error rates in recognizing words were interesting and pointed to a different model where the meaning representations and syntatic functions of words have an active role in sentence retention.     

Predictability,controllability, and inoculation against learned helplessness

Nondepressed human subjects were divided into seven groups. On a series of discrimination problems, a helplessness group received insoluble problems, a solvable group received contingent feedback, and a no treatment control group received no feedback. For two other groups insoluble problems were preceded by success feedback on a different task presented according to a fixed ratio (FR) or variable ratio (VR) schedule of reinforcement. Two control groups received either FR or VR schedules of success but were not examined on the discrimination problems. All groups were tested for escape/avoidance performance on a human shuttle box. Both FR and VR schedules produced an inoculation against learned helplessness; escape performance by the helplessness group was significantly worse than that of FR and VR inoculation groups. These latter groups performed similar to the solvable and three control groups. Significantly worse than that of FR and VR inoculation groups. These latter groups performed similar to the solvable and three control groups. Significantly fewer subjects in the VR inoculation group exhibited avoidance responses than their counterparts in the FR inoculation group. despite similar escape performance. The findings indicate that learned helplessness can be prevented in humans and suggest different sources of interference produced by unpredictable and uncontrollable events.     

Individualized behavior therapy for alcoholics: a third year independent double-blind follow-up.

An independently conducted double-blind study providing third year follow-up data for 53 of 70 male alcoholics who served as subjects in an experiment evaluating “Individualized Behavior Therapy (IBT)” techniques is reported. In the original study, subjects were initially assigned to either a controlled drinking or non-drinking (abstinence) treatment goal, and were then randomly assigned to either an experimental group receiving IBT or a control group receiving conventional state hospital treatment oriented towards abstinence. Previously reported results showed that IBT subjects functioned significantly better than control subjects throughout the first year of follow-up. Second year follow-up results found that IBT subjects treated with a controlled drinking goal functioned significantly better than their respective control subjects on a variety of measures, including drinking behavior. Differences between IBT subjects treated with a non-drinking goal and their control subjects did not retain statistical significance during the second year of follow-up. The independently determined follow-up data showed that during the third post-treatment year, subjects in the experimental groups generally functioned better than their respective control subjects. Comparison of the controlled drinking experimental group with its control group showed the significantly better functioning of subjects in the experimental condition on a number of drinking and other life functioning measures. Comparison of the non-drinking groups indicated only one possible difference on the drinking related measures involving a trend for subjects in the experimental condition to abstain more than those in the non-drinker control group. On other life functioning measures, however, subjects in the non-drinking experimental condition showed consistent improvement over their respective control subjects. Both of the controlled drinking groups reported more controlled drinking days than the non-drinking groups throughout the third year follow-up period.     

THE BEHAVIOR OBSERVATION INSTRUMENT: A METHOD OF DIRECT OBSERVATION FOR PROGRAM EVALUATION1

The background and development of a multicategory direct observation system, the Behavior Observation Instrument (BOI), is described. This time-sampling procedure for recording the behavior of persons is demonstrated in several treatment settings and the results applied to issues of program evaluation. Elements that have prevented direct observation from being widely adopted, such as costs, manpower, and training requirements, are systematically analyzed. A basic psychometric analysis of the instrument is used to determine optimum frequency and duration of observation intervals as well as observer agreement. The results imply that direct observation methods, once assumed by some to belong to the special province of the single-subject design, can be used to assess the effects of programs on groups of psychiatric clients in an efficient and economic manner.     

Comparative judgments with numerical reference points

A model of subjective magnitude comparisons is explored, which assumes that subjects compare symbolic stimulus magnitudes with respect to a reference point. The reference point may be established implicitly by the question (e.g., “Which is larger?” vs “Which is smaller?”) or be presented explicitly (e.g., “Choose the stimulus closer to X.”). The model was tested in five experiments in which subjects judged which of two comparison digits was closer to (or further from) a reference digit. Regression analyses in three experiments revealed that reaction time depended on the ratio of the distances from the comparison items to the reference point. The other two experiments provided evidence that subjects can strategically vary the processes by which they compare stimuli to a reference point. The results indicated that subjects can perform various types of “analog arithmetic” using either the linear number scale or a nonlinear scale of subjective digit magnitude.