Evidence and inference in educational assessment

Educational assessment concerns inference about students' knowledge, skills, and accomplishments. Because data are never so comprehensive and unequivocal as to ensure certitude, test theory evolved in part to address questions of weight, coverage, and import of data. The resulting concepts and techniques can be viewed as applications of more general principles for inference in the presence of uncertainty. Issues of evidence and inference in educational assessment are discussed from this perspective.Probability isn't really about numbers; it's about the structure of reasoning.Presidential address to the Psychometric Society, presented June 25, 1994, in Champaign, Illinois.Supported by (1) Contract No. N00014-91-J-4101, R&T 4421573-01, from the Cognitive Science Program, Cognitive and Neural Sciences Division, Office of Naval Research, (2) the National Center for Research on Evaluation, Standards, Student Testing (CRESST), Educational Research and Development Program, cooperative agreement number R117G10027 and CFDA catalog number 84.117G, as administered by the Office of Educational Research and Improvement, U.S. Department of Education, and (3) the Statistical and Psychometric Research Division of Educational Testing Service. I am grateful for comments and suggestions from Henry Braun, Drew Gitomer, Richard Patz, Jonathan Troper, and Howard Wainer.     

Hypothetical and real consequences in experimental auctions for insurance against low-probability risks

Laboratory experiments have provided useful information about how people behave in risky situations. In particular, such experiments have allowed for the observation of many deviations from the predictions of expected utility theory. There is some dispute, however, over the effect of the use of real versus hypothetical money in these experiments and over the impact of experience. In this paper we describe auction experiments in which subjects faced a low-probability risk with either hypothetical or real-money consequences and varying amounts of experience. Using hypothetical payoffs resulted in behavior that provides a reasonable qualitative prediction of behavior with real consequences, but that seems to indicate a less diligent altitude toward spending the money as well as less concern about the risk.     

More Triviality

This paper uses the framework of Popper and Miller's work on axiom systems for conditional probabilities to explore Adams' thesis concerning the probabilities of conditionals. It is shown that even very weak axiom systems have only a very restricted set of models satisfying a natural generalisation of Adams' thesis, thereby casting severe doubt on the possibility of developing a non-Boolean semantics for conditionals consistent with it.     

Partitions and Conditionals

The literature on conditionals is rife with alternate formulations of the abstract semantics of conditional logic. Each formulation has its own advantages in terms of applications and generalizations; nevertheless, they are for the most part equivalent, in the sense that they underwrite the same range of logical systems. The purpose of the present note is to bring under this umbrella the partition semantics introduced by Brian Skyrms in (Skyrms, 1984).     

Preferences for and against stimuli paired with food.

Pigeons were presented with a concurrent-chains schedule in which terminal-link entries were assigned to two response keys on a percentage basis. The terminal links were fixed delays that sometimes ended with food and sometimes did not. In most conditions, 80% of the terminal links were assigned to one key, but a smaller percentage of the terminal links ended with food for this key, so the number of food reinforcers delivered by the two alternatives was equal. When the same terminal-link stimuli (orange houselights) were used for both alternatives, the pigeons showed a preference for whichever alternative delivered more frequent terminal links. When different terminal-link stimuli (green vs. red houselights) were used for the two alternatives, the pigeons showed a preference for whichever alternative delivered fewer terminal links when terminal-link durations were long, and no systematic preferences when terminal-link durations were short. This pattern of results was consistent with the predictions of Grace's (1994) contextual choice model. Preference for the alternative that delivered more frequent terminal links was usually stronger in the first few sessions of a condition than at the end of a condition, suggesting that the conditioned reinforcing effect of the additional terminal-link presentation was, in part, transitory.     

Negotiating the Boundaries of Science and Religion: The Case of Henry Margenau

The life of Henry Margenau (1901–1997) offers a case study in the complexity of the science-religion relation. As a physicist-philosopher at Yale University, he pursued a public program of "amalgamating religion with science." He drew upon his authority as a physicist and a tradition of philosophical idealism to advocate a "reciprocity" between the two spheres. He argued that a "new modesty" and "metaphysical attitude" among scientists created new opportunities for collaboration. At the same time, his view of faith and his sense of the religiousness of science created troubling ambiguities. In the end, Margenau embodied the ambivalent relation between science and religion while revealing the limits of renegotiating the boundaries.     

Reinforcement probability and ordinal position of response in fixed-interval schedules

Four rats pressed levers and received food pellets under fixed-interval reinforcement schedules of 20, 60, and 180 seconds. The number of responses in each interval was recorded. From these data, the probability of reinforcement was determined as a function of response count. These functions were generally increasing. This finding is consistent with previous suggestions that increasing response rates within fixed intervals may be a function of response count in addition to or instead of elapsed or remaining time.     

Avoidance based on shock intensity reduction with no change in shock probability

Rats were trained on a free-operant avoidance procedure in which shock intensity was controlled by interresponse time. Shocks were random at a density of about 10 shocks per minute. Shock probability was response independent. As long as interresponse times remained less than the limit in effect, any shocks received were at the lower of two intensities (0.75 mA). Whenever interresponse times exceeded the limit, any shocks received were at the higher intensity (1.6 mA). The initial limit of 15 seconds was decreased in 3-second steps to either 6 or 3 seconds. All animals lever pressed to avoid higher intensity shock. As the interresponse time limit was reduced, the response rate during the lower intensity shock and the proportion of brief interresponse times increased. Substantial warmup effects were evident, particularly at the shorter interresponse-time limits. Shock intensity reduction without change in shock probability was effective in the acquisition and maintenance of avoidance responding, as well as in differentiation of interresponse times. This research suggests limitations on the generality of a safety signal interpretation of avoidance conditioning.     

Assessing the Predictive Accuracy of hMLH1 and hMSH2 Mutation Probability Models

Hereditary nonpolyposis colorectal cancer (HNPCC) is characterized by a susceptibility to colorectal and extra-colonic cancers. Several guidelines exist for the identification of families suspected of having HNPCC, however these guidelines lack adequate sensitivity and specificity. In an attempt to improve accuracy for the detection of individuals with HNPCC, the Wijnen pre-test probability model (1998) and Myriad Genetics Laboratory prevalence table (2004) were developed. Here we evaluate the Wijnen model and Myriad table at predicting the presence of a mutation in individuals undergoing genetic testing for HNPCC. Forty-nine patients who had undergone genetic testing for germline mutations in hMLH1 and/or hMSH2 were part of our analysis. Our results revealed that the revised Bethesda guidelines performed with the highest sensitivity for germline mutations (94.4%), however the specificity was low (12.9%). Using a 10.0% mutation probability threshold, the Wijnen model and Myriad table had sensitivities of 55.6 and 60.0%, respectively and specificities of 54.8 and 23.8%, respectively. The Wijnen model and Myriad table were poor predictors of mutation prevalence, which is shown by the areas underneath their corresponding receiver operator characteristic curves (0.616 and 0.400, respectively). The results of this study demonsrate that neither the Wijnen model nor the Myriad table are sensitive or specific enough to be used as the only indication when to offer genetic testing for HNPCC.     

Critical appraisal of the discussion on delay discounting by Bailey et al. and Stein et al.: A scientific proposal for a reinforcer pathology theory 3.0

Delay discounting is one of the most studied phenomena in Psychology in the last decades. Recently, it has been proposed as a transdiagnostic variable accounting for several psychopathological problems. A review of current practices and clinical application (Bailey et al., 2021) and a response article (Stein et al., 2022) have been recently published. Despite both of them being stimulating, the articles raise issues needing further discussion. The first part of the present article critically appraises both publications by highlighting their strengths/limitations and offers an alternate perspective that clarifies some theoretical issues and allows new lines in research. In the second part, one of these new lines of research is presented by describing a research proposal based on the analysed critiques and current developments in both psychopathology and addiction research. This article contributes to the discussion on delay discounting suggesting alternative interpretations and future studies to improve theoretical models.