On Guessing: An Alternative Adjusted Positive Learning Estimator and Comparing Probability Misspecification With Monte Carlo Simulations

Practitioners in the sciences have used the “flow” of knowledge (post-test score minus pre-test score) to measure learning in the classroom for the past 50 years. Walstad and Wagner, and Smith and Wagner moved this practice forward by disaggregating the flow of knowledge and accounting for student guessing. These estimates are sensitive to misspecification of the probability of guessing correct. This work provides guidance to practitioners and researchers facing this problem. We introduce a transformed measure of true positive learning that under some knowable conditions performs better when students’ ability to guess correctly is misspecified and converges to Hake’s normalized learning gain estimator under certain conditions. We then use simulations to compare the accuracy of two estimation techniques under various violations of the assumptions of those techniques. Using recursive partitioning trees fitted to our simulation results, we provide the practitioner concrete guidance based on a set of yes/no questions.     

Judgments in a hurry: Time pressure affects how judges assess unfairly shared losses and unfairly shared gains

Human society has evolved to allow third parties—the court system, parents or other societal arbitrators—to punish norm violators and compensate victims. Few studies explore the effect of stress or time pressure on a third‐party judge. Under time pressure, people will likely show a more instinctual reaction or judgment style. We investigated third‐party punishment and compensation within the context of unfairly shared losses and gains in a dictator game under time pressure. Our results show that under no time pressure, participants were inclined to punish dictators who unfairly split windfall gains; however, participants chose to compensate victims more than punish the norm‐violating dictators in the context of unfairly shared losses. With added time pressure, third‐parties were disposed to inflict punishment upon the dictator in both the gain and loss contexts—punishment became the action of choice. Our results shed light on the way observed behavior and stress affect social cognition and decision making in the context of altruistic social interventions and the enforcement of social norms.     

Causal inferences with group based trajectory models

A central theme of research on human development and psychopathology is whether a therapeutic intervention or a turning-point event, such as a family break-up, alters the trajectory of the behavior under study. This paper lays out and applies a method for using observational longitudinal data to make more confident causal inferences about the impact of such events on developmental trajectories. The method draws upon two distinct lines of research: work on the use of finite mixture modeling to analyze developmental trajectories and work on propensity scores. The essence of the method is to use the posterior probabilities of trajectory group membership from a finite mixture modeling framework, to create balance on lagged outcomes and other covariates established prior to t for the purpose of inferring the impact of first-time treatment at t on the outcome of interest. The approach is demonstrated with an analysis of the impact of gang membership on violent delinquency based on data from a large longitudinal study conducted in Montreal. The research has been supported by the National Science Foundation (NSF) (SES-99113700) and the National Institute of Mental Health (RO1 MH65611-01A2). It also made heavy use of data collected with the support from Québec’s CQRS and FCAR funding agencies, Canada’s NHRDP and SSHRC funding agencies, and the Molson Foundation. We thank Stephen Fienberg, Susan Murphy, Paul Rosenbaum, the editor, Paul De Boeck, and two anonymous reviewers for their insightful suggestions.     

Flexible contrast gain control in the right hemisphere

This study investigates whether the right hemisphere has more flexible contrast gain control settings for the identification of spatial frequency. Right-handed participants identified 1 and 9 cycles per degree sinusoidal gratings presented either to the left visual field-right hemisphere (LVF-RH) or the right visual field-left hemisphere (RVF-LH). When luminance contrast was randomized across a wide range (20-60%), performance gradually improved with contrast in the LVF-RH. Conversely, performance in the RVF-LH was disrupted and saturated for 20 and 60% of contrast, respectively, leading to a LVF-RH advantage for these contrast levels. When contrast was blocked or randomized for a smaller range (30-50%), the LVF-RH advantage was diminished. Flexible contrast gain control is needed when contrast is randomized across a wide range, but not when it is blocked or randomized across a smaller range. The results therefore suggest that the right hemisphere is able to process spatial frequency information across a wider range of contrast levels than is the left hemisphere.     

Classical Test Theory as a first-order Item Response Theory: Application to true-score prediction from a possibly nonparallel test

We give an account of Classical Test Theory (CTT) in terms of the more fundamental ideas of Item Response Theory (IRT). This approach views classical test theory as a very general version of IRT, and the commonly used IRT models as detailed elaborations of CTT for special purposes. We then use this approach to CTT to derive some general results regarding the prediction of the true-score of a test from an observed score on that test as well from an observed score on a different test. This leads us to a new view of linking tests that were not developed to be linked to each other. In addition we propose true-score prediction analogues of the Dorans and Holland measures of the population sensitivity of test linking functions. We illustrate the accuracy of the first-order theory using simulated data from the Rasch model, and illustrate the effect of population differences using a set of real data.This research is collaborative in every respect and the order of authorship is alphabetical. It was begun when both authors were on the faculty of the Graduate School of Education at the University of California, Berkeley.We would like to thank both Neil Dorans, Skip Livingston and two anonymous referees for many suggestions that have greatly improved this paper.     

Non-semantic contributions to “semantic” redundancy gain

Recently, two groups of researchers have reported redundancy gains (enhanced performance with multiple, redundant targets) in tasks requiring semantic categorization. Here we report two experiments aimed at determining whether the gains found by one of these groups resulted from some form of semantic coactivation. We asked undergraduate psychology students to complete choice RT tasks requiring the semantic categorization of visually presented words, and compared performance with redundant targets from the same semantic category to performance with redundant targets from different semantic categories. If the redundancy gains resulted from the combination of information at a semantic level, they should have been greater in the former than the latter situation. However, our results showed no significant differences in redundancy gain (for latency and accuracy) between same-category and different-category conditions, despite gains appearing in both conditions. Thus, we suggest that redundancy gain in the semantic categorization task may result entirely from statistical facilitation or combination of information at non-semantic levels.     

Analyzing Complex Treatment Effects in Nonrandomized Observational Studies: The Case of Retention of Students in Grade

Should low-achieving students be promoted to the next grade or be retained (held back) in the prior grade? This special section presents a discussion of the application of marginal structural models to the challenging problem of estimating the effect of promotion versus retention in grade on math scores in elementary school. Vandecandelaere, De Fraine, Van Damme, and Vansteelandt provide a didactic presentation of the marginal structural modeling approach, noting retention is a time-varying treatment because promoted low-achieving students may be retained in a subsequent grade. Steiner, Park, and Kim's commentary presents a detailed analysis of the treatment effects being estimated in same-age versus same-grade comparisons from the perspective of the potential outcomes model. Reshetnyak, Cham, and Kim's commentary clarifies the conditions under which same-age versus same-grade comparisons might be preferred; they also identify methods of further improving the estimation of retention effects. In their rejoinder, Vandecandelaere and Vansteelandt discuss tradeoffs in comparing the promoted and retained groups and highlight sensitivity analysis as a method of probing the robustness of treatment effect estimates. Our hope is that this combined didactic presentation and critical evaluation will encourage researchers to add marginal structural models to their methodological toolkits.     

Effects of the use of percentage agreement on behavioral observation reliabilities: A reassessment

The percentage agreement index has been and continues to be a popular measure of interobserver reliability in applied behavior analysis and child development, as well as in other fields in which behavioral observation techniques are used. An algebraic method and a linear programming method were used to assess chance-corrected reliabilities for a sample of past observations in which the percentage agreement index was used. The results indicated that, had kappa been used instead of percentage agreement, between one-fourth and three-fourth of the reported observations could be judged as unreliable against a lenient criterion and between one-half and three-fourths could be judged as unreliable against a more stringent criterion. It is suggested that the continued use of the percentage agreement index has seriously undermined the reliabilities of past observations and can no longer be justified in future studies.