"Perhaps it would be better not to know everything."

The advent of statistical methods for evaluating the data of individual-subject designs invites a comparison of the usual research tactics of the group-design paradigm and the individual-subject-design paradigm. That comparison can hinge on the concept of assigning probabilities of Type 1 and Type 2 errors. Individual-subject designs are usually interpreted with implicit, very low probabilities of Type 1 errors, and correspondingly high probabilities of Type 1 errors, and correspondingly high probabilities of Type 2 errors. Group designs are usually interpreted with explicit, moderately low probabilities of Type 1 errors, and therefore with not such high probabilities of Type 2 errors as in the other paradigm. This difference may seem to be a minor one, considered in terms of centiles on a probability scale. However, when it is interpreted in terms of the substantive kinds of results likely to be produced by each paradigm, it appears that the individual-subject-design paradigm is more likely to contribute to the development of a technology of behavior, and it is suggested that this orientation should not be abandoned.     

Bayesian item selection criteria for adaptive testing

Owen (1975) proposed an approximate empirical Bayes procedure for item selection in computerized adaptive testing (CAT). The procedure replaces the true posterior by a normal approximation with closed-form expressions for its first two moments. This approximation was necessary to minimize the computational complexity involved in a fully Bayesian approach but is no longer necessary given the computational power currently available for adaptive testing. This paper suggests several item selection criteria for adaptive testing which are all based on the use of the true posterior. Some of the statistical properties of the ability estimator produced by these criteria are discussed and empirically characterized.Portions of this paper were presented at the 60th annual meeting of the Psychometric Society, Minneapolis, Minnesota, June, 1995. The author is indebted to Wim M. M. Tielen for his computational support.     

Predicting the Outcome of Treatment

The clinical question—“Which treatment(s) for which patients with what set of subgrouping characteristics working by what mechanism(s)?”—rests at the heart of differential therapeutics. Experimentally, this question reduces to a test of how well we can predict the outcome of treatment using the treatment conditions plus other moderating and mediating variables. Reflecting the discussions held at a recent National Institute of Mental Health (NIMH) conference on psychosocial treatments, and using pediatric anxiety disorders as a case in point, we discuss the problem of prediction in treatment outcome studies from the standpoint of definition of terms, using the general linear model of prediction. We also outline types of studies that may be useful in testing potential predictors, and put forward a possible matrix of predictor variables as currently implemented in an NIMH-funded treatment outcome study of pediatric obsessive-compulsive disorder (OCD). We conclude by making specific suggestions for implementing a broader approach to the study of predictors.     

Goodness of fit tests for paired comparison models

A goodness of fit test presented by Andersen is shown to be incorrect. The correct test is described and a re-analysis of Andersen's data is provided.     

A new Bayesian procedure for the estimation of the referees' decision thresholds and agreement. The case of professional handball refereeing

In double refereeing, agreement between referees is fundamental for the achievement of a flawless and smooth refereeing activity. Nonetheless, the agreeement can be affected by several external and internal factors, with a negative impact on the consistency of the refereeing and the fluidity of the game. The referee’s Decision Threshold (DT) is the limit above which the evidence of a foul results in the call of an infraction by the referee. It represents the individual refereeing style, and it is one of the factors that can contribute to low agreement between the referees. We present an application of a new Bayesian procedure to estimate referee’s DT in a typical refereeing task. To this end, 56 italian professional handball (FIGH) referees were asked to evaluate 96 potential foul plays, belonging to four different infraction types: 7-m throw, passive play, offensive foul, disciplinary sanctions. The proposed method provides information about: (i) referee’s individual DT; (ii) the agreement between pairs of referees (i.e., one-to-one); (iii) the agreement between each referee and all the other referees (i.e., one-to-total); (iv) the agreement between each referee and a reference referee (i.e., one-to-expert); (v) the proportion of agreement between the referees and a reference referee for each potential foul play. Sport federations would profit by this procedure in different ways: by using the information about the DT to train referees, by detecting referees with a low agreement with their colleagues or with the reference referee, by focusing training on specific plays to improve the arbitration class internal consistency.     

A Comparison of ML,WLSMV, and Bayesian Methods for Multilevel Structural Equation Models in Small Samples: A Simulation Study

Multilevel structural equation models are increasingly applied in psychological research. With increasing model complexity, estimation becomes computationally demanding, and small sample sizes pose further challenges on estimation methods relying on asymptotic theory. Recent developments of Bayesian estimation techniques may help to overcome the shortcomings of classical estimation techniques. The use of potentially inaccurate prior information may, however, have detrimental effects, especially in small samples. The present Monte Carlo simulation study compares the statistical performance of classical estimation techniques with Bayesian estimation using different prior specifications for a two-level SEM with either continuous or ordinal indicators. Using two software programs (Mplus and Stan), differential effects of between- and within-level sample sizes on estimation accuracy were investigated. Moreover, it was tested to which extent inaccurate priors may have detrimental effects on parameter estimates in categorical indicator models. For continuous indicators, Bayesian estimation did not show performance advantages over ML. For categorical indicators, Bayesian estimation outperformed WLSMV solely in case of strongly informative accurate priors. Weakly informative inaccurate priors did not deteriorate performance of the Bayesian approach, while strong informative inaccurate priors led to severely biased estimates even with large sample sizes. With diffuse priors, Stan yielded better results than Mplus in terms of parameter estimates.     

Spatial sequence memory and spatial error monitoring in the Groton Maze Learning Task (GMLT): A validation study of GMLT sub-measures in healthy children

The stepping-stone variant of the hidden pathway maze learning (HPML) task paradigm has been extensively used to investigate cognitive functions in neuropsychology and neuropharmacology. Previous studies have used total error across trials, as well as rule-break errors and learning errors, to define spatial memory and/or executive function in healthy and impaired adults and children. However, the construct validity of performance measures on HPML tasks has not been established in healthy children. To assess the construct validity of measures of exploratory and rule-break errors on the Groton Maze Learning Task (GMLT) measures of spatial sequence memory (Corsi Blocks Task) and spatial error monitoring (Continuous Paired Associate Learning; CPAL) were used. The results indicate that Corsi span predicted GMLT spatial sequence memory and CPAL accuracy predicted GMLT spatial error monitoring. The construct validity of the GMLT as a measure of spatial memory and executive function are discussed with regard to prior research using HPML tasks in neuropsychological contexts.     

Introversion,inhibition, and displayed anxiety: The role of error reactivity processes

Three studies involving 176 undergraduates examined the personality-related correlates of tendencies to slow down following errors in choice reaction time tasks. Such tendencies were hypothesized to tap individual differences in threat reactivity processes and therefore be relevant to the prediction of phobic-like fear (Study 1) and displayed anxiety as rated by informants (Studies 2 and 3). However, on the basis of the idea that high levels of extraversion may suppress threat reactivity processes, it was hypothesized that extraversion and post-error slowing tendencies would interact in predicting the dependent measures. The studies supported the latter interactive hypothesis in that post-error slowing tendencies were predictive of displayed anxiety at low, but not high, levels of extraversion. The discussion focuses on the respective roles of error-reactivity processes and extraversion in predicting behavioral inhibition and displayed anxiety.     

Visual awareness judgments are sensitive to accuracy feedback in stimulus discrimination tasks

In this study we tested the hypothesis that perceptual awareness judgments are sensitive to accuracy feedback about the previous action. We used a perceptual discrimination task in which participants reported their stimulus awareness. We created two conditions: No-feedback and Feedback (discrimination accuracy feedback was provided at the end of each trial). The results showed that visual awareness judgments are related to the accuracy of current and previous responses. Participants reported lower stimulus awareness for incorrectly versus correctly discriminated stimuli in both conditions; they also reported lower stimulus awareness in trials preceded by incorrect discrimination responses, compared to trials preceded by correct discrimination responses. This difference was significantly stronger in the Feedback condition, in which we also observed post-error slowing for PAS ratings. We discuss the relation between visual awareness and the effects of performance monitoring and interpret the results in the context of current theories of consciousness.