Measures of observer agreement when binomial data are collected in free operant situations

This paper examines a model and defines reasonable assumptions underlying different measures of observer agreement for categorical data collected in free operant situations. It is assumed that two or more observers classify operant behaviors of subjects into occurrences and nonoccurrences by recognition by validated response classes (categories) such that the rates of false positives and observer biases are acceptably low. Thus errors are mostly omissions, i.e., failing to observe events that occur. Four alternative cases are derived, together with formulas for calculating significance tests, variances, and standard errors, three of which do not depend on knowledge of the proportion of time points at which the event does not occur.We wish to acknowledge NICHD Grant HD-10570, The Neuropharmacology of Developmental Disorders, George Breese, Ph.D., and C. T. Gualtieri, M.D., Principal Investigators; NIEHS Grant ES-01104; USPHS Grant HD-03110; and MCH Project 916 to the Division for Disorders of Development and Learning.     

Statistics as psychometrics

In this paper, modern statistics is considered as a branch of psychometrics and the question of how the central problems of statistics can be resolved using psychometric methods is investigated. Theories and methods developed in the fields of test theory, scaling, and factor analysis are related to the principle problems of modern statistical theory and method. Topics surveyed include assessment of probabilities, assessment of utilities, assessment of exchangeability, preposterior analysis, adversary analysis, multiple comparisons, the selection of predictor variables, and full-rank ANOVA. Reference is made to some literature from the field of cognitive psychology to indicate some of the difficulties encountered in probability and utility assessment. Some methods for resolving these difficulties using the Computer-Assisted Data Analysis (CADA) Monitor are described, as is some recent experimental work on utility assessment.1980 Psychometric Society presidential address.I am indebted to Paul Slovic and David Libby for valuable consultation on the issues discussed in this paper and to Nancy Turner and Laura Novick for assistance in preparation.Research reported herein was supported under contract number N00014-77-C-0428 from the Office of Naval Research to The University of Iowa, Melvin R. Novick, principal investigator. Opinions expressed herein reflect those of the author and not those of sponsoring agencies.     

Binary programming and test design

An algorithmic approach to test design, using information functions, is presented. The approach uses a special branch of linear programming, i.e. binary programming. In addition, results of some benchmark problems are presented. Within the same framework, it is also possible to formulate the problem of individualized testing.I would like to thank my colleagues N. Veldhuijzen, H. Verstralen and M. Zwarts for their suggestions and comments. Furthermore, I would like to thank Professor W. van der Linden, Department of Educational Measurement and Data Analysis, Technological University Twente, for offering facilities at his department; Ellen Timminga of the same department and S. Baas, department of Operational Research at the same University for their efforts in linear programming.     

On more powerful tests of judge agreement with a known standard

To establish the existence of his abilities, a judge is given the task of classifying each ofN=rs subjects into one ofr known categories, each containings of the subjects. An incomplete design is proposed whereby the judge is presented withb groups, each one containingn=rs/b<r subjects. Then different categories corresponding to members of the group are known. Using the total number of correct classifications, this method of grouping is compared to that in which the group size is equal to the number of categories. The incomplete grouping is shown to yield a more powerful test for discriminating between the null hypothesis that the judge is guessing the classifications and the alternative hypothesis that he has some definite abilities. The incomplete design is found to be most effective (powerful) when the number of subjects in a group is limited to two or three.The author is grateful for the suggestions of the referees and the editor, which greatly improved the paper.     

Psychometric properties and reference data for Danish versions of Free and Cued Selective Reminding Test,Category Cued Memory Test and Logical Memory

Memory assessment is a key element in neuropsychological testing. Gold standard evaluation is based on updated normative data, but in many small countries (e.g. in Scandinavia) such data are sparse. In Denmark, reference data exist for non‐verbal memory tests and list‐learning tests but there is no normative data for memory tests which capture narrative recall and cued recall. In a nation‐wide study, Free and Cued Selective Reminding Test (FCSRT ), WMS ‐III Logical Memory (LM ) and a newly developed test Category Cued Memory Test (CCMT ‐48) were applied in 131 cognitively intact persons (aged 60–96 years). Regression‐based reference data for Danish versions of FCSRT , CCMT ‐48 and LM adjusted for age, education and gender are provided. Gender and age‐group had a significant impact on the expected scores, whereas the effect of education had a limited effect on expected scores. Test performances were significantly correlated in the range 0.21–0.51. Based on these findings and previous results it may be relevant to assess both free recall, cued recall and recognition to tap the earliest changes associated with neurodegeneration, and this study therefore provides an important supplement to existing Danish normative data. Future studies should investigate the discriminative validity of the tests and the clinical utility of the presented reference data.     

Testing Process Factor Analysis Models Using the Parametric Bootstrap

Process factor analysis (PFA) is a latent variable model for intensive longitudinal data. It combines P-technique factor analysis and time series analysis. The goodness-of-fit test in PFA is currently unavailable. In the paper, we propose a parametric bootstrap method for assessing model fit in PFA. We illustrate the test with an empirical data set in which 22 participants rated their effects everyday over a period of 90 days. We also explore Type I error and power of the parametric bootstrap test with simulated data.     

Misspecification in Latent Change Score Models: Consequences for Parameter Estimation,Model Evaluation,and Predicting Change

Latent change score models (LCS) are conceptually powerful tools for analyzing longitudinal data (McArdle & Hamagami, 2001). However, applications of these models typically include constraints on key parameters over time. Although practically useful, strict invariance over time in these parameters is unlikely in real data. This study investigates the robustness of LCS when invariance over time is incorrectly imposed on key change-related parameters. Monte Carlo simulation methods were used to explore the impact of misspecification on parameter estimation, predicted trajectories of change, and model fit in the dual change score model, the foundational LCS. When constraints were incorrectly applied, several parameters, most notably the slope (i.e., constant change) factor mean and autoproportion coefficient, were severely and consistently biased, as were regression paths to the slope factor when external predictors of change were included. Standard fit indices indicated that the misspecified models fit well, partly because mean level trajectories over time were accurately captured. Loosening constraint improved the accuracy of parameter estimates, but estimates were more unstable, and models frequently failed to converge. Results suggest that potentially common sources of misspecification in LCS can produce distorted impressions of developmental processes, and that identifying and rectifying the situation is a challenge.     

Accounting for Missing Correlation Coefficients in Fixed-Effects MASEM

Meta-analytic structural equation modeling (MASEM) is increasingly applied to advance theories by synthesizing existing findings. MASEM essentially consists of two stages. In Stage 1, a pooled correlation matrix is estimated based on the reported correlation coefficients in the individual studies. In Stage 2, a structural model (such as a path model) is fitted to explain the pooled correlations. Frequently, the individual studies do not provide all the correlation coefficients between the research variables. In this study, we modify the currently optimal MASEM-method to deal with missing correlation coefficients, and compare its performance with existing methods. This study is the first to evaluate the performance of fixed-effects MASEM methods under different levels of missing correlation coefficients. We found that the often used univariate methods performed very poorly, while the multivariate methods performed well overall.     

Rat choice in rapidly changing concurrent schedules

In two experiments, experimentally naïve rats were trained in concurrent variable‐interval schedules in which the reinforcer ratios changed daily according to a pseudorandom binary sequence. In Experiment 1, relative response rates showed clear sensitivity to current‐session reinforcer ratios, but not to previous sessions' reinforcer ratios. Within sessions, sensitivity to the current session's reinforcement rates increased steadily, and by session end, response ratios approached matching to the current‐session reinforcer ratios. Across sessions, sensitivity to the current session's reinforcer ratio decreased with continued exposure to the pseudorandom binary sequence, contrary to expectations based on previous studies demonstrating learning sets. Using a second group of naïve rats, Experiment 2 replicated the main results from Experiment 1 and showed that although there were increases over sessions in both changeover rate and response rate during the changeover delay, neither could explain the accompanying reductions in sensitivity. We consider the role of reinforcement history, showing that our results can be simulated using two separate representations, one local and one nonlocal, but a more complex approach will be needed to bring together these results and other history effects such as learning sets and spontaneous recovery.