Identifying Misfitting Achievement Estimates in Performance Assessments: An Illustration Using Rasch and Mokken Scale Analyses

Researchers apply individual person fit analyses as a procedure for checking model-data fit for individual test-takers. When a test-taker misfits, it means that the inferences from their test score regarding what they know and can do may not be accurate. One problem in applying individual person fit procedures in practice is the question of how much misfit it takes to make the test score an untrustworthy estimate of achievement. In this paper, we argue that if a person’s responses generally follow a monotonic pattern, the resulting test score is “good enough” to be interpreted and used. We present an approach that applies statistical procedures from the Rasch and Mokken measurement perspectives to examine individual person fit based on this good enough criterion in real data from a performance assessment. We discuss how these perspectives may facilitate thinking about applying individual person fit procedures in practice.     

I’m lonely,can’t you tell? Convergent validity of self- and informant ratings of loneliness

To what degree do self-ratings of loneliness converge with informant ratings? In this study, we obtained self-ratings of loneliness from 463 young adults and informant ratings from their parents, friends, and romantic partners. Convergence among these ratings was estimated using structural equation models for multitrait-multimethod data and compared to self-informant convergence of life-satisfaction ratings. Self- and informant ratings were moderately correlated and comparable to self-informant correlations obtained for life satisfaction. Romantic partners were more accurate in their judgments than both friends and parents, who did not differ significantly from each other in terms of accuracy. Together, these findings indicate that informant ratings of loneliness can be used as valid indicators of loneliness in applied contexts and in future research.     

Effects of measurement aggregation on predicting externalizing problems from preschool behaviour

In long‐term studies of psychological development, the initial assessment of etiologically significant child behaviours is often carried out at a single point in time only. However, one‐time assessments of behaviour are likely to possess limited reliability, leading to attenuated longitudinal correlation coefficient magnitudes. How much this bias might have affected behavioural continuity estimates in longitudinal research is presently unknown. Using a data set from the Mauritius Child Health Project, we particularize the attenuating effects of single‐occasion behavioural assessments on consistency estimates of impulsive–aggressive behaviour over time. Specifically, two nursery teachers provided 15 consecutive weekly ratings of the aggressive behaviour of 99 four‐year‐old children. The same children were reassessed for the presence of externalizing behaviour problems at the ages of 8 and 10. There were substantial increases in both reliability and predictive correlation coefficient magnitudes when the preschool scores were aggregated across several weekly ratings. A further increase resulted after the two outcome assessments were combined into a composite score of school‐age externalizing symptoms. A generalized procedure, developed from the correction for attenuation formula, is introduced to describe the relation of aggregation to predictive validity in longitudinal research.