Examiner error in curriculum-based measurement of oral reading

Although curriculum based measures of oral reading (CBM-R) have strong technical adequacy, there is still a reason to believe that student performance may be influenced by factors of the testing situation, such as errors examiners make in administering and scoring the test. This study examined the construct-irrelevant variance introduced by examiners using a cross-classified multilevel model. We sought to determine the extent of variance in student CBM-R scores attributable to examiners and, if present, the extent to which it was moderated by students' grade level and English learner (EL) status. Fit indices indicated that a cross-classified random effects model (CCREM) best fits the data with measures nested within students, students nested within schools, and examiners crossing schools. Intraclass correlations of the CCREM revealed that roughly 16% of the variance in student CBM-R scores was associated between examiners. The remaining variance was associated with the measurement level, 3.59%; between students, 75.23%; and between schools, 5.21%. Results were moderated by grade level but not by EL status. The discussion addresses the implications of this error for low-stakes and high-stakes decisions about students, teacher evaluation systems, and hypothesis testing in reading intervention research.     

研究生入学考试写作评分的概化理论研究与多面Rasch分析

为了评价和改进硕士研究生入学考试一般能力测试的写作评分,研究者采用概化理论和多面Rasch分析对113位考生的写作样本的评分误差来源、评分信度等进行了探讨.概化理论研究显示,评分者和题目对评分准确性影响不大,以两道写作题的考试设计而言,评分者为2人即可保证评分信度在0.75以上.多面Rasch分析显示,评分者宽严度的估计值及其误差均在可接受的范围内,评分者之间在宽严度上不存在显著差异,且评分者自身在评分时总体上比较稳定.但个别评分者在特定考生特定题目上表现出特殊偏向.概化理论和多面Rasch分析丰富了写作评分研究的量化指标,证实了硕士研究生入学考试一般能力测试的写作评分具有较高的信度.     

Exploring Differential Subgroup Functioning on SAT Writing Items: What Happens When English Is Not a Test Taker's Best Language?

The purpose of this study is to explore patterns in model-data fit related to subgroups of test takers from a large-scale writing assessment. Using data from the SAT, a calibration group was randomly selected to represent test takers who reported that English was their best language from the total population of test takers (N = 322,011). A reference scale for the items was constructed based on EBL responses. Response behaviors of test takers who reported that English was not their best language (ENBL) were examined in relationship to this reference scale. This study illustrates the use of differential subgroup analyses to identify patterns related to person misfit within subgroups, as well as subsets of items, that may affect the validity of writing scores for ENBL test takers. The methodology described here offers an approach that can be used to explore, understand, and improve the validity of scores obtained from ENBL test takers in large-scale writing assessments.     

Validation and Standardization of the Dutch Trauma Symptom Checklist for Young Children in a Normative and Clinical Sample

The study aimed to describe the psychometric properties of the Dutch version of the Trauma Symptom Checklist for Young Children (TSCYC) in normative and clinical populations in the Netherlands. Caregivers’ ratings on the TSCYC were obtained for 1,802 children from the normal population, and for 515 children from a clinical population of traumatized children. In the clinical sample, additional measures were taken. The internal consistency and test-retest reliability of TSCYC scales were adequate. Confirmatory factor analysis showed acceptable fit on the putative scale structure. Regarding criterion validity, the clinical sample scored significantly higher on all clinical scales when compared to the normal population sample. Within the clinical sample, significant associations were found between TSCYC scales and convergent scales of other instruments. The Posttraumatic Stress-Total subscale demonstrated excellent discriminative ability between traumatized children and children from the normal population. The Dutch version of the TSCYC proved a valid and reliable instrument to measure trauma symptoms in young children through caregiver report, similar to the original American version. Further comparisons with diagnostic interviews are warranted.     

An Evaluation of the Word Triad Method for Monitoring Spelling Progress

Although spelling skill progress has typically been studied within the context of students' responses to written story starters (Deno, Marsten, & Mirkin, 1982; Fuchs & Fuchs, 2011; Hosp, Hosp, & Howell, 2007; Shinn & Shinn, 2002), there has been little research conducted within a curriculum-based measurement framework that has studied spelling progress monitoring on a weekly basis throughout an entire school year. The authors evaluated spelling progress using word triads, defined as a group of 3 words (with the same number of letters) that have a similar phonetic or morphological structure. Successful spelling of word triads as the unit of analysis has the potential to be more useful diagnostically to teachers than successful spelling of single words as the unit of analysis. An alternating series of three 15-word spelling tests were administered weekly to 10 third- and fourth-grade classrooms in the same school district throughout the school year for 8 rounds (1 round = 3 weeks of different but phonetically and/or morphologically similar word lists). Results indicate steady progress throughout the school year in the correct spelling of word triads (despite teacher reporting of no explicit spelling instruction in classrooms using in the words employed in the study). Correlations between the number of correctly spelled word triads with standard scores from a group-administered communication arts test were significant and comparable to alternate spelling test scoring methods. The authors conclude with limitations of the study and implications for practitioners.     

The Multigroup Ethnic Identity Measure – Revised (MEIM-R): Psychometric evaluation with adolescents from diverse ethnocultural groups in Italy

The Multigroup Ethnic Identity Measure – Revised (MEIM-R) is an extensively used questionnaire assessing ethnic identity. However, studies on its measurement characteristics in the European context are lacking. The current study addressed this gap by investigating the MEIM-R psychometric proprieties across multiple ethnocultural groups in Italy. Participants were 1445 adolescents (13–18 years) of Italian, East European, and North African origin. Results showed that the MEIM-R has good internal consistency. Multigroup confirmatory factor analyses revealed configural and metric invariance, i.e., an equal, correlated two-factor structure (ethnic identity exploration and commitment) and equal factor loadings across groups. Scalar invariance, i.e., equal item intercepts, was found only for the commitment scores that showed no group differences in latent factor mean levels. Partial structural invariance was evidenced, with the factor covariances varying across groups. These findings suggest that the MEIM-R is a valuable tool to assess the correlates of ethnic identity, although further research is needed.     

Children's Sensitivity to Cost and Reward in Decision Making across Distinct Domains of Probability,Effort, and Delay

Many behavioral paradigms used to study individuals' decision‐making tendencies do not capture the decision components that contribute to behavioral outcomes, such as differentiating decisions driven toward a reward from decisions driven away from a cost. This study tested a novel decision‐making task in a sample of 403 children (age 9 years) enrolled in an ongoing longitudinal study. The task consisted of three blocks representing distinct cost domains (delay, probability, and effort), wherein children were presented with a deck of cards, each of which consisted of a reward and a cost. Children elected whether to accept or skip the card at each trial. Reward–cost pairs were selected by using an adaptive algorithm to strategically sample the decision space in the fewest number of trials. Using person‐specific regression models, decision preferences were quantified for each cost domain with respect to general tolerance (intercept), as well as parameters estimating the effect of incremental increases in reward or cost on the probability of accepting a card. Results support the relative independence of decision‐making tendencies across cost domains, with moderate correlations observed between tolerance for delay and effort. Specific decision parameters showed unique associations with cognitive and behavioral measures including executive function, academic motivation, anxiety, and hyperactivity. Evidence indicates that sensitivity to reward is an important factor in incentivizing decisions to work harder or wait longer. Dissociating the relative contributions of reward and cost sensitivity in multiple domains may facilitate the identification of heterogeneity in suboptimal decision making. Copyright © 2017 John Wiley & Sons, Ltd.     

Integrating Covariates into Social Relations Models: A Plausible Values Approach for Handling Measurement Error in Perceiver and Target Effects

The Social Relations Model (SRM) is a conceptual and analytical approach to examining dyadic behaviors and interpersonal perceptions within groups. In an SRM, the perceiver effect describes a person's tendency to perceive other group members in a certain way, whereas the target effect measures the tendency to be perceived by others in certain ways. In SRM research, it is often of interest to relate these individual SRM effects to covariates. However, the estimated individual SRM effects might not provide a very reliable measure of the true, unobserved SRM effects, resulting in distorted estimates of associations with other variables. This article introduces a plausible values approach that allows users to correct for measurement error when assessing the association of individual SRM effects with other individual difference variables. In the plausible values approach, the latent, true individual SRM effects are treated as missing values and are imputed from an imputation model by applying Bayesian estimation techniques. In a simulation study, the statistical properties of the plausible values approach are compared with two approaches that have been used in previous research. A data example from educational psychology is presented to illustrate how the plausible values approach can be implemented with the software WinBUGS.