Examiner error in curriculum-based measurement of oral reading

Although curriculum based measures of oral reading (CBM-R) have strong technical adequacy, there is still a reason to believe that student performance may be influenced by factors of the testing situation, such as errors examiners make in administering and scoring the test. This study examined the construct-irrelevant variance introduced by examiners using a cross-classified multilevel model. We sought to determine the extent of variance in student CBM-R scores attributable to examiners and, if present, the extent to which it was moderated by students' grade level and English learner (EL) status. Fit indices indicated that a cross-classified random effects model (CCREM) best fits the data with measures nested within students, students nested within schools, and examiners crossing schools. Intraclass correlations of the CCREM revealed that roughly 16% of the variance in student CBM-R scores was associated between examiners. The remaining variance was associated with the measurement level, 3.59%; between students, 75.23%; and between schools, 5.21%. Results were moderated by grade level but not by EL status. The discussion addresses the implications of this error for low-stakes and high-stakes decisions about students, teacher evaluation systems, and hypothesis testing in reading intervention research.     

The utility and accuracy of oral reading fluency score types in predicting reading comprehension

This study used data from the Dynamic Indicators of Basic Early Literacy Skills (DIBELS; Good & Kaminski, 2002) oral reading fluency (ORF) probes to examine variation among different ORF score types (i.e., the median of three passages, the mean of all three passages, the mean of passages 2 and 3, and the score from passage 3) in predicting reading comprehension as a function of student reading fluency level and to compare the screening accuracy of these score types in predicting student reading comprehension. The results revealed that the relation between oral reading fluency and reading comprehension varied as a function of students' oral reading fluency and that different score types had varying predictive validity for year-end reading comprehension. The mean of all three passages demonstrated a marginally better balance in screening efficiency from September to December of grade one (especially for low-performing students), whereas in grades two and three, the median score was the best predictor. Furthermore, across all grades, increasing reading rates were observed for the three administered passages within an assessment period. The observed patterns mimicked previous experimental studies (Francis et al., 2008; Jenkins, Graff, & Miglioretti, 2009), suggesting that practice effects are an important consideration in the administration of multiple passages assessing oral reading fluency.