Curriculum-based measurement of math problem solving: A methodology and rationale for establishing equivalence of scores

The purpose of this article is to discuss curriculum-based measurement (CBM) as it is currently utilized in research and practice and to propose a new approach for developing measures to monitor the academic progress of students longitudinally. To accomplish this, we first describe CBM and provide several exemplars of CBM in reading and mathematics. Then, we present the research context for developing a set of seven curriculum-based measures for monitoring student progress in math problem solving. The rationale for and advantages of using statistical equating methodology are discussed. Details of the methodology as it was applied to the development of these math problem solving measures are provided.     

Curriculum-based measurement of oral reading (R-CBM): A diagnostic test accuracy meta-analysis of evidence supporting use in universal screening

A great deal of research over the past decade has examined the appropriateness of curriculum-based measurement of oral reading (R-CBM) in universal screening. Multiple researchers have meta-analyzed available correlational evidence, yielding support for the interpretation of R-CBM as an indicator of general reading proficiency. In contrast, researchers have yet to synthesize diagnostic accuracy evidence, which pertains to the defensibility of the use of R-CBM for screening purposes. The overall purpose of this research was to therefore conduct the first meta-analysis of R-CBM diagnostic accuracy research. A systematic search of the literature resulted in the identification of 34 studies, including 20 peer-reviewed articles, 7 dissertations, and 7 technical reports. Bivariate hierarchical linear models yielded generalized estimates of diagnostic accuracy statistics, which predominantly exceeded standards for acceptable universal screener performance. For instance, when predicting criterion outcomes within a school year (≤ 9 months), R-CBM sensitivity ranged between .80 and .83 and specificity ranged between .71 and .73. Multiple moderators of R-CBM diagnostic accuracy were identified, including the (a) R-CBM cut score used to define risk, (b) lag in time between R-CBM and criterion test administration, and (c) percentile rank corresponding to the criterion test cut score through which students were identified as either truly at risk or not at risk. Follow-up analyses revealed substantial variability of extracted cut scores within grade and time of year (i.e., fall, winter, and spring). This result called into question the inflexible application of a single cut score across contexts and suggested the potential necessity of local cut scores. Implications for practices, directions for future research, and limitations are discussed.     

Examiner error in curriculum-based measurement of oral reading

Although curriculum based measures of oral reading (CBM-R) have strong technical adequacy, there is still a reason to believe that student performance may be influenced by factors of the testing situation, such as errors examiners make in administering and scoring the test. This study examined the construct-irrelevant variance introduced by examiners using a cross-classified multilevel model. We sought to determine the extent of variance in student CBM-R scores attributable to examiners and, if present, the extent to which it was moderated by students' grade level and English learner (EL) status. Fit indices indicated that a cross-classified random effects model (CCREM) best fits the data with measures nested within students, students nested within schools, and examiners crossing schools. Intraclass correlations of the CCREM revealed that roughly 16% of the variance in student CBM-R scores was associated between examiners. The remaining variance was associated with the measurement level, 3.59%; between students, 75.23%; and between schools, 5.21%. Results were moderated by grade level but not by EL status. The discussion addresses the implications of this error for low-stakes and high-stakes decisions about students, teacher evaluation systems, and hypothesis testing in reading intervention research.     

An investigation of the psychometric evidence of CBM-R passage equivalence: Utility of readability statistics and equating for alternate forms

Claims have been made that grade appropriate curriculum-based measurement of reading (CBM-R) passages are of comparable difficulty and can be used interchangeably to monitor student progress. Empirical evidence to support claims of equivalence has been lacking. This research investigated the basis for making claims of equivalence. The use of readability statistics to justify passage equivalence was found to be lacking. Using a general measurement model for congeneric tests, CBM-R passages were found to measure a single latent factor with a high degree of reliability. However, evidence indicated that the raw scores, words read correctly per minute, across passages did not provide equivalent measurements. Statistical equating was investigated as an approach to overcome the lack of equivalence with promising results.     

Curriculum-based measures of writing across grade levels

This study compared the technical adequacy of curriculum-based measures in written expression across diverse grade levels. Students (n = 484) completed two narrative writing samples in response to story starters. Samples were scored for the number of words written (TW), the number of correct word sequences (CWS), and the number of correct minus incorrect word sequences (CWS-ICWS). Alternate-form reliability and criterion-related validity coefficients of the curriculum-based measures were examined to determine differences between the coefficients according to grade level.Results indicated moderate to strong alternate-form reliability coefficients and variable criterion-related validity coefficients. The data from this study support the need for curriculum-based measures of writing to become more complex as students age and become more skilled.     

A preliminary investigation of within-year growth patterns in reading and mathematics curriculum-based measures

This study examined patterns of growth across benchmark assessments for curriculum-based measures (CBM) over 2 academic years, with the twofold purpose of replicating earlier findings of growth patterns on R-CBM and conducting a preliminary investigation of growth patterns on M-CBM and CBM-Maze. The sample included 898 cases from 3rd through 5th grade over 2 academic years from one elementary school in the rural Midwest. All students participated in tri-annual benchmark assessments in which they were administered R-CBM, CBM-Maze, and M-CBM. Repeated measures analysis of variance was used to examine patterns of growth for all three measures. Students demonstrated greater R-CBM weekly growth during winter-to-spring than fall-to-winter across grade levels. Patterns for CBM-Maze and M-CBM were somewhat inconsistent across school years and grade levels. Results are discussed in terms of contextual variables that may impact within-year growth patterns and technical characteristics of CBM slopes.     

Incremental and predictive utility of formative assessment methods of reading comprehension

Formative assessment measures are commonly used in schools to assess reading and to design instruction accordingly. The purpose of this research was to investigate the incremental and concurrent validity of formative assessment measures of reading comprehension. It was hypothesized that formative measures of reading comprehension would contribute more to our understanding of students' overall reading abilities than simply oral reading fluency (ORF). It was also hypothesized that measures could be modeled in a meaningful way to explain student performance on criterion measures of academic competence. Four formative measures of reading comprehension – maze (MZ), retell fluency (RTF), written retell (WRT), and sentence verification technique (SVT) – were used to measure unique aspects of reading comprehension through production-type responses. Results suggested that reading comprehension measures, when combined with ORF, added to the total variance associated with reading ability and were reliable indicators of student performance on a high stakes criterion-referenced assessment of grade-level literacy.     

Examining the role of time and language type in reading development for English Language Learners

The purpose of this study was to examine the development of English reading achievement among English Language Learners (ELLs) and to determine whether the time that an ELL's family was in the United States and the type of native language spoken affected their reading development. Participants were 300 third-grade ELLs from two different native language backgrounds (93 Somali-speaking and 207 Spanish-speaking students) who attended a large Midwestern urban school district. Students' reading achievement was assessed using curriculum-based measurement and a statewide reading assessment. Moderated multiple regression and multiple-group latent growth curve analyses were conducted. Results indicated that the time an ELL's family had been in the U.S. was an important factor in understanding the development of ELLs' reading achievement, whereas language type did not appear to be as important. Implications for research and practice associated with understanding and promoting English reading development among ELLs are discussed.     

Teaching reading and spelling: exclusion and stimulus equivalence

In Experiment 1, 7 nonreading children were exposed to a program designed to teach reading of 51 training words. The program featured an exclusion-based procedure in which the children (a) matched printed to dictated words and (b) constructed (copied) printed words with movable letters and named them. All children learned to read the training words. Five children also read generalization words and showed progress in spelling. Experiment 2 applied the program to 4 different children, omitting the word-construction task. They also learned to read the training words, but only 1 participant read generalization words. The data support a stimulus equivalence account of reading acquisition and suggest that reading generalization may be obtained, especially when the teaching program includes word construction.     

出声阅读和默读之间的差异

阅读有两种基本模式：默读和出声阅读。文章从三个方面总结了两种阅读模式差异的研究成果：(1)不同阅读模式下阅读的行为差异; (2)不同阅读模式下阅读的认知机制差异; (3)不同阅读模式下阅读的脑机制差异。在此基础上, 文章提出了未来研究需要解决的问题：(1)出声阅读向默读转变的具体阶段; (2)两种阅读模式发展过程中眼动行为的差异; (3)语音和背景音乐对两种阅读模式下阅读的影响; (4)两种阅读模式下儿童的脑机制差异等。     

默读和出声阅读的副中央凹预视效应

以往研究大多考察默读中是否存在副中央凹预视效应及存在何种类型的预视效应,很少有研究探讨出声阅读中的副中央凹预视效应以及阅读中预视信息提取的时间进程。本研究采用边界范式,通过两个眼动实验考察默读和出声阅读中的副中央凹预视效应及预视信息提取的时间进程。实验1发现在默读和出声阅读中都存在副中央凹预视效应,但是默读中副中央凹预视效应更大。实验2发现出声阅读中在注视中央凹词汇的早期不能提取副中央凹处信息。两个实验的结果表明默读和出声阅读条件下的副中央凹加工是不同的,需要用不同的眼动模型解释。     

A review of research and theory on the relation between oral reading rate and reading comprehension

We review the conceptual and empirical literature on the relation between oral reading rate and reading comprehension. Three lines of conceptual analysis converge on this relation: (a) application of basic behavior analytic principles suggests that fluent decoding should produce better reading comprehension through direct and indirect relations, (b) behavior analytic understanding of the importance of the rate of behavior as developed by Skinner, Lindsley and Haughton implies that higher reading rate contributes to improved comprehension, and (c) cognitive theory of automaticity explicitly states that high rate reading sets the stage for effective comprehension. A wealth of correlational evidence indicates that reading rate and reading comprehension covary. These results have been replicated across elementary grades and across a variety of measures of reading comprehension. However, experimental analyses have not convincingly demonstrated a functional relation between the two. Experimental work has yielded results that are mixed at best. Examination of experimental design issues shows that although this is not a simple relation to investigate, behavior analysts can make major contributions to understanding the possible functional relation between reading rate and reading comprehension.     

In search of average growth: Describing within-year oral reading fluency growth across Grades 1–8

Measures of oral reading fluency (ORF) are perhaps the most often used assessment to monitor student progress as part of a response to intervention (RTI) model. Rates of growth in research and aim lines in practice are used to characterize student growth; in either case, growth is generally defined as linear, increasing at a constant rate. Recent research suggests ORF growth follows a nonlinear trajectory, but limitations related to the datasets used in such studies, composed of only three testing occasions, curtails their ability to examine the true functional form of ORF growth. The purpose of this study was to model within-year ORF growth using up to eight testing occasions for 1448 students in Grades 1 to 8 to assess (a) the average growth trajectory for within-year ORF growth, (b) whether students vary significantly in within-year ORF growth, and (c) the extent to which findings are consistent across grades. Results demonstrated that for Grades 1 to 7, a quadratic growth model fit better than either linear or cubic growth models, and for Grade 8, there was no substantial, stable growth. Findings suggest that the expectation for linear growth currently used in practice may be unrealistic.     

Performance Indicators in Math: Implications for Brief Experimental Analysis of Academic Performance

Brief experimental analysis (BEA) can be used to specify intervention characteristics that produce positive learning gains for individual students. A key challenge to the use of BEA for intervention planning is the identification of performance indicators (including topography of the skill, measurement characteristics, and decision criteria) that meaningfully relate to longer term success in the learning environment. This study investigates the utility of various curriculum-based assessment and measurement estimates of mathematics performance for predicting functional outcomes (i.e., retention of learned skills over time and faster learning of related content in the future). All children in grades 2–5 at the participating school participated in protocol-based computational fluency-building intervention 4 days per week for an entire school year. Specific criteria were applied each week to systematically increase intervention difficulty classwide according to a pre-established sequence of computational skill objectives. Three measurements were routinely obtained. Each week children completed a timed probe of the skill for which intervention was currently occurring and a timed probe of previously mastered skills from the sequence of computational skill objectives. Each month, all children completed a timed probe of mathematics skills representing computational skills that students were expected to master by year’s end at each grade level. At all grade levels, learning a skill that appeared early in the hierarchy or sequence of skills related positively to learning of future related and more complex computational skills. Fluency criteria were specified that predicted retention of the skill over several months.     

Parafoveal preview benefits during silent and oral reading: Testing the parafoveal information extraction hypothesis

The preview of a parafoveally visible word conveys benefits when it is subsequently fixated. The current study examined whether these benefits are determined by the effectiveness of parafoveal information extraction, as implied by current models of eye movement control during reading, or by the effectiveness with which extracted information is integrated when a previewed word is fixated. For this, the boundary technique was used to manipulate the extent to which parafoveal information could be extracted, and text was read silently or orally. Consistent with prior work, a parafoveal target word preview conveyed fewer benefits when less parafoveal information could be extracted, target viewing durations were longer during oral than during silent reading, and the two factors interacted in the target fixation data, with smaller preview benefits during oral than during silent reading. Survival analyses indicated that this occurred because parafoveal information use occurred at later point in time during oral reading. Diminished opportunity for parafoveal information extraction also diminished target skipping rate, and it resulted in smaller saccades to target words, but these effects were not influenced by reading mode. Parafoveally extracted information was thus used less effectively during oral reading only when it involved the integration of parafoveally extracted information during subsequent target viewing. The dissociation of extraction from integration challenges current models of eye movement control.     

Further development of measures of early math performance for preschoolers

The purpose of this study was to examine the progress monitoring and screening accuracy for a set of curriculum-based measures (CBM) of early mathematics skills. Measures included counting objects, selecting numbers, naming numbers, counting, and visual discrimination. Measures were designed to be administered with preschoolers in a short period of time using a developmentally appropriate format, and to constitute minimal disruption to the classroom routine. Previous research indicated that each of these measures produced scores with acceptable consistency across alternate forms on consecutive days. Scores yielded on the experimental probes correlated moderately, in most cases, with two commonly used standardized measures (i.e., the Brigance Screens and TEMA-2). Performance on the probes also correlated with teacher rankings and ratings of child performance. In the present study, data were collected in two phases. In the first phase, a cohort of children were administered the early math measures in spring of preschool and were followed into winter of kindergarten where they were administered kindergarten CBM probes to examine the degree to which performance in preschool (as measured by the early math probes) could predict performance on similar tasks in kindergarten. Second, risk criteria were examined and compared to identification by the Brigance Screens. These analyses provided data about the potential accuracy of the probes for screening. In the second phase, sensitivity of the probes to performance differences was examined by comparing mean performance on the probes of preschoolers and kindergartners. The lowest-performing children were provided with seven intervention sessions to examine the degree to which probes could detect growth resulting from intervention. Directions for future research and implications for practice are discussed.     

Improving oral reading fluency: a brief experimental analysis of combining an antecedent intervention with consequences

A brief experimental analysis was used to evaluate the relative effectiveness of combining two consequences (contingent reinforcement or performance feedback) with an antecedent intervention (listening passage preview and repeated readings) on the oral reading fluency of 6 elementary students. The antecedent intervention increased the number of correctly read words per minute for all 6 students. For 4 of the students, pairing the antecedent intervention with either of the consequences resulted in higher reading rates over the antecedent intervention alone. Undifferentiated results were obtained for the remaining 2 participants. These results suggest that combining an antecedent intervention with consequences may enhance the oral reading fluency of students with reading problems. However, individual responsiveness to the different intervention components indicates that brief experimental analyses are warranted to identify the most effective intervention.     

Simplifying reading: applying the simplicity principle to reading

Debates concerning the types of representations that aid reading acquisition have often been influenced by the relationship between measures of early phonological awareness (the ability to process speech sounds) and later reading ability. Here, a complementary approach is explored, analyzing how the functional utility of different representational units, such as whole words, bodies (letters representing the vowel and final consonants of a syllable), and graphemes (letters representing a phoneme) may change as the number of words that can be read gradually increases. Utility is measured by applying a Simplicity Principle to the problem of mapping from print to sound; that is, assuming that the "best" representational units for reading are those which allow the mapping from print to sounds to be encoded as efficiently as possible. Results indicate that when only a small number of words are read whole-word representations are most useful, whereas when many words can be read graphemic representations have the highest utility.     

The utility and accuracy of oral reading fluency score types in predicting reading comprehension

This study used data from the Dynamic Indicators of Basic Early Literacy Skills (DIBELS; Good & Kaminski, 2002) oral reading fluency (ORF) probes to examine variation among different ORF score types (i.e., the median of three passages, the mean of all three passages, the mean of passages 2 and 3, and the score from passage 3) in predicting reading comprehension as a function of student reading fluency level and to compare the screening accuracy of these score types in predicting student reading comprehension. The results revealed that the relation between oral reading fluency and reading comprehension varied as a function of students' oral reading fluency and that different score types had varying predictive validity for year-end reading comprehension. The mean of all three passages demonstrated a marginally better balance in screening efficiency from September to December of grade one (especially for low-performing students), whereas in grades two and three, the median score was the best predictor. Furthermore, across all grades, increasing reading rates were observed for the three administered passages within an assessment period. The observed patterns mimicked previous experimental studies (Francis et al., 2008; Jenkins, Graff, & Miglioretti, 2009), suggesting that practice effects are an important consideration in the administration of multiple passages assessing oral reading fluency.     

The effects of linearity on sentence comprehension in oral and silent reading

This study examined the effects of linearity of language (e.g., de Saussure, 1916/1972) on performance on a sentence comprehension task, manipulating (a) presentation method (simultaneous, phrase, or flow) and (b) reading method (oral or silent reading). The study also examined the predictive powers of the digit span test (DST) and the reading span test (RST) for task performance. Fifty‐four undergraduate and graduate students participated in the experiment. The results showed that the DST score predicted performance on both garden‐path (GP) and non‐garden‐path (NGP) sentences, whereas the RST score did so only for GP sentences. This suggests that the sentence comprehension task requires not only comprehension but also temporarily storing verbal information, and that the RST score reflects the degree of tolerance for deviation from linearity. The results also revealed that silent reading has an advantage over oral reading in simultaneous presentation and vice versa in flow presentation, whereas the difference between reading methods disappeared in phrase presentation. This suggests that oral and silent reading require different strategies to facilitate sentence comprehension.