Engineering a Twenty-First Century Reading Comprehension Assessment System Utilizing Scenario-Based Assessment Techniques

The construct of reading comprehension has changed significantly in the twenty-first century; however, some test designs have not evolved sufficiently to capture these changes. Specifically, the nature of literacy sources and skills required has changed (wrought primarily by widespread use of digital technologies). Modern theories of comprehension and discourse processes have been developed to accommodate these changes, and the learning sciences have followed suit. These influences have significant implications for how we think about the development of comprehension proficiency across grades. In this paper, we describe a theoretically driven, developmentally sensitive assessment system based on a scenario-based assessment paradigm, and present evidence for its feasibility and psychometric soundness.     

Moving words: dynamic representations in language comprehension

Eighty-two participants listened to sentences and then judged whether two sequentially presented visual objects were the same. On critical trials, participants heard a sentence describe the motion of a ball toward or away from the observer (e.g., “The pitcher hurled the softball to you”). Seven hundred and fifty milliseconds after the offset of the sentence, a picture of an object was presented for 500 ms, followed by another picture. On critical trials, the two pictures depicted the kind of ball mentioned in the sentence. The second picture was displayed 175 ms after the first. Crucially, it was either slightly larger or smaller than the first picture, thus suggesting movement of the ball toward or away from the observer. Participants responded more quickly when the implied movement of the balls matched the movement described in the sentence. This result provides support for the view that language comprehension involves dynamic perceptual simulations.     

Highlights as an Early Predictor of Student Comprehension and Interests

When engaging with a textbook, students are inclined to highlight key content. Although students believe that highlighting and subsequent review of the highlights will further their educational goals, the psychological literature provides little evidence of benefits. Nonetheless, a student’s choice of text for highlighting may serve as a window into her mental state—her level of comprehension, grasp of the key ideas, reading goals, and so on. We explore this hypothesis via an experiment in which 400 participants read three sections from a college-level biology text, briefly reviewed the text, and then took a quiz on the material. During initial reading, participants were able to highlight words, phrases, and sentences, and these highlights were displayed along with the complete text during the subsequent review. Consistent with past research, the amount of highlighted material is unrelated to quiz performance. Nonetheless, highlighting patterns may allow us to infer reader comprehension and interests. Using multiple representations of the highlighting patterns, we built probabilistic models to predict quiz performance and matrix factorization models to predict what content would be highlighted in one passage from highlights in other passages. We find that quiz score prediction accuracy reliably improves with the inclusion of highlighting data (by about 1%–2%), both for held-out students and for held-out student questions (i.e., questions selected randomly for each student), but not for held-out questions. Furthermore, an individual’s highlighting pattern is informative of what she highlights elsewhere. Our long-term goal is to design digital textbooks that serve not only as conduits of information into the reader’s mind but also allow us to draw inferences about the reader at a point where interventions may increase the effectiveness of the material.