Randomization-based inference about latent variables from complex samples

Standard procedures for drawing inferences from complex samples do not apply when the variable of interest cannot be observed directly, but must be inferred from the values of secondary random variables that depend on stochastically. Examples are proficiency variables in item response models and class memberships in latent class models. Rubin's multiple imputation techniques yield approximations of sample statistics that would have been obtained, had been observable, and associated variance estimates that account for uncertainty due to both the sampling of respondents and the latent nature of. The approach is illustrated with data from the National Assessment for Educational Progress.This research was supported by Grant No. NIE-G-83-0011 of the Office for Educational Research and Improvement, Center for Education Statistics, and Contract No. N00014-88-K-0304, R&T 4421552 from the Cognitive Sciences Program, Cognitive and Neural Sciences Division, Office of Naval Research. It does not necessarily reflect the views of either agency. I am grateful to R. Darrell Bock for calling my attention to the applicability of multiple imputation to the assessment setting; to Albert Beaton and Eugene Johnson for enlightening discussions on the topic; and to Henry Braun, Ben King, Debra Kline, Gary Phillips, Paul Rosenbaum, Don Rubin, John Tukey, Ming-Mei Wang, Kentaro Yamamoto, Rebecca Zwick, and two anonymous reviewers for comments on earlier drafts. Example 4 is based on the analysis of the 1984 National Assessment for Educational Progress reading survey, carried out at Educational Testing Service through the tireless efforts of too many people to mention by name, under the direction of Albert Beaton, Director of NAEP Data Analyses. David Freund, Bruce Kaplan, and Jennifer Nelson conducted additional analyses of the 1984 and 1988 data for the example.     

The efficiency of behavior rating scales to assess inattentive-overactive and oppositional-defiant behaviors: applying generalizability theory to streamline assessment

Although the efficiency with which a wide range of behavioral data can be obtained makes behavior rating scales particularly attractive tools for the purposes of screening and evaluation, feasibility concerns arise in the context of formative assessment. Specifically, informant load, or the amount of time informants are asked to contribute to the assessment process, likely has a negative impact on the quality of data over time and the informant's willingness to participate. Two important determinants of informant load in progress monitoring are the length of the rating scale (i.e., the number of items) and how frequently informants are asked to provide ratings (i.e., the number of occasions). The purpose of the current study was to investigate the dependability of the IOWA Conners Teacher Rating Scale (Loney & Milich, 1982), which is used to differentiate inattentive-overactive from oppositional-defiant behaviors. Specifically, the facets of items and occasions were examined to identify combinations of these sources of error necessary to reach an acceptable level of dependability for both absolute and relative decisions. Results from D studies elucidated a variety of possible item-occasion combinations reaching the criteria for adequate dependability. Recommendations for research and practice are discussed.     

Further development of measures of early math performance for preschoolers

The purpose of this study was to examine the progress monitoring and screening accuracy for a set of curriculum-based measures (CBM) of early mathematics skills. Measures included counting objects, selecting numbers, naming numbers, counting, and visual discrimination. Measures were designed to be administered with preschoolers in a short period of time using a developmentally appropriate format, and to constitute minimal disruption to the classroom routine. Previous research indicated that each of these measures produced scores with acceptable consistency across alternate forms on consecutive days. Scores yielded on the experimental probes correlated moderately, in most cases, with two commonly used standardized measures (i.e., the Brigance Screens and TEMA-2). Performance on the probes also correlated with teacher rankings and ratings of child performance. In the present study, data were collected in two phases. In the first phase, a cohort of children were administered the early math measures in spring of preschool and were followed into winter of kindergarten where they were administered kindergarten CBM probes to examine the degree to which performance in preschool (as measured by the early math probes) could predict performance on similar tasks in kindergarten. Second, risk criteria were examined and compared to identification by the Brigance Screens. These analyses provided data about the potential accuracy of the probes for screening. In the second phase, sensitivity of the probes to performance differences was examined by comparing mean performance on the probes of preschoolers and kindergartners. The lowest-performing children were provided with seven intervention sessions to examine the degree to which probes could detect growth resulting from intervention. Directions for future research and implications for practice are discussed.     

Should “multiple imputations” be treated as “multiple indicators”?

Rubin's multiple imputation approach to missing data creates synthetic data sets, in which each missing variable is replaced by a draw from its predictive distribution, conditional on the observed data. By construction, analyses of such filled-in data sets as if the imputations were true values have the correct expectations for population parameters. In a recent paper, Mislevy showed how this approach can be applied to estimate the distributions of latent variables from complex samples. Multiple imputations for a latent variable bear a surface similarity to classical multiple indicators of a latent variable, as might be addressed in structural equation modelling or hierarchical modelling of successive stages of random sampling. This note demonstrates with a simple example why analyzing multiple imputations as if they were multiple indicators does not generally yield correct results; they must instead be analyzed by means concordant with their construction.I am grateful to Frank Jenkins, John Mazzeo, Kentaro Yamamoto, and Rebecca Zwick for comments on earlier versions of this paper.     

Considerations for evaluating universal screening assessments

Universal screening is a critical prerequisite to providing early school-based prevention and intervention services for students at risk for or with academic, behavioral, or emotional difficulties. Although use of academic and behavioral screening has become more prevalent, criteria for making informed decisions about appropriate screening tools are not readily available to those who work in educational settings. The purpose of this article is to (a) highlight several considerations about the appropriateness, technical adequacy, and usability of academic or behavioral screeners and (b) provide a general assessment of the current state of science pertaining to universal screening. The article concludes with considerations for policy and practice. The framework presented may be useful in guiding those interested in evaluating, selecting, and researching school-based universal screening assessments.