Generalizability theory: A practical guide to study design,implementation, and interpretation

Generalizability Theory (GT) offers increased utility for assessment research given the ability to concurrently examine multiple sources of variance, inform both relative and absolute decision making, and determine both the consistency and generalizability of results. Despite these strengths, assessment researchers within the fields of education and psychology have been slow to adopt and utilize a GT approach. This underutilization may be due to an incomplete understanding of the conceptual underpinnings of GT, the actual steps involved in designing and implementing generalizability studies, or some combination of both issues. The goal of the current article is therefore two-fold: (a) to provide readers with the conceptual background and terminology related to the use of GT and (b) to facilitate understanding of the range of issues that need to be considered in the design, implementation, and interpretation of generalizability and dependability studies. Given the relevance of this analytic approach to applied assessment contexts, there exists a need to ensure that GT is both accessible to, and understood by, researchers in education and psychology. Important methodological and analytical considerations are presented and implications for applied use are described.     

Neural network applications in consumer behavior

This article introduces the concepts and terminology of artificial neural networks. The approach is demonstrated on data that represent a domain of interest to the consumer psychologist. ANNs are mathematical models that are commonly used in business applications to model relationships between variables. A key strength of ANNs is their flexibility, i.e. their ability to easily accommodate linear and non-linear relationships without any a priori functional form specification. They can easily be used to study topics of interest to consumer psychologists such as persuasion, influence, segmentation etc and can offer distinct advantages over traditional statistical techniques such as ANOVA and regression. We demonstrate the application of ANNs in three different areas: regression, non-linear principal component analysis and classification.     

A Comparative Effectiveness Study of Two High-Frequency Word Interventions: Traditional Drill and WordSheets

Traditional drill and practice (TDP) has been shown to be both effective and efficient for increasing word recognition. WordSheets (WS) is a novel drill procedure similar to TDP with two key differences: (a) WS increases opportunities to respond, which may increase efficiency and (b) WS present instructional stimuli in a context that more closely resembles actual reading passages, which may enhance generalization to connected text. The study sought to compare the effectiveness and efficiency of TDP and WS in a sample of first-grade students (n = 27) identified by their teachers as struggling readers. Results indicated that, although WS led to significantly more opportunities to respond, it was both less effective and efficient than TDP with regard to cumulative retention. However, there were no significant differences across interventions in regard to the maintenance or generalization of treatment gains. Limitations and implications of the current study for school-based professionals are discussed.     

Daily Behavior Report Cards and Systematic Direct Observation: An Investigation of the Acceptability,Reported Training and Use,and Decision Reliability Among School Psychologists

More than ever, educators require assessment procedures and instrumentation that are technically adequate as well as efficient to guide data-based decision making. Thus, there is a need to understand perceptions of available tools, and the decisions made when using collected data, by the primary users of those data. In this paper, two studies that surveyed members of the National Association of School Psychologists with regard to two procedures useful in formative assessment, (i.e., Daily Behavior Report Cards; Systematic Direct Observation), are presented. Participants reported greater overall levels of training and use of Systematic Direct Observation than Daily Behavior Report Cards, yet both techniques were rated as equally acceptable for use in formative assessment. Furthermore, findings supported that school psychologists tend to make similar intervention decisions when presented with both types of data. Implications, limitations, and future directions are discussed.     

A Comparison of Rubrics for Identifying Empirically Supported Practices with Single-Case Research

The use of single-case research methods for validating academic and behavioral interventions has gained considerable attention in recent years. As such, there has been a proliferation of methods for evaluating whether, and to what extent, primary research reports provide evidence of intervention effectiveness. Despite the recent interest in harnessing single-case research to identify empirically supported strategies, examination of these tools has revealed that there is a lack of consistency in the methodological criteria sampled and scoring procedures used to evaluate primary research reports. The present study examined the extent to which various evidence rubrics addressed specific methodological features of single-case research and classified studies into similar evidence categories. Results indicated that the methodological criteria included within rubrics tended to vary, particularly for criteria related to determining the generality of the intervention under study. Moreover, there was substantial discordance observed in the evidence classifications assigned to reviewed studies. These findings are discussed in the context of the still-developing nature of single-case evidence reviews. Recommendations for both research and practice are provided.     

An investigation of the generalizability and dependability of Direct Behavior Rating Single Item Scales (DBR-SIS) to measure academic engagement and disruptive behavior of middle school students

A total of 4 raters, including 2 teachers and 2 research assistants, used Direct Behavior Rating Single Item Scales (DBR-SIS) to measure the academic engagement and disruptive behavior of 7 middle school students across multiple occasions. Generalizability study results for the full model revealed modest to large magnitudes of variance associated with persons (students), occasions of measurement (day), and associated interactions. However, an unexpectedly low proportion of the variance in DBR data was attributable to the facet of rater, as well as a negligible variance component for the facet of rating occasion nested within day (10-min interval within a class period). Results of a reduced model and subsequent decision studies specific to individual rater and rater type (research assistant and teacher) suggested degree of reliability-like estimates differed substantially depending on rater. Overall, findings supported previous recommendations that in the absence of estimates of rater reliability and firm recommendations regarding rater training, ratings obtained from DBR-SIS, and subsequent analyses, be conducted within rater. Additionally, results suggested that when selecting a teacher rater, the person most likely to substantially interact with target students during the specified observation period may be the best choice.     

Evidence for a late-occurring effect of phoneme repetition during silent reading

Two reading experiments were conducted to investigate the effect of phoneme repetition during silent reading (also referred to as the visual tongue-twister effect, see McCutchen & Perfetti, 1982; Perfetti & McCutchen, 1982) in conjunction with end-of-clause and end-of-sentence wrap-up effects (Just & Carpenter, 1982; Rayner, Sereno, Morris, Schmauder, & Clifton, 1989). In both experiments, reading time was measured on sentences containing either six or zero words sharing the same initial phoneme. Sentences were presented in a phrase-by-phrase moving window, with each phrase either containing one word that involved a repeated phoneme or a matched word not involving a repeated phoneme. The sixth presentation region either contained or did not contain a comma (Experiment 1) or a period (Experiment 2). The results showed that the effect of phoneme repetition occurred relatively late during sentence processing, only after integration processing related to end-of-clause and end-of-sentence wrap-up occurred.     

A Comparison of Two Flashcard Drill Methods Targeting Word Recognition

Traditional drill and practice (TD) and incremental rehearsal (IR) are two flashcard drill instructional methods previously noted to improve word recognition. The current study sought to compare the effectiveness and efficiency of these two methods, as assessed by next day retention assessments, under 2 conditions (i.e., opportunities to respond held constant across methods, length of instructional session held constant across methods) with 4 first-grade students identified by their teachers as struggling readers. Social validity of the two intervention methods also was evaluated. Results suggested important differences in response to instruction across students. Differences in effectiveness between TD and IR were minimal both when holding opportunities to respond and length of instructional session constant across methods. Minimal differences in efficiency were noted between TD and IR when holding instructional time constant across conditions. However, TD was clearly more efficient than IR for all students when holding opportunities to respond constant across conditions. Social validity data indicated that half of the students preferred the TD method and half preferred the IR method. Limitations and implications of the current study for school-based professionals are discussed.     

The efficiency of behavior rating scales to assess inattentive-overactive and oppositional-defiant behaviors: applying generalizability theory to streamline assessment

Although the efficiency with which a wide range of behavioral data can be obtained makes behavior rating scales particularly attractive tools for the purposes of screening and evaluation, feasibility concerns arise in the context of formative assessment. Specifically, informant load, or the amount of time informants are asked to contribute to the assessment process, likely has a negative impact on the quality of data over time and the informant's willingness to participate. Two important determinants of informant load in progress monitoring are the length of the rating scale (i.e., the number of items) and how frequently informants are asked to provide ratings (i.e., the number of occasions). The purpose of the current study was to investigate the dependability of the IOWA Conners Teacher Rating Scale (Loney & Milich, 1982), which is used to differentiate inattentive-overactive from oppositional-defiant behaviors. Specifically, the facets of items and occasions were examined to identify combinations of these sources of error necessary to reach an acceptable level of dependability for both absolute and relative decisions. Results from D studies elucidated a variety of possible item-occasion combinations reaching the criteria for adequate dependability. Recommendations for research and practice are discussed.     

Stability Balls and Student on-Task Behavior

Stability balls have become a common option for alternative seating in classrooms. While the social validity of stability ball classroom seating remains high among students and teachers, findings on the effectiveness of this alternative seating method are mixed. This is particularly true when the intent of the intervention is to increase student on-task behavior. This study examined the effects of stability ball seating on students’ on-task behavior in a general education setting, using a within-subjects repeated measures design. Researchers observed twenty-four third-grade students in a public-school classroom across three conditions: all chairs, all balls, or choice. Across observation days, the study controlled for time of day, teacher, and classroom environment. Results indicate students are on-task less frequently when sitting on a stability ball. Student reports on social validity replicated prior findings; students found the stability ball to be desirable and easy to use.