Generalizability theory: A practical guide to study design,implementation, and interpretation

Generalizability Theory (GT) offers increased utility for assessment research given the ability to concurrently examine multiple sources of variance, inform both relative and absolute decision making, and determine both the consistency and generalizability of results. Despite these strengths, assessment researchers within the fields of education and psychology have been slow to adopt and utilize a GT approach. This underutilization may be due to an incomplete understanding of the conceptual underpinnings of GT, the actual steps involved in designing and implementing generalizability studies, or some combination of both issues. The goal of the current article is therefore two-fold: (a) to provide readers with the conceptual background and terminology related to the use of GT and (b) to facilitate understanding of the range of issues that need to be considered in the design, implementation, and interpretation of generalizability and dependability studies. Given the relevance of this analytic approach to applied assessment contexts, there exists a need to ensure that GT is both accessible to, and understood by, researchers in education and psychology. Important methodological and analytical considerations are presented and implications for applied use are described.     

The effects of environmental support and secondary tasks on visuospatial working memory

In the present experiments, we examined the effects of environmental support on participants’ ability to rehearse locations and the role of such support in the effects of secondary tasks on memory span. In Experiment 1, the duration of interitem intervals and the presence of environmental support for visuospatial rehearsal (i.e., the array of possible memory locations) during the interitem intervals were both manipulated across four tasks. When support was provided, memory spans increased as the interitem interval durations increased, consistent with the hypothesis that environmental support facilitates rehearsal. In contrast, when environmental support was not provided, spans decreased as the duration of the interitem intervals increased, consistent with the hypothesis that visuospatial memory representations decay when rehearsal is impeded. In Experiment 2, the ratio of interitem interval duration to intertrial interval duration was kept the same on all four tasks, in order to hold temporal distinctiveness constant, yet forgetting was still observed in the absence of environmental support, consistent with the decay hypothesis. In Experiment 3, the effects of impeding rehearsal were compared to the effects of verbal and visuospatial secondary processing tasks. Forgetting of locations was greater when presentation of to-be-remembered locations alternated with the performance of a secondary task than when rehearsal was impeded by the absence of environmental support. The greatest forgetting occurred when a secondary task required the processing visuospatial information, suggesting that in addition to decay, both domain-specific and domain-general effects contribute to forgetting on visuospatial working memory tasks.     

Influence of emotional valence and arousal on the spread of activation in memory

Controversy still persists on whether emotional valence and arousal influence cognitive activities. Our study sought to compare how these two factors foster the spread of activation within the semantic network. In a lexical decision task, prime words were varied depending on the valence (pleasant or unpleasant) or on the level of emotional arousal (high or low). Target words were carefully selected to avoid semantic priming effects, as well as to avoid arousing specific emotions (neutral). Three SOA durations (220, 420 and 720 ms) were applied across three independent groups. Results indicate that at 220 ms, the effect of arousal is significantly higher than the effect of valence in facilitating spreading activation while at 420 ms, the effect of valence is significantly higher than the effect of arousal in facilitating spreading activation. These findings suggest that affect is a sequential process involving the successive intervention of arousal and valence.     

A model‐based approach to limb apraxia in Alzheimer's disease

Limb apraxia is a neurological deficit characterized by an inability to pantomime and/or imitate gestures, which can result from neurodegenerative disorders such as Alzheimer's disease (AD). The major goal of the study was to describe comprehensively the apraxia deficits observed in AD patients and to relate those deficits to general cognitive status, measures of daily activity, and other neuropsychological measures. Limb apraxia was assessed on a variety of conceptual and gesture production tasks in 30 AD patients. As a group, AD patients were impaired across gesture production tasks: of note was the greater impairment in imitation, as opposed to pantomime, which was especially pronounced when patients were imitating with a delay. Imitation performance was best predicted by measures of visuospatial processing, while imitation with delay was best predicted by measures of working memory. In addition, pantomime in response to pictures of tools was less accurate than Pantomime to Verbal Command and holding the tool during performance did not decrease the participants' impairment, while introducing a verbal cue during imitation increased the severity of deficits. Furthermore, investigation into patterns of deficits clearly demonstrated that the nature of limb apraxia deficits observed in AD can be quite heterogeneous and that dissociations between the conceptual and the production system exist. Finally, we also report on significant correlations between general cognitive status and limb apraxia.     

Threats to the Internal Validity of Spinal Surgery Outcome Assessment: Recalibration Response Shift or Implicit Theories of Change?

A recalibration response shift will cause the patient to think about a self-report measure’s response options differently after a health state change. Commonly assessed using the retrospective-pretest design (“then-test”), recent guidelines suggest adjusting then-test estimates for competing explanations. This prospective longitudinal study investigated recalibration response shift after adjusting for implicit theories of change in patients undergoing spinal surgery. The Oswestry Disability Index (ODI) and Short Form-36 (SF-36) were collected before surgery and at 6 weeks and 3 months after spinal decompression surgery. Then-tests of the measures were also collected at all post-tests. Recalibration response shift was operationalized as the then-minus-pre difference score on the evaluative SF-36. Implicit theories of change were operationalized as the then-minus-pre difference score on the perception-based ODI. Improved vs. No-Effect patient groups were compared using the Minimally Important Difference (±15 points) as a cut-off on the Visual Analogue Scale (VAS) items for back and leg pain. Logistic regression analyses investigated whether recalibration response shift had an independent effect distinguishing patient groups, after adjusting for implicit theories of change. The sample (baseline n?=?169, mean age 52, 39 % female) was well-educated, and 1/3 were working. All then-minus-pre difference scores were non-zero at each time point and were stable over time. In the adjusted models distinguishing Improved versus No Effect groups, then-minus-pre ODI difference scores were significant in the majority of the adjusted models at all timepoints, but only one then-minus-pre SF-36 difference score—for physical functioning recalibration—was significant and only at 6-weeks post-surgery. This suggests that implicit theories of change bias the estimation of post-surgical outcomes, but that recalibration response shift biased only the estimation of physical functioning and only at 6 weeks post-surgery. Recalibration response shift and implicit theories of change can both be sources of bias in patient-reported outcome assessment. Our findings suggest that implicit theories of change are a greater threat to validity in this patient sample. Future research using the then-test should control for implicit theories of change to minimize misspecification of effects.     

Holistic processing of faces is intact in adults with autism spectrum disorder

Are face recognition difficulties in Autism Spectrum Disorders (ASD) related to impaired holistic processing? Gauthier et al. [Gauthier, I., Klaiman, C., &; Schultz, R. T. (2009). Face composite effects reveal abnormal face processing in Autism spectrum disorders. Vision Research, 49, 470–478. doi:10.1016/j.visres.2008.12.007] used the face composite task in adolescents with autism and found a congruency effect that was not modulated by alignment, a result which was interpreted as reflecting qualitatively different face processing mechanisms. In the present study we tested adults with ASD in a composite face task where presentation times were manipulated and further explored whether these participants were sensitive to holistic information in faces using a new version of the composite task: VHFPT 2.0 (The Vanderbilt Holistic Face Processing Test 2.0). Results suggest that adults with ASD process faces holistically and that their facial identity processing abilities are qualitatively similar and as efficient as that of typical adults. The difference between the results of Gauthier et al. (2009) with adolescents and the results with adults here reported are interpreted in terms of a developmental delay.     

Latino parents’ science beliefs and support of high school students’ motivational beliefs: Do the relations vary across gender and familism values?

Based on the Eccles’ expectancy-value theory, the objectives of this study were (1) to examine the longitudinal relations between Latino parents’ science beliefs, parents’ science support, and high school students’ science motivational beliefs, and (2) to test whether these relations varied by familism values and adolescent gender. Multi-informant longitudinal survey data were collected from 104 Latino parents and adolescents during 9th and 10th grade. Parents’ perceptions of their adolescents’ science ability and value of science in 9th grade predicted higher parent science support at home in 10th grade if they were parents of boys. In contrast, these relations were not statistically significant for parents of daughters. Parent support in 9th grade predicted higher adolescent motivational beliefs one year later. This work provides evidence that parent support is important for high school student success in this understudied population.