Generalizability theory: A practical guide to study design,implementation, and interpretation

Generalizability Theory (GT) offers increased utility for assessment research given the ability to concurrently examine multiple sources of variance, inform both relative and absolute decision making, and determine both the consistency and generalizability of results. Despite these strengths, assessment researchers within the fields of education and psychology have been slow to adopt and utilize a GT approach. This underutilization may be due to an incomplete understanding of the conceptual underpinnings of GT, the actual steps involved in designing and implementing generalizability studies, or some combination of both issues. The goal of the current article is therefore two-fold: (a) to provide readers with the conceptual background and terminology related to the use of GT and (b) to facilitate understanding of the range of issues that need to be considered in the design, implementation, and interpretation of generalizability and dependability studies. Given the relevance of this analytic approach to applied assessment contexts, there exists a need to ensure that GT is both accessible to, and understood by, researchers in education and psychology. Important methodological and analytical considerations are presented and implications for applied use are described.     

Assessing vividness of mental imagery: The Plymouth Sensory Imagery Questionnaire

Mental imagery may occur in any sensory modality, although visual imagery has been most studied. A sensitive measure of the vividness of imagery across a range of modalities is needed: the shorter version of Bett's Questionnaire upon Mental Imagery (Sheehan, 1967 , J. Clin. Psychology, 23, 386) uses outdated items and has an unreliable factor structure. We report the development and initial validation of the Plymouth Sensory Imagery Questionnaire (Psi‐Q) comprising items for each of the following modalities: Vision, Sound, Smell, Taste, Touch, Bodily Sensation, and Emotional Feeling. An exploratory factor analysis on a 35‐item form indicated that these modalities formed separate factors, rather than a single imagery factor, and this was replicated by confirmatory factor analysis. The Psi‐Q was validated against the Spontaneous Use of Imagery Scale (Reisberg et al., 2003 , Appl. Cogn. Psychology, 17, 147) and Marks' ( 1995 , J. Mental Imagery, 19, 153) Vividness of Visual Imagery Questionnaire‐2 (VVIQ‐2). A short 21‐item form comprising the best three items from the seven factors correlated with the total score and subscales of the full form, and with the VVIQ‐2. Inspection of the data shows that while visual and sound imagery is most often rated as vivid, individuals who rate one modality as strong and the other as weak are not uncommon. Findings are interpreted within a working memory framework and point to the need for further research to identify the specific cognitive processes underlying the vividness of imagery across sensory modalities.     

Axis I anxiety and mental health disorders among stuttering adolescents

PurposeThe purpose of this study was to evaluate anxiety and psychological functioning among adolescents seeking speech therapy for stuttering using a structured, diagnostic interview and psychological questionnaires. This study also sought to determine whether any differences in psychological status were evident between younger and older adolescents.MethodParticipants were 37 stuttering adolescents seeking stuttering treatment. We administered the Computerized Voice Version of the Diagnostic Interview Schedule for Children, and five psychometric tests. Participants were classified into younger (12–14 years; n = 20) and older adolescents (15–17 years; n = 17).ResultsThirty-eight percent of participants attained at least one diagnosis of a mental disorder, according to the diagnostic criteria of the Diagnostic and Statistical Manual of Mental Disorders, Fourth Edition (DSM-IV; APA, 2000), with the majority of these diagnoses involving anxiety. This figure is double current estimates for general adolescent populations, and is consistent with our finding of moderate and moderate–severe quality of life impairment. Although many of the scores on psychological measures fell within the normal range, older adolescents (15–17 years) reported significantly higher anxiety, depression, reactions to stuttering, and emotional/behavioral problems, than younger adolescents (12–14 years). There was scant evidence that self-reported stuttering severity is correlated with mental health issues. There are good reasons to believe these results are conservative because many participants gave socially desirable responses about their mental health status.DiscussionThese results reveal a need for large-scale, statistically powerful assessments of anxiety and other mental disorders among stuttering adolescents with reference to control populations.Educational Objectives: The reader will be able to: (a) explain the clinical importance of assessing for mental health with stuttering adolescents, (b) state the superior method for adolescent mental health assessment and (c) state a major issue with determining the genuineness of stuttering adolescent responses to psychological assessment.     

Criterion Validity is Maintained When Items are Evaluatively Neutralized: Evidence from a Full‐scale Five‐factor Model Inventory

The original version and an evaluatively neutralized version (with items rephrased to reduce popularity) of a personality inventory were compared. The results revealed (i) similar criterion validity across three different sets of self‐rated behaviours, (ii) stronger relations to the rated social desirability of criteria for the original version and (iii) less correlation between factors for the neutralized version. We take the results to indicate that evaluative neutralization is a viable technique for reducing social desirability in self‐ratings. Implications for test construction are discussed. Copyright © 2014 European Association of Personality Psychology     

Exploring Differential Subgroup Functioning on SAT Writing Items: What Happens When English Is Not a Test Taker's Best Language?

The purpose of this study is to explore patterns in model-data fit related to subgroups of test takers from a large-scale writing assessment. Using data from the SAT, a calibration group was randomly selected to represent test takers who reported that English was their best language from the total population of test takers (N = 322,011). A reference scale for the items was constructed based on EBL responses. Response behaviors of test takers who reported that English was not their best language (ENBL) were examined in relationship to this reference scale. This study illustrates the use of differential subgroup analyses to identify patterns related to person misfit within subgroups, as well as subsets of items, that may affect the validity of writing scores for ENBL test takers. The methodology described here offers an approach that can be used to explore, understand, and improve the validity of scores obtained from ENBL test takers in large-scale writing assessments.     

An Initial Investigation of the Factor-Analytic Structure of the Impact of Event Scale-Revised With a Volunteer Firefighter Sample

We provide an initial evaluation of the factor structure of the Impact of Event Scale-Revised (IES-R) when used with a volunteer firefighter and a similar community participant sample. A volunteer firefighter sample (n = 65) and a sample of similar community respondents (n = 103) completed a questionnaire study, including responses to the IES-R. The IES-R data from both groups were entered into a three-factor principal components analysis with direct oblimin rotation. We found further support for the validity of the IES-R when used with a community sample. However, our data suggested that when using the IES-R with a community sample, the choice between a two- and a three-factor model may depend on the composition of the participants. For volunteer firefighters, the factor-analytic structure of the IES-R appeared to be similar to that of the community sample, with more scatter in terms of item loadings. To our knowledge, there is no previous research considering the use of the IES-R with a strictly volunteer firefighter sample. In addition, despite adequate research on the factor-analytic structure of the original IES, little research has considered the factor-analytic structure of the more recent IES-R, even with community samples.     

When do employees cross boundaries? Individual and contextual determinants of career mobility

This study investigates the joint effects of individual characteristics and the labour market on career mobility. We propose that level of education, openness to experience, and a favourable labour market relate positively to employees crossing organizational, industrial, and occupational boundaries. Management programme alumni (N = 503) provided information through an online survey about their career histories, their level of education, and their openness to experience. Additionally, we used the unemployment rate as an indicator for yearly changes in the labour market. The results of our cross-classified multilevel analysis indicate that both individual characteristics and the labour market are determinants of career mobility. Level of education had a positive effect on organizational and industrial boundary crossing, and changes in the labour market related to organizational boundary crossing. Against our assumptions, openness to experience had no effect on career mobility, and none of the predictors were related to occupational boundary crossing. Our results demonstrate the importance of investigating career mobility from a boundary perspective combined with a focus on both individual and contextual characteristics. The dominance of education compared to personality and the difficulty of explaining occupational mobility open new research avenues and yield practical implications for employees, career counsellors, and organizations.     

A taxonomic foundation for evidence-based research on employee performance management

This study presents a taxonomic foundation for research on employee performance management practices based on a comprehensive review of the literature (198 articles and book chapters). The taxonomy consists of 50 practices organized within seven topic categories, including an evaluation of the amount of research evidence supporting each practice. This taxonomic foundation facilitates the aggregation, integration, interpretation, and explanation of performance management research based on a role-theoretic perspective derived from the behavioural approach to strategic human resource management. The proposed direct-linkage path model shows how this taxonomic foundation ties performance management practices to behaviours and results. We build on this Practice – Behaviour – Results (PBR) critical path model with moderators and mediators based on cognitive and social factors identified in the extensive previous research. This PBR model provides a foundation for orderly and structured growth for future research that will enhance the connection between research and improved organizational practices in performance management, as well as a guide to best practices in performance management.     

Failing safely: Increasing theology and religious studies students’ resilience and academic confidence via risk‐taking in formative assessment

Students increasingly appear anxious, risk‐averse, and worried about getting things “wrong.” They may appear to lack intellectual curiosity, and be unwilling to engage in independent study. This essay explores how teaching and assessment in theology and religious studies might help students learn to take intellectual risks, and increase their resilience. One approach is to encourage students to experiment and “fail safely,” to increase their confidence that they understand what is expected of them, and to help them begin to understand learning as more broadly formational, not always directed toward a grade. I suggest three strategies: more formative assessment; a stronger narrative about the purpose of formative assessment; and an appeal to values, virtue, and the cultivation of character. Via these approaches, students might be encouraged to understand assessment in less utilitarian terms and increase their resilience for a world characterized by volatility, uncertainty, complexity, and ambiguity, prepared both critically and dispositionally to thrive and contribute positively to society.