Generalizability theory: A practical guide to study design,implementation, and interpretation

Generalizability Theory (GT) offers increased utility for assessment research given the ability to concurrently examine multiple sources of variance, inform both relative and absolute decision making, and determine both the consistency and generalizability of results. Despite these strengths, assessment researchers within the fields of education and psychology have been slow to adopt and utilize a GT approach. This underutilization may be due to an incomplete understanding of the conceptual underpinnings of GT, the actual steps involved in designing and implementing generalizability studies, or some combination of both issues. The goal of the current article is therefore two-fold: (a) to provide readers with the conceptual background and terminology related to the use of GT and (b) to facilitate understanding of the range of issues that need to be considered in the design, implementation, and interpretation of generalizability and dependability studies. Given the relevance of this analytic approach to applied assessment contexts, there exists a need to ensure that GT is both accessible to, and understood by, researchers in education and psychology. Important methodological and analytical considerations are presented and implications for applied use are described.     

Assessing vividness of mental imagery: The Plymouth Sensory Imagery Questionnaire

Mental imagery may occur in any sensory modality, although visual imagery has been most studied. A sensitive measure of the vividness of imagery across a range of modalities is needed: the shorter version of Bett's Questionnaire upon Mental Imagery (Sheehan, 1967 , J. Clin. Psychology, 23, 386) uses outdated items and has an unreliable factor structure. We report the development and initial validation of the Plymouth Sensory Imagery Questionnaire (Psi‐Q) comprising items for each of the following modalities: Vision, Sound, Smell, Taste, Touch, Bodily Sensation, and Emotional Feeling. An exploratory factor analysis on a 35‐item form indicated that these modalities formed separate factors, rather than a single imagery factor, and this was replicated by confirmatory factor analysis. The Psi‐Q was validated against the Spontaneous Use of Imagery Scale (Reisberg et al., 2003 , Appl. Cogn. Psychology, 17, 147) and Marks' ( 1995 , J. Mental Imagery, 19, 153) Vividness of Visual Imagery Questionnaire‐2 (VVIQ‐2). A short 21‐item form comprising the best three items from the seven factors correlated with the total score and subscales of the full form, and with the VVIQ‐2. Inspection of the data shows that while visual and sound imagery is most often rated as vivid, individuals who rate one modality as strong and the other as weak are not uncommon. Findings are interpreted within a working memory framework and point to the need for further research to identify the specific cognitive processes underlying the vividness of imagery across sensory modalities.     

Criterion Validity is Maintained When Items are Evaluatively Neutralized: Evidence from a Full‐scale Five‐factor Model Inventory

The original version and an evaluatively neutralized version (with items rephrased to reduce popularity) of a personality inventory were compared. The results revealed (i) similar criterion validity across three different sets of self‐rated behaviours, (ii) stronger relations to the rated social desirability of criteria for the original version and (iii) less correlation between factors for the neutralized version. We take the results to indicate that evaluative neutralization is a viable technique for reducing social desirability in self‐ratings. Implications for test construction are discussed. Copyright © 2014 European Association of Personality Psychology     

Exploring Differential Subgroup Functioning on SAT Writing Items: What Happens When English Is Not a Test Taker's Best Language?

The purpose of this study is to explore patterns in model-data fit related to subgroups of test takers from a large-scale writing assessment. Using data from the SAT, a calibration group was randomly selected to represent test takers who reported that English was their best language from the total population of test takers (N = 322,011). A reference scale for the items was constructed based on EBL responses. Response behaviors of test takers who reported that English was not their best language (ENBL) were examined in relationship to this reference scale. This study illustrates the use of differential subgroup analyses to identify patterns related to person misfit within subgroups, as well as subsets of items, that may affect the validity of writing scores for ENBL test takers. The methodology described here offers an approach that can be used to explore, understand, and improve the validity of scores obtained from ENBL test takers in large-scale writing assessments.     

An Initial Investigation of the Factor-Analytic Structure of the Impact of Event Scale-Revised With a Volunteer Firefighter Sample

We provide an initial evaluation of the factor structure of the Impact of Event Scale-Revised (IES-R) when used with a volunteer firefighter and a similar community participant sample. A volunteer firefighter sample (n = 65) and a sample of similar community respondents (n = 103) completed a questionnaire study, including responses to the IES-R. The IES-R data from both groups were entered into a three-factor principal components analysis with direct oblimin rotation. We found further support for the validity of the IES-R when used with a community sample. However, our data suggested that when using the IES-R with a community sample, the choice between a two- and a three-factor model may depend on the composition of the participants. For volunteer firefighters, the factor-analytic structure of the IES-R appeared to be similar to that of the community sample, with more scatter in terms of item loadings. To our knowledge, there is no previous research considering the use of the IES-R with a strictly volunteer firefighter sample. In addition, despite adequate research on the factor-analytic structure of the original IES, little research has considered the factor-analytic structure of the more recent IES-R, even with community samples.     

Assessment of Parenting Practices Related to Conduct Problems: Development and Validation of the Management of Children's Behavior Scale

We report on the development and initial validation of the parent-report scale, Management of Children's Behavior Scale (MCBS), designed to evaluate parenting practices related to conduct problems in children. Children (N=396, ages 2-14) referred for outpatient treatment and their parents served as participants. We evaluated the composition and consistency of the scale and provided evidence pertaining to concurrent, predictive, and incremental validity. Evidence for each type of validity was consistent with the conceptualization of the scale and the pertinence to child conduct problems. The measure also was sensitive to therapeutic changes. Parenting practices targeted in treatment (parent management training) improved as predicted over time. The results suggest the measure may be useful in evaluating parenting practices known to relate to conduct problems and often targeted for intervention in parent- and family-based treatment.     

A taxonomic foundation for evidence-based research on employee performance management

This study presents a taxonomic foundation for research on employee performance management practices based on a comprehensive review of the literature (198 articles and book chapters). The taxonomy consists of 50 practices organized within seven topic categories, including an evaluation of the amount of research evidence supporting each practice. This taxonomic foundation facilitates the aggregation, integration, interpretation, and explanation of performance management research based on a role-theoretic perspective derived from the behavioural approach to strategic human resource management. The proposed direct-linkage path model shows how this taxonomic foundation ties performance management practices to behaviours and results. We build on this Practice – Behaviour – Results (PBR) critical path model with moderators and mediators based on cognitive and social factors identified in the extensive previous research. This PBR model provides a foundation for orderly and structured growth for future research that will enhance the connection between research and improved organizational practices in performance management, as well as a guide to best practices in performance management.     

Failing safely: Increasing theology and religious studies students’ resilience and academic confidence via risk‐taking in formative assessment

Students increasingly appear anxious, risk‐averse, and worried about getting things “wrong.” They may appear to lack intellectual curiosity, and be unwilling to engage in independent study. This essay explores how teaching and assessment in theology and religious studies might help students learn to take intellectual risks, and increase their resilience. One approach is to encourage students to experiment and “fail safely,” to increase their confidence that they understand what is expected of them, and to help them begin to understand learning as more broadly formational, not always directed toward a grade. I suggest three strategies: more formative assessment; a stronger narrative about the purpose of formative assessment; and an appeal to values, virtue, and the cultivation of character. Via these approaches, students might be encouraged to understand assessment in less utilitarian terms and increase their resilience for a world characterized by volatility, uncertainty, complexity, and ambiguity, prepared both critically and dispositionally to thrive and contribute positively to society.     

Sexual‐perception processes in acquaintance‐targeted sexual aggression

This study analyzes data from seven published studies to examine whether three performance‐based indices of men's misperception of women's sexual interest (MSI), derived from a self‐report questionnaire, are associated with sexual‐aggression history, rape‐supportive attitudes, sociosexuality, problem drinking, and self‐reported MSI. Almost 2000 undergraduate men judged the justifiability of a man's increasingly unwanted advances toward a woman on the Heterosocial Perception Survey‐Revised. Participants self‐reported any sexual‐aggression history, and some completed questionnaires assessing rape‐supportive attitudes, sociosexuality, problem drinking, and self‐reported MSI. A three‐parameter logistic function was fitted to participants’ justifiability ratings within a non‐linear mixed‐effects framework, which provided precise participant‐specific estimates of three sexual‐perception processes (baseline justifiability, bias, and sensitivity). Sexual‐aggression history and rape‐supportive attitudes predicted: (a) reduced sensitivity to women's affect; (b) more liberal biases, such that the woman's affect had to be more negative before justifiability ratings dropped substantially; and (c) greater baseline justifiability of continued advances after a positive response. Sexual‐aggression history and attitudes correlated more strongly with sensitivity than baseline justifiability; remaining variables showed the opposite pattern. This work underscores the role of sexual‐perception processes in sexual aggression and illustrates the derivation of performance‐based estimates of sexual‐perception processes from questionnaire responses.