Generalizability theory: A practical guide to study design,implementation, and interpretation

Generalizability Theory (GT) offers increased utility for assessment research given the ability to concurrently examine multiple sources of variance, inform both relative and absolute decision making, and determine both the consistency and generalizability of results. Despite these strengths, assessment researchers within the fields of education and psychology have been slow to adopt and utilize a GT approach. This underutilization may be due to an incomplete understanding of the conceptual underpinnings of GT, the actual steps involved in designing and implementing generalizability studies, or some combination of both issues. The goal of the current article is therefore two-fold: (a) to provide readers with the conceptual background and terminology related to the use of GT and (b) to facilitate understanding of the range of issues that need to be considered in the design, implementation, and interpretation of generalizability and dependability studies. Given the relevance of this analytic approach to applied assessment contexts, there exists a need to ensure that GT is both accessible to, and understood by, researchers in education and psychology. Important methodological and analytical considerations are presented and implications for applied use are described.     

Assessing vividness of mental imagery: The Plymouth Sensory Imagery Questionnaire

Mental imagery may occur in any sensory modality, although visual imagery has been most studied. A sensitive measure of the vividness of imagery across a range of modalities is needed: the shorter version of Bett's Questionnaire upon Mental Imagery (Sheehan, 1967 , J. Clin. Psychology, 23, 386) uses outdated items and has an unreliable factor structure. We report the development and initial validation of the Plymouth Sensory Imagery Questionnaire (Psi‐Q) comprising items for each of the following modalities: Vision, Sound, Smell, Taste, Touch, Bodily Sensation, and Emotional Feeling. An exploratory factor analysis on a 35‐item form indicated that these modalities formed separate factors, rather than a single imagery factor, and this was replicated by confirmatory factor analysis. The Psi‐Q was validated against the Spontaneous Use of Imagery Scale (Reisberg et al., 2003 , Appl. Cogn. Psychology, 17, 147) and Marks' ( 1995 , J. Mental Imagery, 19, 153) Vividness of Visual Imagery Questionnaire‐2 (VVIQ‐2). A short 21‐item form comprising the best three items from the seven factors correlated with the total score and subscales of the full form, and with the VVIQ‐2. Inspection of the data shows that while visual and sound imagery is most often rated as vivid, individuals who rate one modality as strong and the other as weak are not uncommon. Findings are interpreted within a working memory framework and point to the need for further research to identify the specific cognitive processes underlying the vividness of imagery across sensory modalities.     

Axis I anxiety and mental health disorders among stuttering adolescents

PurposeThe purpose of this study was to evaluate anxiety and psychological functioning among adolescents seeking speech therapy for stuttering using a structured, diagnostic interview and psychological questionnaires. This study also sought to determine whether any differences in psychological status were evident between younger and older adolescents.MethodParticipants were 37 stuttering adolescents seeking stuttering treatment. We administered the Computerized Voice Version of the Diagnostic Interview Schedule for Children, and five psychometric tests. Participants were classified into younger (12–14 years; n = 20) and older adolescents (15–17 years; n = 17).ResultsThirty-eight percent of participants attained at least one diagnosis of a mental disorder, according to the diagnostic criteria of the Diagnostic and Statistical Manual of Mental Disorders, Fourth Edition (DSM-IV; APA, 2000), with the majority of these diagnoses involving anxiety. This figure is double current estimates for general adolescent populations, and is consistent with our finding of moderate and moderate–severe quality of life impairment. Although many of the scores on psychological measures fell within the normal range, older adolescents (15–17 years) reported significantly higher anxiety, depression, reactions to stuttering, and emotional/behavioral problems, than younger adolescents (12–14 years). There was scant evidence that self-reported stuttering severity is correlated with mental health issues. There are good reasons to believe these results are conservative because many participants gave socially desirable responses about their mental health status.DiscussionThese results reveal a need for large-scale, statistically powerful assessments of anxiety and other mental disorders among stuttering adolescents with reference to control populations.Educational Objectives: The reader will be able to: (a) explain the clinical importance of assessing for mental health with stuttering adolescents, (b) state the superior method for adolescent mental health assessment and (c) state a major issue with determining the genuineness of stuttering adolescent responses to psychological assessment.     

Responsabilidad social universitaria: influencia de valores y empatía en la autoatribución de comportamientos socialmente responsables

Training socially responsible professionals at university level has become a UNESCO-recognized mission. This paper reviews the study of Academic Social Responsibility from the Psychology standpoint, and develops a novel analysis approach for assessing the impact of higher education in acquiring social responsibility behaviors, based on a MIMIC analysis. Three scales were administered, based on an accidental non-probabilistic sampling and a transversal survey research: human values, multidimensional empathy and self-attribution of socially responsible behaviors. The sample included 860 students from Ibero-American Universities. The results show that university students display a high frequency of socially responsible behaviors but not a more prosocial intentionality in these behaviors. As regards frequency scale of socially responsible behavior, influence is evident of value dimensions, such as conservation and openness to change with the subscales of empathy perspective taking and personal distress. Regarding the intentionality scale, the variables that contribute to the self-attribution of socially responsible behavior are value dimensions: conservation and self-transcendence, along with the subscales of empathy appertaining to fantasy. Conclusion This research provides information on the psychological variables that have implications for humans to be socially responsible, which will allow the University to respond to the impact of training qualified professionals with great respect to their duty in the society.     

Criterion Validity is Maintained When Items are Evaluatively Neutralized: Evidence from a Full‐scale Five‐factor Model Inventory

The original version and an evaluatively neutralized version (with items rephrased to reduce popularity) of a personality inventory were compared. The results revealed (i) similar criterion validity across three different sets of self‐rated behaviours, (ii) stronger relations to the rated social desirability of criteria for the original version and (iii) less correlation between factors for the neutralized version. We take the results to indicate that evaluative neutralization is a viable technique for reducing social desirability in self‐ratings. Implications for test construction are discussed. Copyright © 2014 European Association of Personality Psychology     

Exploring Differential Subgroup Functioning on SAT Writing Items: What Happens When English Is Not a Test Taker's Best Language?

The purpose of this study is to explore patterns in model-data fit related to subgroups of test takers from a large-scale writing assessment. Using data from the SAT, a calibration group was randomly selected to represent test takers who reported that English was their best language from the total population of test takers (N = 322,011). A reference scale for the items was constructed based on EBL responses. Response behaviors of test takers who reported that English was not their best language (ENBL) were examined in relationship to this reference scale. This study illustrates the use of differential subgroup analyses to identify patterns related to person misfit within subgroups, as well as subsets of items, that may affect the validity of writing scores for ENBL test takers. The methodology described here offers an approach that can be used to explore, understand, and improve the validity of scores obtained from ENBL test takers in large-scale writing assessments.     

An Initial Investigation of the Factor-Analytic Structure of the Impact of Event Scale-Revised With a Volunteer Firefighter Sample

We provide an initial evaluation of the factor structure of the Impact of Event Scale-Revised (IES-R) when used with a volunteer firefighter and a similar community participant sample. A volunteer firefighter sample (n = 65) and a sample of similar community respondents (n = 103) completed a questionnaire study, including responses to the IES-R. The IES-R data from both groups were entered into a three-factor principal components analysis with direct oblimin rotation. We found further support for the validity of the IES-R when used with a community sample. However, our data suggested that when using the IES-R with a community sample, the choice between a two- and a three-factor model may depend on the composition of the participants. For volunteer firefighters, the factor-analytic structure of the IES-R appeared to be similar to that of the community sample, with more scatter in terms of item loadings. To our knowledge, there is no previous research considering the use of the IES-R with a strictly volunteer firefighter sample. In addition, despite adequate research on the factor-analytic structure of the original IES, little research has considered the factor-analytic structure of the more recent IES-R, even with community samples.     

The “impact factor style of thinking”: A new theoretical framework

Over the last few years, university policy in developed countries has placed too much emphasis on assessing publications on the basis of the impact factor. Any scientific material that has not been published in a journal indexed in the Journal Citation Reports is academically irrelevant. Hence, researchers strive hard to publish in such journals. Failure to achieve this means not having a successful career or receiving any academic recognition. A thorough analysis of the effects of this phenomenon on the researcher community leads to the conclusion that the overestimation of the impact factor is likely to cause disappointment among many researchers. Researchers only have two options: adopting a politically correct behavior, that is, following the “impact factor style of thinking”, or accepting the futility of their research regardless of its relevance. This is an example of the involvement of university policy habitus obsessed impact index. O policy impact index is accepted, or is doomed to academic ostracism, and exclusion from participation in all university policy.     

Dispositional Mindfulness,Posttraumatic Stress Disorder Symptoms and Academic Burnout in Chinese Adolescents Following a Tornado: The Role of Mediation through Regulatory Emotional Self-Efficacy

Limited research has been conducted on dispositional mindfulness, posttraumatic stress disorder (PTSD) symptoms and academic burnout in Chinese adolescents following a tornado. The present study investigated the ways in which dispositional mindfulness is related to PTSD symptoms and academic burnout in Chinese adolescents following a tornado by considering the role of regulatory emotional self-efficacy. A total of 431 Chinese adolescents (mean age: 14.75 years) who had experienced a severe tornado 9 months prior to this study were recruited for this study. The results indicated that our model fit the data well [χ²/df = 2.774, CFI = 0.952, TLI = 0.934, RMSEA (90% CI) = 0.064 (0.051–0.077)], and revealed that regulatory emotional self-efficacy partially mediates the relationships between dispositional mindfulness and PTSD symptoms and academic burnout, respectively. The clinical implications and limitations of our research, and recommendations for future research, are discussed in this paper.