Methods Used to Document Procedural Fidelity in School-Based Intervention Research

There has been a paucity of guidance on the methodological details needed for measuring and sampling the independent variable or actual intervention occurrences in research and practice. Furthermore, the planning and support necessary to document the independent variable in both circumstances may be considerable. The current study extends prior research by reviewing the methods used to document procedural fidelity in school-based intervention research with student participants published between 2005 and 2012 in journals known for publishing school-related intervention studies. Two hundred sixty-six articles met the inclusion criteria and the majority (70 %) used fidelity data in the analysis of intervention effectiveness. By systematically examining contributions of school-based intervention researchers across targeted variables as well as qualities of interventions and settings, suggestions can be made for (a) planning measurement of fidelity across phases of the intervention, (b) sampling fidelity occurrences, (c) preventing intervention failure by providing needed supports, and (d) responding to patterns of fidelity required for intervention and outcome measurement in both research and practice.     

How to launch a national Internet-based panel study quickly: lessons from studying how American are coping with the tragedy of September 11, 2001?

This article reports on the planning, development, and implementation of a large national Internet-based panel study of how Americans are coping with the terrorist attacks of September 11, 2001. The study was designed to determine predictors and correlates of risk and resilience, both cross-sectionally and longitudinally. In order to acquire timely and meaningful data, we developed/adapted an extensive set of measures, obtained human subjects approval, and posted a research Web site just 17 days after the attacks. This article describes the major hurdles we confronted and the guidelines we recommend regarding these topics, including the methodological trade-offs inherent in Internet-based research, information technology requirements and tribulations, human subjects issues, selection of measures and securing permission for their use, and the challenges of participant recruitment. We also discuss issues that we did not anticipate, including the survey intervention. We focus not on findings, but on the concrete procedural, administrative, technical, and scientific challenges we encountered and the solutions we devised under considerable time and resource pressures.     

Personality Research and Assessment in the Era of Machine Learning

The increasing availability of high-dimensional, fine-grained data about human behaviour, gathered from mobile sensing studies and in the form of digital footprints, is poised to drastically alter the way personality psychologists perform research and undertake personality assessment. These new kinds and quantities of data raise important questions about how to analyse the data and interpret the results appropriately. Machine learning models are well suited to these kinds of data, allowing researchers to model highly complex relationships and to evaluate the generalizability and robustness of their results using resampling methods. The correct usage of machine learning models requires specialized methodological training that considers issues specific to this type of modelling. Here, we first provide a brief overview of past studies using machine learning in personality psychology. Second, we illustrate the main challenges that researchers face when building, interpreting, and validating machine learning models. Third, we discuss the evaluation of personality scales, derived using machine learning methods. Fourth, we highlight some key issues that arise from the use of latent variables in the modelling process. We conclude with an outlook on the future role of machine learning models in personality research and assessment.     

Ambulatory assessment for physical activity research: State of the science,best practices and future directions

Technological and digital progress benefits physical activity (PA) research. Here we compiled expert knowledge on how Ambulatory Assessment (AA) is utilized to advance PA research, i.e., we present results of the 2nd International CAPA Workshop 2019 “Physical Activity Assessment – State of the Science, Best Practices, Future Directions” where invited researchers with experience in PA assessment, evaluation, technology and application participated. First, we provide readers with the state of the AA science, then we give best practice recommendations on how to measure PA via AA and shed light on methodological frontiers, and we furthermore discuss future directions. AA encompasses a class of methods that allows the study of PA and its behavioral, biological and physiological correlates as they unfold in everyday life. AA includes monitoring of movement (e.g., via accelerometry), physiological function (e.g., via mobile electrocardiogram), contextual information (e.g., via geolocation-tracking), and ecological momentary assessment (EMA; e.g., electronic diaries) to capture self-reported information. The strengths of AA are data assessments near real-time, which minimize retrospective biases in real-world settings, consequentially enabling ecological valid findings. Importantly, AA enables multiple assessments across time within subjects resulting in intensive longitudinal data (ILD), which allows unraveling within-person determinants of PA in everyday life. In this paper, we show how AA methods such as triggered e-diaries and geolocation-tracking can be used to measure PA and its correlates, and furthermore how these findings may translate into real-life interventions. In sum, AA provides numerous possibilities for PA research, especially the opportunity to tackle within-subject antecedents, concomitants, and consequences of PA as they unfold in everyday life. In-depth insights on determinants of PA could help us design and deliver impactful interventions in real-world contexts, thus enabling us to solve critical health issues in the 21st century such as insufficient PA and high levels of sedentary behavior.     

Advancements in Procedural Fidelity Assessment and Intervention: Introduction to the Special Issue

Interest in procedural fidelity has grown rapidly since 1981 as evidenced by the growing numbers of research publications across disciplines on this topic. As a result, the past 30 years of research has yielded a variety of procedures to guide research and practice, which we hope translates into better educational practices and services. Despite our progress, much remains unknown. As an introduction to the special issue on advancements in procedural fidelity, this article will briefly describe the conceptual and definitional issues surrounding this topic, summarize the relevance of procedural fidelity to education and research, and highlight the articles in this issue that address proposed research priorities.     

Individual differences in ideological attitudes and prejudice: evidence from peer-report data

Our knowledge on the personality basis of ideological attitudes and prejudice, while based on a substantial body of research, suffers from a potentially serious methodological limitation: an overreliance on the method of self-reports. Across 2 studies (Ns = 193, 424), we examined associations between the Big Five personality dimensions, Right-Wing Authoritarianism (RWA), Social Dominance Orientation (SDO), and generalized prejudice, using both self-report and peer-report data stemming from 1 (Study 1) or 2 (Study 2) peer rater/s. Correlational and regression analyses as well as structural equation modeling showed that (a) the associations between personality dimensions, ideological attitudes, and prejudice were largely similar to previous research for both data sources; (b) RWA and prejudice showed a similar level of self-peer agreement to personality dimensions; (c) most of the known associations between personality, ideological attitudes, and prejudice were replicated also when measured by independent methods; (d) peer reports had some incremental validity in predicting ideological attitudes and prejudice; and (e) there was evidence that Openness to Experience and Agreeableness predicted prejudice directly and not only indirectly via RWA and SDO, respectively. Implications for the status of RWA, SDO, and prejudice as individual-difference constructs and for their bases in personality dimensions are discussed.     

The Rorschach and psychopathy: toward a more accurate understanding of the research findings

In this article we present the reader with an understandable essay on the relation between the Rorschach and psychopathy. Some degree of sophistication and applied knowledge of the Hare Psychopathy Checklist-Revised (Hare, 1991) and the Rorschach (Rorschach, 1921/1942) are necessary to wade through the literature, weigh the relative merits of arguments made by proponents and detractors of Rorschach assessment, and meaningfully interpret the findings of relevant studies. Often studies reviewing the Rorschach's utility in assessing Antisocial Personality Disorder (American Psychiatric Association, 1994) and psychopathy exhibit a flawed or superficial understanding of essential theoretical and methodological issues. Argument derived from a suspect or specious premise, such as the notion that the Rorschach was designed or aspires to correspond with formal diagnosis based on the Diagnostic and Statistical Manual of Mental Disorders (American Psychiatric Association, 1952, 1994), vitiates conclusions based on such a premise. In this article, we discuss theoretical and methodological issues that can aid the reader or reviewer in achieving a more accurate understanding of this body of research.     

How to Measure and Explain Achievement Change in Large-Scale Assessments: A Rejoinder

In this rejoinder, we discuss substantive and methodological validity issues of large-scale assessments of trends in student achievement, commenting on the discussion paper by Van den Heuvel-Panhuizen, Robitzsch, Treffers, and Köller (2009). We focus on methodological challenges in deciding what to measure, how to measure it, and how to foster stability. Next, we discuss what to do with trends that are found. Finally, we reflect on how the research findings were received.     

Interobserver agreement and procedural fidelity: An odd asymmetry

We examined articles with experiments published in the Journal of Applied Behavior Analysis and in Behavior Analysis in Practice from 2017 through 2021 to determine how frequently procedural fidelity was assessed. When procedural fidelity was assessed, we determined how often a measure of interobserver agreement for those fidelity data was provided. We also determined how often a measure of interobserver agreement for participants' behavior was provided. Across both journals and all years, 54.7% of relevant articles provided a measure of procedural fidelity. Of them, 17.7% provided a measure of interobserver agreement for procedural fidelity. In marked contrast, 96.4% provided interobserver agreement data for participants' behavior. It is unfortunate that applied behavior analysts frequently fail to provide procedural fidelity data and, when they do, often fail to provide interobserver agreement data for the fidelity data. Reviewers for, and editors of, behavior-analytic journals are encouraged to strongly consider the relative value of procedural fidelity and agreement on procedural fidelity measures when rendering recommendations on the suitability of a given submission.     

The Phenomenological Film Collective: Introducing a cinematic‐phenomenological research method for social advocacy filmmaking

This paper introduces the vision for the Phenomenological Film Collective ( pfcollective.com ), a research and filmmaking group which utilizes phenomenological research in the service of social advocacy filmmaking. I outline the theoretical and methodological underpinnings of developing a “cinematic‐phenomenological research method” for PFCollective in order to illuminate lived experiences of sociocultural oppression for public viewership via cinematography. The roots of cinematic phenomenology are situated within the theoretical framework of liberation psychology, which calls on psychologists to pursue research that can facilitate consciousness‐raising and social justice across society. The paper provides a methodological overview of existential‐hermeneutic phenomenological research and discusses how its research findings can be disseminated to the widespread public in artistic formats. I demonstrate how filmmaking is an appropriate aesthetic language through which to disseminate phenomenological research about lived experiences of oppression, grounding this proposition in phenomenological philosophy and liberation psychology. I outline the procedural steps for conducting cinematic‐phenomenological research to produce phenomenological films about oppressive sociocultural phenomena. The paper concludes with suggestions for academic disciplines to become more interdisciplinary in our collective pursuit for social advocacy and change.     

Theory in Child Psychosocial Treatment Research: Have It or Had It? A Pragmatic Alternative

In the area of child psychosocial intervention research, substantial progress in addressing challenging methodological issues has been made. In this article we suggest that continued movement forward requires maintaining a balance between methodological and theoretical issues, and that it would be advantageous to begin to think about matching advances in methodological sophistication by similar advances in theoretical sophistication. The need to advance our understanding of theoretical issues is especially important in light of the challenges raised by postmodernism, particularly as these challenges relate to knowledge development. We discuss these challenges and describe how a pragmatic orientation may be a useful way to respond to them. We also describe how a pragmatic orientation has been useful in our work in developing, implementing and evaluating psychosocial interventions for children with phobic and anxiety disorders.     

IMPROVING INTERVENTION INTEGRITY OF DIRECT‐SERVICE PRACTITIONERS THROUGH PERFORMANCE FEEDBACK AND PROBLEM SOLVING CONSULTATION

The purpose of this study was to improve intervention integrity of behavior support plan implementation by teachers at a specialized school. During the first phase of a multiple baseline design, performance feedback was given to the teachers during (correction) and immediately following (praise) classroom observations. In a second phase, the teachers received problem solving consultation that addressed behavior support plan misapplication and respective procedural modifications. Only one of the three teachers demonstrated improved intervention integrity with performance feedback, whereas problem solving consultation produced uniform improvement. We discuss these findings and implications for intervention integrity assessment and research. Copyright © 2014 John Wiley & Sons, Ltd.     

Effects of social dominance orientation and right‐wing authoritarianism on corrupt intention: The role of moral outrage

Previous research suggested that dominance orientation and authoritarianism may be associated with corruption, but little research has verified this assumption or uncovered its psychological processes. In this article, we examined empirically the relationships between social dominance orientation (SDO), right‐wing authoritarianism (RWA) and corrupt intention and explored the mediating role of moral outrage on these relationships. A total of 677 college students participated in the study and completed measures of SDO, RWA, moral outrage and corrupt intention. Our findings demonstrated that both SDO and RWA were positively associated with corrupt intention. Additionally, moral outrage partially mediated the relation between SDO and corrupt intention and fully mediated the relation between RWA and corrupt intention. Specifically, the results indicated that higher SDO or RWA was associated with reduced moral outrage and increased corrupt intention. This implies that the enhancement of morality and moral outrage may inhibit corrupt intention.     

Bayesian data analysis for newcomers

This article explains the foundational concepts of Bayesian data analysis using virtually no mathematical notation. Bayesian ideas already match your intuitions from everyday reasoning and from traditional data analysis. Simple examples of Bayesian data analysis are presented that illustrate how the information delivered by a Bayesian analysis can be directly interpreted. Bayesian approaches to null-value assessment are discussed. The article clarifies misconceptions about Bayesian methods that newcomers might have acquired elsewhere. We discuss prior distributions and explain how they are not a liability but an important asset. We discuss the relation of Bayesian data analysis to Bayesian models of mind, and we briefly discuss what methodological problems Bayesian data analysis is not meant to solve. After you have read this article, you should have a clear sense of how Bayesian data analysis works and the sort of information it delivers, and why that information is so intuitive and useful for drawing conclusions from data.     

Content Analysis��A Methodological Primer for Gender Research

This article is intended to serve as a primer on methodological standards for gender scholars pursuing content analytic research. The scientific underpinnings of the method are explored, including the roles of theory, past research, population definition, objectivity/intersubjectivity, reliability, validity, generalizability, and replicability. Both human coding and computer coding are considered. The typical process of human-coded content analysis is reviewed, including the steps of unitizing, sampling, measurement, coder training, reliability assessment, and reportage of methods. Numerous applications to research on gender roles and related issues are reviewed. Practical checklists are offered for content analysis preparation and methodological execution.     

Is early identification of children of alcoholics necessary for preventive intervention? Reaction to Havey and Dodd

Havey and Dodd (1995, this issue) address the question of how to best identify early adolescents who are at risk for later substance abuse, with a particular focus on children of alcoholics (COAs). They conclude that stressful family events are preferable to COA status as an at-risk indicator. In addition, they question the well-substantiated at-risk status of COAs, claiming support for the null hypothesis regarding individual and family differences between COAs and non-COAs. This article addresses both conceptual and methodological concerns about identification and intervention for COAs in general, and about the Havey and Dodd study in particular. I first discuss a theoretically-empirically based approach to identification and risk prevention that supports a more complex, less parsimonious, model than that suggested by Havey and Dodd. Second, I present research to support the at-risk status of COAs. Third, I discuss issues related to identification of COAs and question the validity of Havey and Dodd's identification techniques. On the basis of conceptual and methodological arguments, I contend that Havey and Dodd's conclusions are not well-founded in their own or others' research and that the at-risk status of COAs cannot be dismissed on the basis of their research. I also contend that early identification may not be necessary for intervention with COAs or other at-risk groups. Alternatively, primary prevention programs provide the context for both service delivery and identification of individuals who might not otherwise be easily identified, particularly those at risk for developing mental health problems because of family environment factors (e.g., family alcoholism or conflict), or those currently experiencing mild or internalized (e.g., depression) adjustment difficulties.     

The Analysis of Intervention Change in Children and Families: Methodological and Conceptual Issues Embedded in Intervention Studies

This article provides an overview of methodological and conceptual issues underlying the assessment of change during psychotherapy with children and families. Three central considerations are discussed: (1) What changes do we measure? (2) When do we measure change? (3) How do we measure change? The difficulties with the traditional methods of assessing change are highlighted, and suggestions for a new paradigm that calls for expansion of the nomological network relevant to the analysis of change, as well as the inclusion of more assessment points and more reliable and valid measures of the change process, are presented. Illustrations of key points are provided from the Adolescent Transitions Program, a theoretically based intervention program targeting high-risk youth.     

To whom does voice in groups matter? Effects of voice on affect and procedural fairness judgments as a function of social dominance orientation

The authors investigated the effects of voice--the opportunity to provide input in decision-making processes--on perceptions of procedural fairness. In particular, the authors studied the moderating role of social dominance orientation (SDO) in shaping this relation. SDO is an important individual differences variable that causes people to favor unequal relationships within and between social groups. Results revealed that voice was more strongly related to fairness judgments when participants had a high rather than low SDO. Moreover, positive affect mediated this moderation effect. The authors interpreted these results to indicate that high-SDO participants were especially sensitive to voice manipulations because such manipulations enhance perceptions of control over group resources and outcomes. The authors conclude by discussing alternative explanations based on other fairness theories.     

Social dominance theory and the dynamics of intergroup relations: Taking stock and looking forward

This chapter reviews the last 15 years of research inspired by social dominance theory, a general theory of societal group-based inequality. In doing so, we sketch the broad outlines of the theory and discuss some of the controversies surrounding it, such as the “invariance hypothesis” regarding gender differences in social dominance orientation (SDO) and the effect of social context on the expression of SDO. We also discuss the central role of gender in the construction and maintenance of group-based inequality, and review some of the new research inspired by the social dominance perspective. Finally, we identify and discuss some of the most important theoretical questions posed by social dominance theory that are yet to be researched.