Ecological Validity as a Key Feature of External Validity in Research on Human Development

This issue addresses the topic of ecological validity in research in human development across the life span. Although recent developments in study design, data collection methods, and data analysis techniques have greatly advanced researchers’ ability to collect large amounts of data on large groups of individuals in natural settings, it is important to approach these data with a reflected understanding of their ecological validity. Just because data were collected using everyday and familiar stimuli or ecological momentary assessment methods does not mean automatically that the ecological validity of these data is guaranteed and can go unquestioned.     

An Ecological Momentary Assessment Study Investigating the Function of Hoarding

This study examined the function of hoarding behaviors and the relations between hoarding and a series of cognitive and affective processes in the moment using ecological momentary assessment. A matched-groups design was used to compare college students with higher hoarding symptoms (n = 31) and matched controls (n = 29). The two groups did not differ in what function they reported acquiring served, and positive automatic reinforcement was the most commonly reported function in both groups. Engaging in hoarding-relevant behaviors did not predict change in positive or negative affect when controlling for previous affect. Emotional reactivity and experiential avoidance in the moment were both elevated in the higher hoarding group compared to controls, while momentary mindfulness and negative affect differentiation were lower. Overall, these findings support the importance of emotion regulation processes in hoarding. They also suggest individuals may not be successfully regulating affect in the moment with hoarding behaviors, despite efforts to do so. It may be useful to evaluate processes such as striving for positive affect in hoarding disorder in the future.     

DRO contingencies: an analysis of variable-momentary schedules.

We conducted several comparative analyses to determine the relative effectiveness of variable-momentary differential-reinforcement-of-other-behavior (VM DRO) schedules. Three individuals who had been diagnosed with mental retardation participated. Results of functional analyses indicated that their self-injurious behavior (SIB) was maintained by social-positive reinforcement. Two individuals participated in a two-stage comparative analysis within multielement and multiple baseline designs. Fixed-interval (FI) and variable-interval (VI) DRO were compared in the first stage; VI DRO and VM DRO were compared in the second. All three schedules effectively reduced the participants' SIB. Treatment for the 3rd individual was conducted in a reversal design to examine the effects of VM DRO when it was implemented in isolation, and results indicated that the procedure was effective in reducing SIB. These findings suggest that VM DRO schedules may represent attractive alternatives to traditional FI schedules because momentary schedules do not require continuous monitoring and may result in higher rates of reinforcement.     

Modeling the Ecological Validity of Neurocognitive Assessment in Adults with Acquired Brain Injury

Neuropsychologists are increasingly asked to make judgments regarding treatment options and rehabilitation strategies in addition to evaluating the degree and scope of neuropsychological impairment following acquired brain injuries. The capacity to make informed clinical decisions relies upon research investigating the relationships between neuropsychological and psychosocial status (i.e., ecological validity). Unfortunately, much of this research employs exploratory analyses, an approach that can lead to theoretical ambiguity and ad-hoc interpretations. The current availability and accessibility of analytical tools, like structural equation modeling (SEM), however, permits the testing of specific hypotheses regarding ecological validity and promotes a-priori theory development. In the current study, a theory-driven model of the ecological validity of a neurocognitive assessment was tested against data obtained from individuals with acquired brain injury using SEM. The results provide confirmatory evidence for the ecological validity of neurocognitive constructs and empirical support for a theory-driven analytical approach to ecological validity research.     

A slice of life: Ecologically valid methods for research on social relationships and health across the life span

This paper describes “slice of life” methods—detailed observations of moment‐to‐moment intraindividual processes that occur over short timescales in the natural environment—as an important research tool in exploring social relationships and health across the life span. Such methods, including ecological momentary assessment, experience sampling, daily diaries, measurement burst, and other slice of life designs, are important supplements to more traditional laboratory, epidemiological, and long‐term longitudinal research methods. Specifically, we highlight how slice of life methods (with a focus on ecological momentary assessment) allow researchers to compare the relative importance of different social relationships (or features thereof) for health outcomes, understand temporal dynamics, identify potentially causal mechanistic pathways at different timescales through which social relationships influence health, and design, implement, and evaluate innovative interventions that target these pathways at different timescales. We conclude with recommendations and future directions for slice of life methods for research on social relationships and health across the life span.     

Conceptualizing and Estimating Process Speed in Studies Employing Ecological Momentary Assessment Designs: A Multilevel Variance Decomposition Approach

Researchers have been making use of ecological momentary assessment (EMA) and other study designs that sample feelings and behaviors in real time and in naturalistic settings to study temporal dynamics and contextual factors of a wide variety of psychological, physiological, and behavioral processes. As EMA designs become more widespread, questions are arising about the frequency of data sampling, with direct implications for participants' burden and researchers' ability to capture and study dynamic processes. Traditionally, spectral analytic techniques are used for time series data to identify process speed. However, the nature of EMA data, often collected with fewer than 100 measurements per person, sampled at randomly spaced intervals, and replete with planned and unplanned missingness, precludes application of traditional spectral analytic techniques. Building on principles of variance partitioning used in the generalizability theory of measurement and spectral analysis, we illustrate the utility of multilevel variance decompositions for isolating process speed in EMA-type data. Simulation and empirical data from a smoking-cessation study are used to demonstrate the method and to evaluate the process speed of smoking urges and quitting self-efficacy. Results of the multilevel variance decomposition approach can inform process-oriented theory and future EMA study designs.     

Conceptualizing and Estimating Process Speed in Studies Employing Ecological Momentary Assessment Designs: A Multilevel Variance Decomposition Approach

Researchers have been making use of ecological momentary assessment (EMA) and other study designs that sample feelings and behaviors in real time and in naturalistic settings to study temporal dynamics and contextual factors of a wide variety of psychological, physiological, and behavioral processes. As EMA designs become more widespread, questions are arising about the frequency of data sampling, with direct implications for participants' burden and researchers' ability to capture and study dynamic processes. Traditionally, spectral analytic techniques are used for time series data to identify process speed. However, the nature of EMA data, often collected with fewer than 100 measurements per person, sampled at randomly spaced intervals, and replete with planned and unplanned missingness, precludes application of traditional spectral analytic techniques. Building on principles of variance partitioning used in the generalizability theory of measurement and spectral analysis, we illustrate the utility of multilevel variance decompositions for isolating process speed in EMA-type data. Simulation and empirical data from a smoking-cessation study are used to demonstrate the method and to evaluate the process speed of smoking urges and quitting self-efficacy. Results of the multilevel variance decomposition approach can inform process-oriented theory and future EMA study designs.     

An Ecological Momentary Intervention Study of Emotional Responses to Smartphone-Prompted CBT Skills Practice and the Relationship to Clinical Outcomes

The practice of therapeutic skills outside of sessions in which they are learned is one presumed key component of cognitive behavioral therapy (CBT). Yet, our understanding of how skills practice relates to clinical outcomes remains limited. Here, we explored patients’ emotional responses to CBT skills practice in a pilot study pairing smartphone-app-delivered skills reminders and guided practice (ecological momentary intervention [EMI]) using ecological momentary assessment (EMA). Participants (n = 25) were adults recently hospitalized for a suicide attempt or severe suicidal thinking. They received brief inpatient CBT (1 to 3 sessions covering core CBT skills from the Unified Protocol), followed by 1 month of EMI and EMA after discharge. On average, participants reported modest reductions in negative affect after skills use (i.e., immediate responses; median time elapsed = 4.30 minutes). Additionally, participants tended to report less negative affect when the timepoint preceding the current assessment included EMI skills practice, rather than EMA alone (i.e., delayed responses; median time elapsed between prompts = 2.17 hours). Immediate effects were unrelated to longer-term clinical outcomes, whereas greater delayed effects were associated with lower symptom severity at follow-up. Future studies should further examine how CBT skills use in daily life may alleviate symptoms.     

Precision of Curriculum-Based Measurement Reading Data: Considerations for Multiple-Baseline Designs

Single-case designs provide an established technology for evaluating the effects of academic interventions. Researchers interested in studying the long-term effects of reading interventions often use curriculum-based measures of reading (CBM-R) as they possess many of the desirable characteristics for use in a time-series design. The reliability of CBM-R scores is often supported by research from group designs, but making idiographic interpretations regarding the change in a student’s oral reading rate requires attention to the precision of static scores and growth estimates. The purpose of this paper is twofold. First, we discuss how recent empirical work on the technical adequacy of CBM-R scores has revealed multiple threats to the data-evaluation validity when CBM-R passages are used to measure oral reading rate. Second, we identify pertinent considerations for conducting a visual analysis of intervention effects based on CBM-R data. We conclude with a brief discussion of implications for researchers considering the use of CBM-R within multiple-baseline designs.     

PTSD Symptom Cluster Severity Predicts Momentary Emotion Dynamics During Couple Conversations

This study investigated the associations between momentary emotion dynamics and posttraumatic stress disorder (PTSD) symptoms. Using a sample of 61 couples (N = 122 individuals) in which all individuals were trauma exposed and at least one partner screened positive for PTSD, we examined the intra- and interpersonal regulation of vocally encoded emotional arousal (fundamental frequency [f₀]) and how these momentary emotion regulatory patterns relate to specific PTSD symptoms during two couple conversations: one designed to elicit conflict and one to elicit intimacy. PTSD symptoms were assessed using a gold standard clinical interview. In both conversations, higher reexperiencing symptoms were associated with greater emotional inertia (i.e., more resistance to change in emotional state following deviation from one’s emotional equilibrium), and higher avoidance symptoms were associated with less emotional inertia (i.e., quicker return to emotional equilibrium). In the intimacy conversations, individuals also responded to their partners’ arousal. Furthermore, individuals whose partners exhibited higher emotional numbing symptoms exhibited more emotional inertia, suggesting that emotion regulation may be a function of both one’s own and one’s partner’s PTSD symptoms. Attending to the interpersonal context of emotion dynamics during PTSD treatment may enhance outcomes.     

A probabilistic model of intensive designs

Without internal validity, experimental data are uninterpretable. With intensive designs, most methods presented to quantify a design's internal validity have been subject to criticism. A probabilistic model of intensive designs is presented that demonstrates the high degree of internal validity of these designs without relying on adaptations from traditional inferential statistics. Where the experimenter is able to conform to the restrictions of the model, the equations provide an estimation of internal validity for either reversal or multiple-baseline designs. More importantly, the model provides mathematical bases for some of the common recommendations and design considerations in intensive research (such as the desirability of within-subject replications and of four or more multiple baselines).     

Ecological Momentary Assessment Research in Behavioral medicine

Behavioral medicine research often seeks to examine relatively complex theoretical models and clinical processes. Traditional designs, based on retrospective self-report in the natural environment or laboratory-based studies, cannot assess complex and temporally dynamic psychological, behavioral, and physiological processes in the natural environment. An alternate research approach, Ecological Momentary Assessment (EMA), provides a method to collect such data. This research approach and these data allow investigation of a number of research questions not amenable to other data collection methods. Examples of research questions uniquely suited for EMA data include, but are not limited to: (1) the comparison of retrospective data to momentary data; (2) evaluation of physiological and psychosocial processes in the natural environment; (3) the relation of symptoms and other disease processes to characteristics of the natural environment; and (4) examination of dynamic changes in psychosocial variables that may mediate improvement following treatment. Despite the strengths of EMA data, a number of challenges exist – including logistic, analytic, and interpretation problems – to the easy implementation of EMA approaches in behavioral medicine.     

Individual differences methods for randomized experiments

Experiments allow researchers to randomly vary the key manipulation, the instruments of measurement, and the sequences of the measurements and manipulations across participants. To date, however, the advantages of randomized experiments to manipulate both the aspects of interest and the aspects that threaten internal validity have been primarily used to make inferences about the average causal effect of the experimental manipulation. This article introduces a general framework for analyzing experimental data to make inferences about individual differences in causal effects. Approaches to analyzing the data produced by a number of classical designs and 2 more novel designs are discussed. Simulations highlight the strengths and weaknesses of the data produced by each design with respect to internal validity. Results indicate that, although the data produced by standard designs can be used to produce accurate estimates of average causal effects of experimental manipulations, more elaborate designs are often necessary for accurate inferences with respect to individual differences in causal effects. The methods described here can be diversely applied by researchers interested in determining the extent to which individuals respond differentially to an experimental manipulation or treatment and how differential responsiveness relates to individual participant characteristics.     

Within-person association of volitional factors and physical activity: Insights from an ecological momentary assessment study

ObjectiveCurrent health behavior models of physical activity (PA) suggest that not all PA intentions are translated into actual PA behavior, resulting in a significant intention-behavior gap (IBG) of almost 50%. These models further suggest that higher self-efficacy and specific planning can aid in decreasing this gap. However, as most evidence stems from between-person (trait level), questionnaire-based research, it is unclear how large short-term IBGs are, how self-efficacy and planning covary within-persons across time and whether they similarly predict smaller IBGs. It is likely that day-to-day changes in circumstances and barriers affect these variables thus the applicability of theoretical models is uncertain. Here, within-person prospective analyses of ecological momentary assessment (EMA) data can provide insights.Methods35 healthy participants (aged 23–67) completed four EMA-based questionnaires every day for three weeks. Each prompt assessed PA (retrospectively, “since the last EMA prompt”); PA intentions, planning specificity, self-efficacy, and intrinsic motivation (prospectively, “until the next EMA prompt”) and momentary affect. Generalized logistic mixed-effect modeling was used to test predictors of PA.ResultsAcross the 2341 answered EMA prompts, PA intentions were not enacted in 25% of the episodes (IBG). In episodes with given intentions, PA likelihood increased with higher levels of self-efficacy, planning specificity, and intrinsic motivation. The latter two also positively predicted PA duration and intensity.ConclusionsShort-term intention behavior gaps seem to be smaller than what is known from more long-term studies, most likely as individuals can anticipate the actual circumstances of PA. Further, current health behavior models show validity in explaining within-person dynamics in IBGs across time. Knowing the relevance of planning specificity, self-efficacy and intrinsic motivation for day-to-day variations in PA enactment can inform respective real-time mHealth interventions for facilitating PA.     

Measurement‐Burst Designs for Social Health Research

Questions about variability and change in human behavior lie at the heart of much research in the behavioral sciences. This commentary examines the use of the measurement‐burst design ( Nesselroade, 1991 ) as an approach to study within‐person processes that transpire over very different temporal intervals. Consisting of repeated bursts of intensive (i.e., daily or momentary) assessments, the burst design can augment the type of information obtained from conventional daily diary and prospective longitudinal designs. We describe how the measurement‐burst approach can improve detection of long‐term intraindividual change, and how it can be used to study intraindividual variability and change in fine‐grained temporal relationships between daily experiences (e.g., hassles) and psychological states (e.g., mood). Some of the difficulties of implementing and analyzing data from measurement‐burst designs are discussed.     

Quantifying false positives in simulated events using partial interval recording and momentary time sampling with dual‐criteria methods

The dual‐criteria (DC) and conservative dual‐criteria (CDC) methods allow clinicians and researchers to quantify the occurrence of false‐positive outcomes within single‐case experimental designs. The purpose of the current study was to use these DC and CDC methods to measure the incidence of false positives with simulated data collected via discontinuous interval methods (i.e., momentary time sampling, partial‐interval recording) as a function of data series length. We generated event data to create 316,800 unique simulated data series for analysis. In Experiment 1, we evaluated how changes in relevant parameters (i.e., interval sizes, event durations, IRT‐to‐event‐run ratios) produced false positives with momentary time sampling procedures. We also assessed the degree that the CDC method produced fewer false positives than the DC method with simulated interval data. In Experiment 2, we used similar procedures to quantify the occurrence of false positives with partial‐interval recording data. We successfully replicated outcomes from previous research in the current study, though such results only highlight the generality of the procedures relating to false positive (and not false negative) outcomes. That is, these results indicate MTS and PIR may adequately control for false positives, but our conclusions are limited by a lack of data on power.     

How often are thoughts metacognitive? Findings from research on self-regulated learning,think-aloud protocols,and mind-wandering

Metacognitive monitoring refers to how people evaluate their cognitive performance. An extensive literature examines how accurately individuals engage in monitoring. The question of how often individuals engage in metacognitive monitoring has been largely neglected, although one might expect situational, group, and individual variability in monitoring frequency. We argue that this is a critical omission, given that the frequency of metacognitive monitoring might have important implications for monitoring accuracy and task performance. Within this review, we highlight findings from three literatures, that each provide insight into how often individuals engage in monitoring. To clarify the important links and potential overlaps between these separate bodies of research, we begin by summarizing the metacognitive monitoring literature, including age-related patterns in monitoring accuracy. We then connect these questions regarding spontaneous monitoring, including age-related patterns in spontaneous monitoring, to targeted reviews of the self-regulated learning, think-aloud protocol, and mind-wandering literatures. We discuss situational and dispositional factors believed to influence monitoring accuracy, and propose that the same factors could potentially influence the frequency of spontaneous monitoring. Additionally, we propose that age-related increases in spontaneous monitoring (as suggested by age-related increases in TRI) may contribute to older adults’ intact monitoring abilities. It is our hope that this review will encourage increased attention and research on the topic of spontaneous monitoring.     

Understanding Change in Recycling and Littering Behavior Across a School Social Network

Understanding how communities change requires examining how individuals’ beliefs and behaviors are shaped by those around them. This paper investigates behavior change across a large social network following a recycling intervention in a New Zealand high school community. We used a mixed methods design, combining focus group data with social network analysis from two waves of a questionnaire that measured friendship networks; recycling and littering behaviors; perceived behavioral norms; and teacher, friend, and parent encouragement for these behaviors. Recycling behavior increased significantly over the course of our study. Supporting the importance of social networks in this context, both littering and recycling behavior showed clear social clustering. Further, the degree of change in an individuals’ littering and recycling behavior across time was predicted by friends’ prior behavior. Focus group data provided insight into students’ perceptions of social interactions and how these contributed to littering and recycling behavior.     

Ecological Networks and Community Attachment and Support Among Recently Resettled Refugees

Interventions aimed at enhancing mental health are increasingly centered around promoting community attachment and support. However, few have examined and tested the specific ecological factors that give rise to these key community processes. Drawing from insights from the ecological network perspective, we tested whether spatial and social overlap in routine activity settings (e.g., work, school, childcare) with fellow ethnic community members is associated with individuals’ attachment to their ethnic communities and access to social resources embedded in their communities. Data on routine activity locations drawn from the Refugee Well‐Being Project (based in a city in the Southwestern United States) were used to reconstruct the ecological networks of recently resettled refugee communities, which were two‐mode networks that comprise individuals and their routine activity locations. Results indicated that respondents’ community attachment and support increased with their ecological network extensity—which taps the extent to which respondents share routine activity locations with other community members. Our study highlights a key ecological process that potentially enhances individuals’ ethnic community attachment that extends beyond residential neighborhoods.