The construct of psychophysiological reactivity: Statistical and psychometric issues

The purpose of this paper is to review major statistical and psychometric issues impacting the study of psychophysiological reactivity and discuss their implications for applied developmental researchers. We first cover traditional approaches such as the observed difference score (DS) and the observed residual score (RS), including a review of classic and recent research on their reliability and validity from two related bodies of work: the measurement of change and the Law of Initial Values. Second, we review several types of latent variable modeling in this context: latent difference score (LDS) models, latent residual score (LRS) models, latent state-trait (LST) models, and latent growth curve (LGC) models. Finally, we provide broad guidelines for applied researchers broken down by key stages of a psychophysiological project: study planning, data analysis, and reporting of results. Our recommendations highlight the need for (1) increased attention to the ubiquitous nature of measurement error in observed variables and the importance of employing latent variable models when possible, and (2) increased specification of theories relating to the construct of reactivity, especially in regards to the distinction between baseline arousal and change over time in broader systems of variables.     

Transient error or specificity? An alternative to the staggered equivalent split-half procedure

The data-based partial and complete reliability coefficients defined by G. Becker (2000) in his staggered equivalent split-half procedure (SESHP) were compared with the model-based specificity and consistency reliability coefficients defined by R. Steyer, M. J. Schmitt, and M. Eid (1999) in their latent state-trait model (LSTM). Partial reliability is based on coefficient alpha, which contains transient error. State and trait anxiety, measured using the State-Trait Anxiety Inventory on 2 occasions among French adults, illustrated both approaches. Theoretical and empirical analyses demonstrated that the specificity and consistency coefficients offer a testable alternative to the SESHP coefficients. In addition, dynamics of the state residual variance could be modeled and estimated in LSTM.     

Evidence of differential stability of state and trait anxiety in adults

This study used Structural Equation Modeling (SEM) to assess the differential longitudinal stabilities of state and trait anxiety. In 1985 and 1987, 227 adult volunteers (ages 22-78) rated themselves on established state and trait questionnaires. In addition, aggregated-state anxiety was measured by self-rated frequency of state anxiety experienced over the previous month. Analysis of split-half subscales from single Trait and State Anxiety scales showed the State Anxiety true score factor to have a significantly lower unstandardized autoregression coefficient (.57 vs. .91), indicating less longitudinal stability. Additional models found longitudinal factor correlations .90, .72, and .66 for Trait, Aggregate-State, and State Anxiety, respectively, further establishing the differential stability of each. There were no significant age differences in longitudinal stability or in within-occasion state-trait factor correlations.     

A latent state-trait anxiety model and its application to determine consistency and specificity coefficients

Abstract

In the present study the consistency model (Steyer, 1987) was applied to data gathered with the German version of the State-Trait Anxiety Inventory (Laux, Glanzmann, Schaffner, and Spielberger, 1981). The questionnaire was presented twice to 64 university students with an interval of two months between first and second testing. The consistency and specificity coefficients, estimated by LISREL (Jöreskog and Sörbom, 1984), support the state-trait distinction. The state variables have high specificity and consistency coefficients; the trait variables, in contrast, have high consistency coefficients but low or even negligible specificity coefficients. The discussion points out the advantages of the consistency model over the stability model; the most important advantage is that the theoretical structure of the consistency model is more appropriate for the type of application considered. It contains a state factor for each occasion of measurement and a trait factor common to all occasions of measurement. In the stability model, only latent states, but no latent traits occur. Consistency, specificity, and reliability can be identified as proportions of variance determined by latent variables specified in the model. Therefore, data analysis provides statistics concerning the practical significance of the trait and of the effects of situations and/or interactions.     

A longitudinal SEM approach to STAI data:two comprehensive multitrait-multistate models

I investigated psychometric properties of data gathered with the State-Trait Anxiety Inventory (STAI; Spielberger, Gorsuch, Lushene, Vagg, & Jacobs, 1983, 1993) using structural equation modeling. Two features were integrated into the modeling: Bistability refers to the duality that is raised once the wording effect is recognized, and second-order, state-residual/trait decomposition refers to the basic bidimensionality of the data, which is revealed in a longitudinal perspective. I tested the approach on 2 samples of French adults (total N = 888) borrowed from Vautier and Jmel (2003). The models provided acceptable goodness of fit and supported the empirical need for hypothesizing 2 trait factors and 4 transient sources of true variance. I assessed the multidimensional composition of the composites using reliability analyses. I discuss theoretical consequences of the approach.     

Analyzing the convergent and discriminant validity of states and traits: development and applications of multimethod latent state-trait models

The analysis of convergent and discriminant validity is an integral part of the construct validation process. Models for analyzing the convergent and discriminant validity have typically been developed for cross-sectional data. There exist, however, only a few approaches for longitudinal data that can be applied for analyzing the construct validity of fluctuating states. In this article, the authors show how models of latent state-trait theory can be combined with models of multitrait-multimethod analysis to develop a model that allows for analyzing convergent and discriminant validity in time: the multimethod latent state-trait model. The model allows for identifying different sources of variance (trait consistency, trait-method specificity, occasion-specific consistency, occasion-specific method specificity, and unreliability). It is applied to the repeated measurement of depression and anxiety in children, which was assessed by self and teacher reports (N = 375). The application shows that the proposed models fit the data well and allow a deeper understanding of method effects in clinical assessment.     

Decomposing person and occasion-specific effects: an extension of latent state-trait (LSI) theory to hierarchical LST models

An extension of latent state-trait (LST) theory to hierarchical LST models is presented. In hierarchical LST models, the covariances between 2 or more latent traits are explained by a general 3rd-order factor, and the covariances between latent state residuals pertaining to different traits measured on the same measurement occasion are explained by 2nd-order latent occasion-specific factors. Analogous to recent developments in multitrait-multimethod methodology, all factors are interpreted in relation to factors taken as comparison standards. An empirical example from test anxiety research illustrates how estimates of additive variance components due to general trait, specific trait, occasion, state residual, method, and measurement error can be obtained using confirmatory factor analysis. Advantages and limitations of these models are discussed.     

Changes in mood caused by staged, moderate physical activity]

This experiment provides a test for the common sense knowledge that moderate physical exercise leads to mood improvement. Furthermore, it was tested whether light exercise intensifies negative feeling states or alleviates them. 30 female and 30 male students of psychology served as subjects (mean age 25.3 years, SD = 4.8). After being exposed to a mood induction procedure designed to elicit either a positive or negative feeling state, the subjects had to pedal a bicycle ergometer with 0, 50 or 75 Watt load. Cardiovascular variables and self-reports of mood states were assessed during a baseline period, after the mood induction, following the ergometer exercise, and after a follow-up period. The mood induction procedures were successful, but only for a short duration. Physiological activation was observed according to the ergometer loadings. Moderate physical exercise led to an increase of positive feeling states (Concentration) and a decrease of negative feeling states (Tiredness) in the follow-up period. Corresponding changes in tension related states could not be observed, probably due to the weakness of the mood induction procedure employed and the low level of energetic activation reached.     

How satisfied are you with your job? Estimating the reliability of scores on a single‐item job satisfaction measure

Estimating the reliability of scores on single‐item measures can be difficult because commonly used internal consistency estimates of reliability cannot be calculated. When longitudinal data is available, statistical models can be used to decompose the variability in the latent variable at each wave into trait versus state variance. Then, reliability can be estimated as a ratio of the sum of the trait variance that is captured in repeated assessments over the total variance. The current study used latent trait‐state‐error models on a nine‐year longitudinal data (N = 5,003) to estimate the test–retest reliability of scores on a single‐item measure of job satisfaction. Results showed that job satisfaction scores were somewhat unreliable (r_xx = .49–.59) and amenable to change.     

Latent state–trait theory: An application in sport psychology

ObjectiveQuestionnaires are often applied in sports psychology to measure a person's trait or state. However, the extent to which the questionnaire captures differences because of trait or state influences is often unclear. The latent state–trait (LST) theory offers the opportunity to separate both variance sources. This separation allows estimating specific reliability coefficients.DesignThe present paper gives a theoretical introduction to LST and its basic ideas. Using a real data set with N = 156 athletes we exemplify the steps necessary to derive the LST coefficients. All athletes filled out a comprehensive inventory assessing competitive anxiety on two occasions.MethodThe data are analysed with structural equation models based on LST principles.ResultsThe results confirm the questionnaire's trait saturation.ConclusionFinally, results are discussed in light of practical and theoretical implications.     

A structural comparison between state and trait self-regulation models

State and trait-regulation models were compared in 315 eleventh-graders using simultaneous confirmatory factor analysis. Each of the first-, second-, and third-order factor models was found to describe the data from both state and trait measures well. In both state and trait self-regulation models, the third-order factor, self-regulation, is composed of metacognition and effort, and the second-order factor, metacognition, is composed of awareness, cognitive strategy, planning, and self-checking. Findings of invariant factor loadings and factor correlations across state and trait measures indicate that the state and trait self-regulatory factor structures are similar. Unlike the state-trait distinctions found in affective personality domain, clear state-trait distinction was not found in this cognitive domain. In both measures, students who engaged more frequently in metacognitive activities and effort use in their mathematical problem solving performed better than those who did not.     

Mixture distribution latent state-trait analysis: basic ideas and applications

Extensions of latent state-trait models for continuous observed variables to mixture latent state-trait models with and without covariates of change are presented that can separate individuals differing in their occasion-specific variability. An empirical application to the repeated measurement of mood states (N=501) revealed that a model with 2 latent classes fits the data well. The larger class (76%) consists of individuals whose mood is highly variable, whose general well-being is comparatively lower, and whose mood variability is influenced by daily hassles and uplifts. The smaller class (24%) represents individuals who are rather stable and happier and whose mood is influenced only by daily uplifts but not by daily hassles. A simulation study on the model without covariates with 5 sets of sample sizes and 5 sets of number of occasions revealed that the appropriateness of the parameter estimates of this model depends on number of observations (the higher the better) and number of occasions (the higher the better). Another simulation study estimated Type I and II errors of the Lo-Mendell-Rubin test.     

Accounting for fluctuations in body dissatisfaction

The present study evaluated whether the strength of relationship between contextual cues (presence of company and mood) and state body dissatisfaction varied as a function of individual differences in key trait measures (body shame, body surveillance tendencies, internalization of appearance standards, and trait affect) which have been linked to trait body dissatisfaction. Fifty-five undergraduate women completed a questionnaire containing the trait-based measures and then carried a Personal Digital Assistant (PDA) for a 7-day period. The PDA prompted participants six times daily to self-report their current mood and state body dissatisfaction. Multi-level modeling revealed that individual differences in body shame predicted inter-individual variability in the strength of the relationships between presence of company and state body dissatisfaction, and positive mood and state body dissatisfaction. Trait positive affect also explained variance in the positive mood state-body dissatisfaction relationship. The implications of the findings for prevention of body image disturbances are discussed.     

Flow at work: An experience sampling approach

One of the core constructs of the positive psychology movement is that of ‘flow’, or optimal experience. The current study investigated the relationship between ‘flow’, the core job dimensions, and subjective well‐being (SWB), as well as distinguishing between the state and trait components of flow. Experience sampling methodology (ESM) was used to track 40 architectural students over a 15 week semester while they engaged in studio work. Hierarchical linear modelling (HLM) indicated that 74% of the variance in flow was attributable to situational characteristics compared to dispositional factors. Results also indicated that academic work that was high in skill variety and autonomy was associated with flow. Flow was found to be correlated with positive mood. Cross‐lagged regression analysis showed that momentary flow was predictive of momentary mood and not vice versa. The strengths and limitations of using ESM to study subjective work experiences and well‐being are discussed, as well as the implications of the study of flow or optimal experience for industrial/organizational psychology.     

Applying Bifactor Statistical Indices in the Evaluation of Psychological Measures

The purpose of this study was to apply a set of rarely reported psychometric indices that, nevertheless, are important to consider when evaluating psychological measures. All can be derived from a standardized loading matrix in a confirmatory bifactor model: omega reliability coefficients, factor determinacy, construct replicability, explained common variance, and percentage of uncontaminated correlations. We calculated these indices and extended the findings of 50 recent bifactor model estimation studies published in psychopathology, personality, and assessment journals. These bifactor derived indices (most not presented in the articles) provided a clearer and more complete picture of the psychometric properties of the assessment instruments. We reached 2 firm conclusions. First, although all measures had been tagged “multidimensional,” unit-weighted total scores overwhelmingly reflected variance due to a single latent variable. Second, unit-weighted subscale scores often have ambiguous interpretations because their variance mostly reflects the general, not the specific, trait. Finally, we review the implications of our evaluations and consider the limits of inferences drawn from a bifactor modeling approach.     

The making of a personality inventory: Help from the WWW

Data collected electronically from the World-Wide Web (WWW) in two samples (n=429 and 1,657) were compared with traditional paper-and-pencil measure data from two university samples (n=760 and 148) in three psychometric studies of a new measure of self-trust, the Self-Trust Questionnaire. With the exception of scale score variance, which was larger in WWW samples, psychometric properties of the scale were comparable across samples collected from the two sources. WWW samples were more similar to psychology student samples (predominantly young females) than to typical Internet users. On balance, findings indicate that the advantages of the WWW as a data source, including large heterogeneous samples, outweigh problems with data accuracy and generalizability, making the WWW an attractive source of data for researchers developing self-report personality inventories.     

Permanency and ephemerality of psychological measures with application to organizational commitment

The constancy or change of an attribute is important to most substantive areas of psychology. During the past decade, 2 independent methodological schools have developed statistical models for the depiction of longitudinal research. One, which might be called the European school, has created latent state-trait models. Alternatively, the American school has formulated models that go by the rubric of latent curve analysis or latent growth models. In this article, the authors integrate both approaches into a detailed unified latent curve and latent state-trait model (LC-LSTM) that includes the significant features from both schools. From the LC-LSTM framework, the permanency and ephemerality of psychological measures are discussed and the concepts of stability and reliability are reformulated. In addition, a comprehensive illustration on organization commitment is presented.     

The interclinician reliability of Rorschach interpretation in four data sets

To examine agreement on Rorschach Comprehensive System (CS; Exner, 2004) interpretations, 55 patient protocols were interpreted by 3 to 8 clinicians across 4 data sets on a representative set of 29 characteristics. Substantial reliability was observed across data sets, although a problematic design produced lower results in one. Unexpectedly, a Q-sort task had slightly lower reliability than a simple rating task. As expected, scales that summarized judgments had higher agreement than judgments to individual interpretive statements, and some clinicians produced more generalizable inferences than others. Interpretations for all clinicians were more strongly associated with patients' psychometric true scores (aggregated judgment M range = .82 to .92) than with the judgments of other clinicians (range = .76 to .89). Compared to meta-analyses of interrater reliability in psychology and medicine, the findings indicate these clinicians could reliably interpret Rorschach CS data.