Individual differences make a difference: On the use and the psychometric properties of difference scores in social psychology

Much social psychological research is concerned with the question whether and how behavior changes because of a “treatment” (e.g., a situation that triggers a psychological reaction). One easy way to investigate such changes would be to analyze intraindividual differences before (Time 1) and after the treatment (Time 2). Interestingly, many scholars refrain from using difference scores because they think they are inherently unreliable. However, the bad reputation of difference scores is, in many cases, unwarranted: difference scores can be sufficiently reliable when standard deviations differ between measurement occasions, and standard deviations are likely to differ between measurement occasions because of differential treatment effects (i.e., interindividual differences in responsiveness to a treatment) and/or “strong situation” treatments. In the present article, we will (1) summarize classic and current arguments regarding the reliability of difference scores, (2) discuss the use of residual change scores as an alternative to difference scores, and (3) argue that latent difference score models are a particularly useful tool that social psychologists should consider using more frequently. Copyright © 2014 John Wiley & Sons, Ltd.     

An Idiographic Examination of Day-to-Day Patterns of Substance Use Craving,Negative Affect,and Tobacco Use Among Young Adults in Recovery

Psychological constructs, such as negative affect and substance use cravings that closely predict relapse, show substantial intraindividual day-to-day variability. This intraindividual variability of relevant psychological states combined with the “one day at a time” nature of sustained abstinence warrant a day-to-day investigation of substance use recovery. This study examines day-to-day associations among substance use cravings, negative affect, and tobacco use among 30 college students in 12-step recovery from drug and alcohol addictions. To account for individual variability in day-to-day process, it applies an idiographic approach. The sample of 20 males and 10 females (mean age = 21) was drawn from members of a collegiate recovery community at a large university. Data were collected with end-of-day data collections taking place over an average of 26.7 days. First-order vector autoregression models were fit to each individual predicting daily levels of substance use cravings, negative affect, and tobacco use from the same 3 variables 1 day prior. Individual model results demonstrated substantial interindividual differences in intraindividual recovery process. Based on estimates from individual models, cluster analyses were used to group individuals into 2 homogeneous subgroups. Group comparisons demonstrate distinct patterns in the day-to-day associations among substance use cravings, negative affect, and tobacco use, suggesting the importance of idiographic approaches to recovery management and that the potential value of focusing on negative affect or tobacco use as prevention targets depends on idiosyncratic processes.     

How shall we study individual differences in cognitive abilities?—Methodological and theoretical perspectives

Selected studies in recent research literature are examined with regard to (a) their methodological adequacy in establishing dimensions of individual differences in information-processing abilities, and (b) the theoretical problems involved in inferring psychological processes from individual difference dimensions. Many methodological inadequacies are noted, including use of small Ns, questionable procedures in establishing variables (e.g., use of raw difference scores), improper or inadequate use of factor analysis and other multivariate statistical techniques, and poor presentation of results. On the whole, little progress has been made thus far in identifying psychological processes through research in individual differences, even though this research approach is viable and potentially useful. Serious theoretical difficulties arise in attempting to infer the nature and operation of psychological processes merely from the identification of individual difference trait dimensions. Promising research, however, is represented by studies in which an effort is made to analyze tasks into their components, to vary task characteristics, and/or to consider the strategies that individuals can employ in performing the tasks.     

Memory Strategy Development: Lessons from Longitudinal Research

Most research in memory strategy development involves cross-sectional studies, with the implicit assumption that the patterns found in these studies are an accurate reflection of intraindividual changes over time. In this article, we review briefly the development of rehearsal and organizational strategies based on cross-sectional research and then review the findings from the few longitudinal studies assessing intraindividual change in these strategies, focusing on results from the Munich Longitudinal Study (LOGIC). Although average, age-related patterns obtained from longitudinal samples provide similar development functions as those obtained from cross-sectional samples, there is much greater individual variability in strategy acquisition observed in the longitudinal data. This suggests that the canonical development patterns inferred from cross-sectional studies often obfuscate actual intraindividual patterns. Longitudinal evidence also indicates that utilization deficiencies are observed less frequently than assumed in cross-sectional research.     

The role of response styles in the assessment of intraindividual personality variability

Although personality psychologists often focus on between-person differences, understanding intraindividual variability is also a critical focus of the subdiscipline. Despite the fact that non-self-report techniques exist for assessing variability, questionnaire-based measures are still the norm. In two studies (N = 149 and N = 202) we examine the possibility that intraindividual variability measures derived from repeated self-report assessments are affected by certain response styles. These studies, which use a variety of techniques for assessing within-person variability, show that standard measures are moderately to strongly correlated with theoretically unrelated variability measures, including those based on ratings of satisfaction with neutral objects or the personality of cartoon characters. These results raise questions about the validity and utility of widely used measures for assessing intraindividual variability.     

The New Person-Specific Paradigm in Psychology

ABSTRACT— Most research methodology in the behavioral sciences employs interindividual analyses, which provide information about the state of affairs of the population. However, as shown by classical mathematical-statistical theorems (the ergodic theorems), such analyses do not provide information for, and cannot be applied at, the level of the individual, except on rare occasions when the processes of interest meet certain stringent conditions. When psychological processes violate these conditions, the interindividual analyses that are now standardly applied have to be replaced by analysis of intraindividual variation in order to obtain valid results. Two illustrations involving analysis of intraindividual variation of personality and emotional processes are given.     

The locus of the benefits of repetition-lag memory training

The current study examined the effects of responses on error-adjacent trials (i.e., those immediately preceding or following errors) on age differences in measures of intraindividual variability and the shape of response time (RT) distributions on a two-back task. Removing error-adjacent responses reduced variability as measured by the coefficient of variation, but did so similarly for younger and older adults. However, older adults’ standard deviations (SDs) were less than those of younger adults with comparable RTs, raising questions regarding the validity of the coefficient of variation. An ex-Gaussian analysis revealed that removing the RTs on error-adjacent trials reduced the length of the tails of distributions and the skewness of the distributions. These properties were reduced more for older adults than for younger adults. These results indicate that the influence of error-adjacent trials should be considered when analyzing intraindividual variability and the shape of RT distributions to test theories of cognitive aging.     

A reliability generalization study of coefficient alpha for the UCLA loneliness scale

Loneliness is a psychological construct that has been reported in a variety of populations and associated with a number of other negative psychological problems. This study was an examination of coefficient alpha of a prominent measure of loneliness: the University of California, Los Angeles (UCLA) Loneliness Scale (Russell, Peplau, & Cutrona, 1980; Russell, 1996). We utilized reliability generalization to provide an aggregate estimate of the reliability of the scale over time and in a variety of populations as well as to assess and identify sampling and demographic characteristics associated with variability in coefficient alpha. Of the 213 studies examined, 80 had reported alpha estimates, and we used them in this analysis. We discuss conditions associated with variability in coefficient alpha along with pertinent implications for practice and future research.     

Measurement‐Burst Designs for Social Health Research

Questions about variability and change in human behavior lie at the heart of much research in the behavioral sciences. This commentary examines the use of the measurement‐burst design ( Nesselroade, 1991 ) as an approach to study within‐person processes that transpire over very different temporal intervals. Consisting of repeated bursts of intensive (i.e., daily or momentary) assessments, the burst design can augment the type of information obtained from conventional daily diary and prospective longitudinal designs. We describe how the measurement‐burst approach can improve detection of long‐term intraindividual change, and how it can be used to study intraindividual variability and change in fine‐grained temporal relationships between daily experiences (e.g., hassles) and psychological states (e.g., mood). Some of the difficulties of implementing and analyzing data from measurement‐burst designs are discussed.     

Modeling Noisy Data with Differential Equations Using Observed and Expected Matrices

Complex intraindividual variability observed in psychology may be well described using differential equations. It is difficult, however, to apply differential equation models in psychological contexts, as time series are frequently short, poorly sampled, and have large proportions of measurement and dynamic error. Furthermore, current methods for differential equation modeling usually consider data that are atypical of many psychological applications. Using embedded and observed data matrices, a statistical approach to differential equation modeling is presented. This approach appears robust to many characteristics common to psychological time series.     

Group differences in intelligence and speed of information-processing

Three samples of different levels of mental ability, but each of average or above- average intelligence, were administered a battery of reaction time test which measure the speed with which persons can execute various elementary cognitive processes. Discriminant analyses revealed significant differences between the groups in reaction times and intraindividual standard deviations, and univariate analyses indicated that higher ability groups tended to obtain significantly faster mean reaction times and to show significantly less intraindividual variability than lower ability groups. Groups which differed the most in mental ability differed to the greatest extent on the more complex reaction time tests but this pattern was reversed for groups which were relatively close in mean IQ. Generally, the results support the hypothesis that speed- of-processing is an important factor unferlying intelligence, but a number of puzzling findings indicate the need for continued research in this area.     

Positional Sequences in Two-Choice Discrimination Problems with Non-Positional Cues

Individuals seem to differ in their ability to maintain consistent performance. This is obviously an important attribute of behavior that has been largely neglected by researchers. There is, however, some evidence to suggest that, in certain types of situations, the behaviors of the mental retardate are characterized by a relatively high degree of variability. Two studies are reported here that deal with some of the developmental implications of intraindividual variability. In one study, involving retardates, reliability and generality of response consistency were examined in reaction time, time estimation, and digit span tasks. It was shown that individual differences in consistency are reliable and that these generalize across tasks. A second experiment was concerned with the developmental aspects of variability. First-, third-, and fifth-grade pupils and college students were given many trials on a reaction-time task. A variety of measures, reflecting typical performance, limit of performance, and variability, were computed for each subject. The ability to react quickly shows a developmental trend not only with respect to overall performance, but also in the efficiency with whichthe individual maintains his optimal level of responding.     

Individual differences in performance on elementary cognitive tasks (ECTs): lawful vs. problematic parameters.

Over the past 2 decades, the cognitive-correlates approach has dominated investigations into the nature of intelligence. This research program relies on a number of processing speed parameters (apart from "average performance"). These measures include the slope, intercept, and intraindividual variability of both decision time and movement time. By correlating these measures with established markers of intelligence, researchers postulate theoretical models underlying these information-processing constructs. However, there is a lack of substantive evidence that these phenomena are as robust within the individual as has been proposed. The authors tested the properties of intraindividual parameters by asking participants (N = 179) to perform 10 elementary cognitive tasks (ECTs). Detailed analyses revealed that average performance parameters, extracted from these ECTs, behaved lawfully. However, up to 40% of participants failed to provide acceptable indices of intraindividual model fit. Similarly, intraindividual variability measures appeared less valid than previously suggested. The implications of these findings for cognitive and biological models of intelligence are discussed.     

Bringing Back the Person into Behavioural Personality Science Using Big Data

Behaviour and the individual person are important but widely neglected topics of personality psychology. We argue that new technologies to collect and new methods to analyse Big (Behavioural) Data have the potential to bring back both more behaviour and the individual person into personality science. The call for studying the individual person in the history of personality science, the related idiographic/nomothetic divide, as well as attempts to reconcile these two approaches are briefly reviewed. Furthermore, different meanings of the term idiographic and some unique selling points that emphasize the importance of idiographic research are highlighted. A nonexhaustive literature review shows that a wealth of behaviours are considered in extant personality studies using such Big Data but only in a nomothetic way. Against this background, we demonstrate the potential of Big Data collection and analysis with regard to four idiographic research topics: (i) unique manifestations of common traits and the resurgence of personal dispositions, (ii) idiographic prediction, (iii) intraindividual consistency versus variability of behaviour and (iv) intraindividual personality trait change through intervention. Methodological, ethical and legal pitfalls of doing Big Data research with individual persons as well as potential countermeasures are considered.     

Methodological issues in the quantitative analysis of dream content

Reviews measurement approaches applied to dream reports and proposes some methodological cautions that may result in more effective utilization of the dream report in research and assessment. Standardization of dream collection techniques and controls for variability in the length of dream reports are suggested. Further investigation of the intraindividual consistency of dream report measures is seen as necessary for a discussion of their validity. Though there are indications that a variety of dream report measures are potentially valid for psychological assessment, measures based on general dream report characteristics show greater promise than specific content analysis. Research on the differences between normal and disturbed subjects highlights the potential, and the problems, inherent in the use of dream reports as tools of psychological inquiry.     

The inter-item standard deviation (ISD): An index that discriminates between conscientious and random responders

Although random responding is prevalent and increases Type II errors, most psychologists avoid trying to identify it because the means to do so are extremely limited. We propose the inter-item standard deviation (ISD), a statistical index of response variance, is suited for this task. We hypothesized that random responders produce large ISDs because they respond to items all over a measure’s response range, whereas conscientious responders produce small ISDs because they respond to items more consistently. We administered a questionnaire containing the NEO-FFI-3 and an embedded validity scale to 134 university students. Another 134 responders were created using a random number generator. For all 268 responders, the ISD was calculated for each of the NEO-FFI-3′s five subscales and an aggregated ISD was calculated by averaging the five ISD indexes. Results showed that (1) random responders produce significantly larger ISDs than conscientious responders, (2) the ISDs were strongly correlated with the embedded validity scale and with one another, and (3) the ISDs correctly identified responders with greater than 80% classification accuracy. The mean ISD yielded greater than 95% classification accuracy. This study shows that responders can be identified by quantifying inter-item response variance.     

Experimental analysis of individual differences and personality

In psychological studies individual differences that result in variability in data and thus mask the effects under investigation have been reduced or eliminated in two ways: (1) through the use of large numbers of subjects and statistical manipulations, or (2) through extensive and controlled studies of individual subjects. The latter, behavior-analytic, method is scientifically better because it permits identification of the variables that result in individual differences. This paper advocates the direct study of individual differences and personality rather than indirect study through experimental control procedures. Some data are presented showing that individual differences in response patterns have orderly characteristics. Extension of the experimental analysis of behavior to the study of individual differences and personality is likely to be important both scientifically and for the future growth of behavior analysis.     

Applying mixed‐effects modeling to single‐subject designs: An introduction

Behavior analysis and statistical inference have shared a conflicted relationship for over fifty years. However, a significant portion of this conflict is directed toward statistical tests (e.g., t‐tests, ANOVA) that aggregate group and/or temporal variability into means and standard deviations and as a result remove much of the data important to behavior analysts. Mixed‐effects modeling, a more recently developed statistical test, addresses many of the limitations of more basic tests by incorporating random effects. Random effects quantify individual subject variability without eliminating it from the model, hence producing a model that can predict both group and individual behavior. We present the results of a generalized linear mixed‐effects model applied to single‐subject data taken from Ackerlund Brandt, Dozier, Juanico, Laudont, & Mick, 2015, in which children chose from one of three reinforcers for completing a task. Results of the mixed‐effects modeling are consistent with visual analyses and importantly provide a statistical framework to predict individual behavior without requiring aggregation. We conclude by discussing the implications of these results and provide recommendations for further integration of mixed‐effects models in the analyses of single‐subject designs.     

THE SUBSTANTIVE NATURE OF PERFORMANCE VARIABILITY: PREDICTING INTERINDIVIDUAL DIFFERENCES IN INTRAINDIVIDUAL PERFORMANCE

The nature of intraindividual performance variability over time, along with individual difference predictors of such variability, was examined using latent growth curve methodology. Quarterly sales performance for a sample of securities analysts ( n = 303) was measured at 8 times. Average intraindividual performance approximated a basic "learning" curve, although there were considerable individual differences in each of the latent performance growth parameters. Individual difference predictors from a biodata inventory were moderately related to these latent growth parameters. Theoretical and practical implications of performance variability for personnel selection are also discussed.     

Sidman or statistics?

Murray Sidman's statements regarding variability, experimental control, and generality are interwoven with examples from the literature on conditional discrimination. Sidman's position was that statistical inferences from group studies produce no information about the behavior of individual subjects and that statistical treatment of individual subject data masks variability which may represent conditions that are not controlled. Sidman's work on conditional discrimination provides excellent examples of how complex discriminations should be examined in detail with accuracy levels obtained for each type of discrimination within an experiment. Sidman made important contributions to the foundation of behavior analysis with extensive basic research as well as applications of methods and principles to clinical and educational settings.