Consequences of Ignoring Guessing Effects on Measurement Invariance Analysis

Pseudo-guessing parameters are present in item response theory applications for many educational assessments. When sample size is not sufficiently large, the guessing parameters may be ignored from the analysis. This study examines the impact of ignoring pseudo-guessing parameters on measurement invariance analysis, specifically, on item difficulty, item discrimination, and mean and variance of ability distribution. Results show that when non-zero guessing parameters are ignored from the measurement invariance analysis, item discrimination estimates tend to decrease particularly for more difficult items, and item difficulty estimates decrease unless the items are highly discriminating and difficult. As the guessing parameter increases, the size of the decrease in item discrimination and difficulty tends to increase, and the estimated mean and variance of ability distribution tend to be inaccurate. When two groups have heterogeneous ability distributions, ignoring the guessing parameter affects the reference group and the focal group differently. Implications of result findings are discussed.     

True or False? Keying Direction and Acquiescence Influence the Validity of Socio-Emotional Skills Items in Predicting High School Achievement

What type of items, keyed positively or negatively, makes social-emotional skill or personality scales more valid? The present study examines the different criterion validities of true- and false-keyed items, before and after correction for acquiescence. The sample included 12,987 children and adolescents from 425 schools of the State of São Paulo Brazil (ages 11–18 attending grades 6–12). They answered a computerized 162-item questionnaire measuring 18 facets grouped into five broad domains of social-emotional skills, i.e.: Open-mindedness (O), Conscientious Self-Management (C), Engaging with others (E), Amity (A), and Negative-Emotion Regulation (N). All facet scales were fully balanced (3 true-keyed and 3 false-keyed items per facet). Criterion validity coefficients of scales composed of only true-keyed items versus only false-keyed items were compared. The criterion measure was a standardized achievement test of language and math ability. We found that coefficients were almost as twice as big for false-keyed items’ scales than for true-keyed items’ scales. After correcting for acquiescence coefficients became more similar. Acquiescence suppresses the criterion validity of unbalanced scales composed of true-keyed items. We conclude that balanced scales with pairs of true and false keyed items make a better scale in terms of internal structural and predictive validity.     

What do we measure and how do we elicit it? The case for the use of repertory grid technique in multi-party psychological contract research

ABSTRACT

The psychological contract is a multi-faceted construct and, with the rise of gig work, increasingly the contract extends beyond the conventional employer-employee relationship to encompass multi-party exchanges. Against this backdrop, the question of what should be measured when assessing the contract and how it should be elicited remains a significant issue for scholars. We argue that the potential of psychological contract theory as an explanatory lens in understanding contemporary multi-party working relationships is constrained by two key limitations inherent in conventional measurement approaches. Firstly, such approaches have favoured singular rather than multiparty perspectives, and secondly, they have broadly accorded equal weight and significance to the content dimensions unearthed, despite the fact that they may differ markedly in how they are understood by each party to the employment relationship. In order to remedy these shortcomings, we make the case for adopting repertory grid technique as a methodological framework to address measurement limitations and to more rigorously assess the content of the complex multi-party psychological contract.     

Toward Assessing and Improving Survey Questions on Religion: The ARDA's Measurement Wizard

Developing survey measures of religion is deceptively difficult. As religion in America becomes more diverse and research projects become more global, finding a shared vocabulary that taps into concepts of interest and selecting response categories that capture the full range of choices becomes ever more challenging. This research project introduces new tools and techniques for exploring and evaluating past survey measures of religion, including a new software tool that draws on the ARDA's massive collection of survey measures. We offer two examples to demonstrate how these resources can serve to evaluate past survey measures of religion and improve future measures.     

WHEN PERFORMANCE IMPROVEMENT IS THE GOAL: A NEW SET OF CRITERIA FOR CRITERIA

Weitz (1961), aware of the lure of tradition and expedience, urged the identification of evaluation standards for dependent variables, which he referred to as criteria for criteria. In this article, five criteria are proposed using the mnemonic SURF & C: the directness of sampling (S); the responsiveness of the target (i.e., making sure the dependent variable is under (U) the worker's control); the reliability (R) of observers; the frequent (F) assessment of the target during the intervention period; and the critical (C) nature of the target. Together the criteria provide guidelines for what and how targets should be assessed. Their necessity is illustrated in two year-long experiments designed to improve the preventive maintenance of heavy equipment in the U.S. Marine Corps. Although the criteria are limited to evaluating dependent variables in field experiments, they are recommended as the foundation for successful performance efforts in any applied setting.     

Estimation of Controlled Direct Effects in Longitudinal Mediation Analyses with Latent Variables in Randomized Studies

Abstract

In a randomized study with longitudinal data on a mediator and outcome, estimating the direct effect of treatment on the outcome at a particular time requires adjusting for confounding of the association between the outcome and all preceding instances of the mediator. When the confounders are themselves affected by treatment, standard regression adjustment is prone to severe bias. In contrast, G-estimation requires less stringent assumptions than path analysis using SEM to unbiasedly estimate the direct effect even in linear settings. In this article, we propose a G-estimation method to estimate the controlled direct effect of treatment on the outcome, by adapting existing G-estimation methods for time-varying treatments without mediators. The proposed method can accommodate continuous and noncontinuous mediators, and requires no models for the confounders. Unbiased estimation only requires correctly specifying a mean model for either the mediator or the outcome. The method is further extended to settings where the mediator or outcome, or both, are latent, and generalizes existing methods for single measurement occasions of the mediator and outcome to longitudinal data on the mediator and outcome. The methods are utilized to assess the effects of an intervention on physical activity that is possibly mediated by motivation to exercise in a randomized study.     

Envisioning more virtuous virtues

ABSTRACT

This response addresses three key themes related to the interdisciplinary study of virtue: limitations of self-reports, demoting of the virtues, and concerns about cultural issues. I suggest that self-regulation theories of how people develop virtues are needed in order to understand how virtues (even just two at a time) may work in tandem to influence behavior. I suggest that psychologists and philosophers approach an interdisciplinary conversation about virtue with an eye towards the limitations in our field’s typical way of approaching constructs. Such an approach provides an opportunity to see problems that are typically masked by disciplinary habits.     

Positive psychology,the classification of character strengths and virtues,and issues of measurement

ABSTRACT

This essay raises concerns about positive psychology’s classification of character strengths and virtues and issues of measurement. Part I examines the process whereby the classification was compiled. Part II turns to issues of measurement and questions about positive psychologists’ sensitivity to variations in the meanings of the constructs they purport to measure, both within and across cultures. I argue that attempts to find a ‘deep structure’ of the character strengths and virtues should proceed hand in hand with efforts to render positive psychology and its measurement tools more sensitive to variability in character strengths and virtues across and within cultures. The essay concludes with suggestions for future research.     

Another way to think about psychological change: experiential vs. incremental

The traditional pre-post treatment difference reflects an incremental notion of change, where a quantity of some psychological function is added to (or subtracted from) a pretreatment quantity. This study presents a complementary, experiential notion of change. Rather than a difference, change is a feeling or experience of having changed, a feeling that one is different than before. Based on a post-treatment interview the Change after Psychotherapy (CHAP) is a method to quantify/rate such ‘differentness’ in terms of how extensively or radically the patient feels having changed. A pre-post quasi-experimental study (N = 49) comparing the CHAP with ratings on the Global Assessment of Functioning, the Karolinska Psychodynamic Profile and the Comprehensive Psychopathological Rating Scale-Self-Affective is reported. The results showed the CHAP to be a reliable, valid and sensitive way to assess an experiential kind of CHAP.