A Dialectic on Validity: Where We Have Been and Where We Are Going

It has been suggested that the primary issue in considerations of validity today is the discrepancy between currently accepted theory and the practice of validation. The reason for this discrepancy may be that although considerable discussion has taken place in educational measurement and psychometrics regarding changes in our conceptions of validity, that has not been the case more generally in the social and behavioral sciences. A dialectic on the current status of validity theory for social and behavioral scientists is presented to help bridge this gap.     

On Johnson's (2000) Relative Weights Method for Assessing Variable Importance: A Reanalysis

This article provides a reanalysis of J. W. Johnson's (2000) “relative weights” method for assessing variable importance in multiple regression. The primary conclusion of the reanalysis is that the derivation of the method is theoretically flawed and has no more validity than the discredited method of Green, Carroll, and DeSarbo (1978) on which it is based. By means of 2 examples, supplemented by other results from the literature, it is also shown that the method can result in materially distorted inferences when it is compared with another widely used importance metric, namely, general dominance (Azen & Budescu, 2003; Budescu, 1993). Our primary recommendation is that J. W. Johnson's (2000) relative weights method should no longer be used as a variable importance metric for multiple linear regression. In the final section of the article, 2 additional recommendations are made based on our analysis, examples, and discussion.     

The Role of Occupational Characteristics and Occupational Imbalance in Explaining Well-being

One of the key factors for promoting well-being lies in balancing one’s daily life occupations and the nature of these occupations. Occupation refers to a group of everyday life goal-directed activities which may be associated to any life domain, not necessarily work, such as leisure, education, or self-care. Yet it is not clear what constitutes occupational imbalance, and its association to other factors has not been examined systematically. This study proposed and tested a theoretical model for specifying the structural relationships between occupational imbalance, occupational characteristics, personality and well-being. 288 working adults completed the following questionnaires: Inter-goal Relations Questionnaire (occupational imbalance), Personal Projects Analysis (occupational characteristics), the Big Five Inventory (personality traits) and the Satisfaction with Life Scale (well-being). Twenty-five models were tested using structural equation modeling. All the models fit the data well. Occupational characteristics, with the exception of stress, served as significant mediators between personality and well-being, yet occupational imbalance did not play a significant role across all models. Our findings provide substantial support for the importance of occupation to well-being, however occupational imbalance is still an elusive concept that requires further investigation theoretically and empirically.     

Further evidence for Coren and Hakstian's "Methodological implications of interaural correlation: count heads not ears" and an alternative correction formula.

Coren and Hakstian (1990) identified a serious methodological problem that arises in auditory research because of interaural correlation. When measures from both ears of the subjects are pooled together in an experimental design that assumes independence of measures, there can be spuriously high apparent statistical significance. The present paper provides further evidence in support of Coren and Hakstian's argument and also derives a formula that effectively corrects inflated test statistics resulting from interaural correlation. This formula is a special case of a more general one that applies in many other experimental contexts in which nonindependence of measures is a problem. We found that statistical tests based on our formula have somewhat greater power to detect differences than the kind of correction method advocated by Coren and Hakstian.     

Uncovering Substantive Patterns in Student Responses in International Large-Scale Assessments—Comparing a Latent Class to a Manifest DIF Approach

In this study, we contrast results from two differential item functioning (DIF) approaches (manifest and latent class) by the number of items and sources of items identified as DIF using data from an international reading assessment. The latter approach yielded three latent classes, presenting evidence of heterogeneity in examinee response patterns. It also yielded more DIF items with larger effect sizes and more consistent item response patterns by substantive aspects (e.g., reading comprehension processes and cognitive complexity of items). Based on our findings, we suggest empirically evaluating the homogeneity assumption in international assessments because international populations cannot be assumed to have homogeneous item response patterns. Otherwise, differences in response patterns within these populations may be under-detected when conducting manifest DIF analyses. Detecting differences in item responses across international examinee populations has implications on the generalizability and meaningfulness of DIF findings as they apply to heterogeneous examinee subgroups.     

Index to Volume 112

For various nonnormal distributions, the power of the Student t test can be increased if continuous measures are transformed to ranks before the test is performed. The power of the test can also be increased almost as much and, even more for some distributions, if measures are replaced by dichotomous variables with the values 0 and 1, instead of ranks. Similarly, the power of a significance test of correlation can be increased if scores are transformed to ranks, that is, with the use of the Spearman rank correlation method. Power can also be increased almost as much and in some cases even more if dichotomous variables are introduced, that is, if the phi coefficient is used as a measure of correlation.     

Population models and simulation methods: The case of the Spearman rank correlation

The purpose of this paper is to highlight the importance of a population model in guiding the design and interpretation of simulation studies used to investigate the Spearman rank correlation. The Spearman rank correlation has been known for over a hundred years to applied researchers and methodologists alike and is one of the most widely used non‐parametric statistics. Still, certain misconceptions can be found, either explicitly or implicitly, in the published literature because a population definition for this statistic is rarely discussed within the social and behavioural sciences. By relying on copula distribution theory, a population model is presented for the Spearman rank correlation, and its properties are explored both theoretically and in a simulation study. Through the use of the Iman–Conover algorithm (which allows the user to specify the rank correlation as a population parameter), simulation studies from previously published articles are explored, and it is found that many of the conclusions purported in them regarding the nature of the Spearman correlation would change if the data‐generation mechanism better matched the simulation design. More specifically, issues such as small sample bias and lack of power of the t‐test and r‐to‐z Fisher transformation disappear when the rank correlation is calculated from data sampled where the rank correlation is the population parameter. A proof for the consistency of the sample estimate of the rank correlation is shown as well as the flexibility of the copula model to encompass results previously published in the mathematical literature.     

Coefficients of feasibility for ordinal response scales

Cronbach's alpha is systematically used in the social sciences to estimate internal consistency. However, this coefficient assumes the continuity of the variables and this assumption is not met by ordinal response items or Likert scales. This work shows two alternatives to the alpha coefficient, the ordinal alpha and beta coefficients. Both of them take into account the ordinal nature of the data. The new coefficients were applied to several scales and compared with the traditional estimation of alpha. This paper also shows how to estimate those coefficients.     

Further evidence for Coren and Hakstian’s “Methodological implications of interaural correlation: Count heads not ears” and an alternative correction formula

Coren and Hakstian (1990) identified a serious methodological problem that arises in auditory research because of interaural correlation. When measures from both ears of the subjects are pooled together in an experimental design that assumes independence of measures, there can be spuriously high apparent statistical significance. The present paper provides further evidence in support of Coren and Hakstian’s argument and also derives a formula that effectively corrects inflated test statistics resulting from interaural correlation. This formula is a special case of a more general one that applies in many other experimental contexts in which nonindependence of measures is a problem. We found that statistical tests based on our formula have somewhat greater power to detect differences than the kind of correction method advocated by Coren and Hakstian.