On the Usefulness of a Multilevel Logistic Regression Approach to Person-Fit Analysis

The logistic person response function (PRF) models the probability of a correct response as a function of the item locations. Reise (2000) Reise, S. P. 2000. Using multilevel logistic regression to evaluate person-fit in IRT models. Multivariate Behavioral Research, 35: 543–568. [Taylor &; Francis Online], [Web of Science ®] , [Google Scholar] proposed to use the slope parameter of the logistic PRF as a person-fit measure. He reformulated the logistic PRF model as a multilevel logistic regression model and estimated the PRF parameters from this multilevel framework. An advantage of the multilevel framework is that it allows relating person fit to explanatory variables for person misfit/fit. We critically discuss Reise's approach. First, we argue that often the interpretation of the PRF slope as an indicator of person misfit is incorrect. Second, we show that the multilevel logistic regression model and the logistic PRF model are incompatible, resulting in a multilevel person-fit framework, which grossly violates the bivariate normality assumption for residuals in the multilevel model. Third, we use a Monte Carlo study to show that in the multilevel logistic regression framework estimates of distribution parameters of PRF intercepts and slopes are biased. Finally, we discuss the implications of these results and suggest an alternative multilevel regression approach to explanatory person-fit analysis. We illustrate the alternative approach using empirical data on repeated anxiety measurements of cardiac arrhythmia patients who had a cardioverter-defibrillator implanted.     

Using Multilevel Logistic Regression to Evaluate Person-Fit in IRT Models

I describe how multilevel logistic regression can be used to assess the consistency of an individual's response pattern with an item response theory measurement model. Specifically, by treating item responses as being nested within individuals, multilevel logistic regression is used to estimate a person-response curve that models how an individual's item endorsement rate decreases as a function of item difficulty. The slope of an individual's person-response curve is used as an indicator of the degree of response consistency or person-fit. I argue that the proposed multilevel modeling approach to person-fit assessment has several potential advantages over traditional techniques. The most important advantage being that the multilevel modeling approach allows explanatory variables to be entered into the model so that the causes of response inconsistency or differential test functioning can be investigated.     

Multilevel modeling in industrial and personnel psychology

Many of the topics of interest in the social and behavior sciences are often hierarchical or multilevel in nature. These multiple levels (e.g., individual versus group) create problems for researchers related to the choice of measurement and analysis. Recent innovations in statistical analysis have made it possible to account for the hierarchical nature of observations. Therefore, in this article we begin with a review of multilevel analysis techniques and discuss advances that have been made in the social sciences using multilevel models. Next, we summarize contemporary research specific to the organizational psychology literature that uses multilevel analysis. Possible applications for industrial and personnel psychology are then discussed. Guidelines for determining if multilevel analysis is appropriate for a given applied research project are provided. We conclude with a summary and call for increased use of multilevel analysis in industrial and personnel psychology.     

A New Multilevel CART Algorithm for Multilevel Data with Binary Outcomes

The multilevel logistic regression model (M-logit) is the standard model for modeling multilevel data with binary outcomes. However, many assumptions and restrictions should be considered when applying this model for unbiased estimation. To overcome these limitations, we proposed a multilevel CART (M-CART) algorithm which combines the M-logit and single level CART (S-CART) within the framework of the expectation-maximization. Simulation results showed that the proposed M-CART provided substantial improvements on classification accuracy, sensitivity, and specific over the M-logit, S-CART, and single level logistic regression model when modeling multilevel data with binary outcomes. This benefit of using M-CART was consistently found across different conditions of sample size, intra-class correlation, and when relationships between predictors and outcomes were nonlinear and nonadditive.     

Multilevel modeling in industrial and personnel psychology

Many of the topics of interest in the social and behavior sciences are often hierarchical or multilevel in nature. These multiple levels (e.g., individual versus group) create problems for researchers related to the choice of measurement and analysis. Recent innovations in statistical analysis have made it possible to account for the hierarchical nature of observations. Therefore, in this article we begin with a review of multilevel analysis techniques and discuss advances that have been made in the social sciences using multilevel models. Next, we summarize contemporary research specific to the organizational psychology literature that uses multilevel analysis. Possible applications for industrial and personnel psychology are then discussed. Guidelines for determining if multilevel analysis is appropriate for a given applied research project are provided. We conclude with a summary and call for increased use of multilevel analysis in industrial and personnel psychology.     

Multilevel Multivariate Meta-analysis with Application to Choice Overload

We introduce multilevel multivariate meta-analysis methodology designed to account for the complexity of contemporary psychological research data. Our methodology directly models the observations from a set of studies in a manner that accounts for the variation and covariation induced by the facts that observations differ in their dependent measures and moderators and are nested within, for example, papers, studies, groups of subjects, and study conditions. Our methodology is motivated by data from papers and studies of the choice overload hypothesis. It more fully accounts for the complexity of choice overload data relative to two prior meta-analyses and thus provides richer insight. In particular, it shows that choice overload varies substantially as a function of the six dependent measures and four moderators examined in the domain and that there are potentially interesting and theoretically important interactions among them. It also shows that the various dependent measures have differing levels of variation and that levels up to and including the highest (i.e., the fifth, or paper, level) are necessary to capture the variation and covariation induced by the nesting structure. Our results have substantial implications for future studies of choice overload.     

Maximizing,choice freedom,and duration judgments in choice making

This paper examines (a) whether people are less accurate in judging choice time as choice tasks involve more choice options, more choice information, or a combination of both and (b) whether people with a higher tendency to look for the best option in choice making (maximizers) have less accurate duration judgments of choice time as compared to people who are more easily settling for a choice outcome that is satisfactory (satisficers). A multilevel analysis is used to explore the relationships of interest using data collected through a series of choice tasks. In general, the results suggest that people seem to misjudge time durations when making choices. Moreover, empirical evidence demonstrates that people with an outspoken tendency to maximize in choice making do not differ significantly in estimating choice time accurately as compared to people who experience almost no need to maximize.     

Unravelling socioeconomic school composition effects on higher education enrollment: the role of students’ individual and shared feelings of futility and self-efficacy

Despite the massification of higher education, social inequality in higher education outcomes still persists. It is known that secondary schools’ socioeconomic composition is at least partly responsible for this social inequality. Nevertheless, the mechanisms by which socioeconomic composition affects higher education enrollment are still poorly understood. Based on the attribution theory, this study expects students’ feelings of academic self-efficacy and sense of futility to act as underlying mechanisms. Additionally, given that individual self-efficacy beliefs and individual sense of futility are shared among students within schools — that is, schools have different self-efficacy and futility cultures — we investigate whether these cultures are consequential for higher education enrollment and program choice. Results of binomial logistic multilevel analyses on the data of the International Study of City Youth (www.iscy.org) in Ghent, consisting of 1214 Flemish students across 29 schools, indicated that students in low socioeconomic composition schools are less likely to enroll in higher education because these schools are characterized by high futility cultures. In contrast, high self-efficacy and self-efficacy cultures are positively associated with the choice for academic bachelor programs, but they did not explain socioeconomic composition effects. We conclude that educational policies should aim at increasing a sense of control over academic outcomes in order to improve social equality in higher education outcomes.

     

An approach to combine the logistic threshold model of psychophysics with the Bradley-Terry-Luce models of choice theory

The paper presents a straightforward extension of the Bradley-Terry-Luce model (BTL model) that can be derived from the logistic threshold model of psychophysics which assumes that psychometric functions are logistic probability functions. It is shown that (under weak side conditions) the logistic threshold model is a submodel of the extended BTL model. Moreover, representation and uniqueness theorems are proven that provide some evidence that the extended BTL model is a useful and widely applicable generalization of the ordinary BTL model. Finally, the logistic shape of the psychometric function is derived from axioms about binary choice probabilities. This characterization of the logistic threshold model can replace goodness of fit tests for the logistic probability distribution.     

Use of multilevel logistic regression to identify the causes of differential item functioning

Given that a key function of tests is to serve as evaluation instruments and for decision making in the fields of psychology and education, the possibility that some of their items may show differential behaviour is a major concern for psychometricians. In recent decades, important progress has been made as regards the efficacy of techniques designed to detect this differential item functioning (DIF). However, the findings are scant when it comes to explaining its causes. The present study addresses this problem from the perspective of multilevel analysis. Starting from a case study in the area of transcultural comparisons, multilevel logistic regression is used: 1) to identify the item characteristics associated with the presence of DIF; 2) to estimate the proportion of variation in the DIF coefficients that is explained by these characteristics; and 3) to evaluate alternative explanations of the DIF by comparing the explanatory power or fit of different sequential models. The comparison of these models confirmed one of the two alternatives (familiarity with the stimulus) and rejected the other (the topic area) as being a cause of differential functioning with respect to the compared groups.     

Transition and protective agency of early childhood learning behaviors as portents of later school attendance and adjustment

This article reports on the study of differential change trajectories for early childhood learning behaviors as they relate to future classroom adjustment and school attendance. A large sample (N = 2152) of Head Start children was followed through prekindergarten, kindergarten, and 1st grade. Classroom learning behaviors were assessed twice each year by teachers who observed gradual declines in Competence Motivation and Attentional Persistence as children transitioned through schooling. Cross-classified multilevel growth models revealed distinct transitional pathways for future adjustment versus maladjustment and sporadic versus chronic absenteeism. Generalized multilevel logistic modeling and receiver operating characteristic curve analyses showed that teachers' earliest assessments were substantially predictive of eventual good classroom adjustment and school attendance, with increasing accuracy for prediction of future sociobehavioral adjustment as time progressed.     

Modeling Clustered Data with Very Few Clusters

Small-sample inference with clustered data has received increased attention recently in the methodological literature, with several simulation studies being presented on the small-sample behavior of many methods. However, nearly all previous studies focus on a single class of methods (e.g., only multilevel models, only corrections to sandwich estimators), and the differential performance of various methods that can be implemented to accommodate clustered data with very few clusters is largely unknown, potentially due to the rigid disciplinary preferences. Furthermore, a majority of these studies focus on scenarios with 15 or more clusters and feature unrealistically simple data-generation models with very few predictors. This article, motivated by an applied educational psychology cluster randomized trial, presents a simulation study that simultaneously addresses the extreme small sample and differential performance (estimation bias, Type I error rates, and relative power) of 12 methods to account for clustered data with a model that features a more realistic number of predictors. The motivating data are then modeled with each method, and results are compared. Results show that generalized estimating equations perform poorly; the choice of Bayesian prior distributions affects performance; and fixed effect models perform quite well. Limitations and implications for applications are also discussed.     

History, Community Milieu, and Christian-Muslim Differentials in Contraceptive Use in Sub-Saharan Africa

In the globalization discourse, Christianity and Islam are often construed as representing two traditions that are conflicted and incompatible. This study engages the "clash of civilizations" discourse by examining Muslim-Christian differentials in the use of modern contraception in Nigeria, where Christians have a much higher contraceptive prevalence, and Tanzania, where Muslims are somewhat more likely to use contraception. Employing data from six nationally representative surveys conducted in the two countries between 1990 and 2004 and multilevel logistic regression, we find that the effects of religion remain strong but operate largely through the community religious milieu. Contraceptive use tends to be highest in religiously mixed areas, but the "optimal" religious makeup differs between the two nations, reflecting the historically shaped configurations of their religious expressions and politics.     

A Comparison of Multilevel Mediation Modeling Methods: Recommendations for Applied Researchers

Multilevel structural equation modeling (MSEM) has been proposed as a valuable tool for estimating mediation in multilevel data and has known advantages over traditional multilevel modeling, including conflated and unconflated techniques (CMM & UMM). Recent methodological research has focused on comparing the three methods for 2-1-1 designs, but in regards to 1-1-1 mediation designs, there are significant gaps in the published literature that prevent applied researchers from making educated decisions regarding which model to employ in their own specific research design. A Monte Carlo study was performed to compare MSEM, UMM, and CMM on relative bias, confidence interval coverage, Type I Error, and power in a 1-1-1 model with random slopes under varying data conditions. Recommendations for applied researchers are discussed and an empirical example provides context for the three methods.     

Predicting variability: Using multilevel modelling to assess differences in variance

Researchers are sometimes interested in the variability of data rather than in their absolute or relative values. An important example of such a situation in social psychology is consensus: the fact that people are more similar to each other in certain conditions. Currently, methods to assess differences in consensus (or variability in general) are not very well‐developed or widely known. We describe an existing tool that allows testing the extent to which variability depends on one or several predictor variables. We explain how multilevel modelling's capacity to model heterogeneity in residual variance can be used to test substantive hypotheses about differences in variance. We illustrate the procedure and warn against potential misuses of multilevel modelling to analyse differences in variability. We also provide MLwiN and SAS PROC MIXED syntax necessary to run this kind of analyses. In some specific, simple cases, SPSS MIXED can also be used. Copyright © 2014 John Wiley & Sons, Ltd.     

On the Multilevel Nature of Meta-Analysis: A Tutorial,Comparison of Software Programs,and Discussion of Analytic Choices

The term “multilevel meta-analysis” is encountered not only in applied research studies, but in multilevel resources comparing traditional meta-analysis to multilevel meta-analysis. In this tutorial, we argue that the term “multilevel meta-analysis” is redundant since all meta-analysis can be formulated as a special kind of multilevel model. To clarify the multilevel nature of meta-analysis the four standard meta-analytic models are presented using multilevel equations and fit to an example data set using four software programs: two specific to meta-analysis (metafor in R and SPSS macros) and two specific to multilevel modeling (PROC MIXED in SAS and HLM). The same parameter estimates are obtained across programs underscoring that all meta-analyses are multilevel in nature. Despite the equivalent results, not all software programs are alike and differences are noted in the output provided and estimators available. This tutorial also recasts distinctions made in the literature between traditional and multilevel meta-analysis as differences between meta-analytic choices, not between meta-analytic models, and provides guidance to inform choices in estimators, significance tests, moderator analyses, and modeling sequence. The extent to which the software programs allow flexibility with respect to these decisions is noted, with metafor emerging as the most favorable program reviewed.     

Hierarchical modeling of paired comparison data

The method of paired comparisons belongs to a small group of techniques that provide explicit information about the consistency of individual and aggregated choices. This article investigates the link between the individual- and group-level judgments by extending R. D. Luce's (1959) model, which was originally developed for individual choice behavior, to a mixed-effects paired comparison model. It is shown that standard multilevel software for binary data can be used to estimate the model. The interpretation of the paired comparison parameters and statistical model tests are discussed in detail. An extensive analysis of an experimental study illustrates the usefulness of a hierarchical approach in modeling multiple pairwise judgments.     

Family structure and community connectedness: Their association with teenage pregnancy in South Africa

This study aimed to examine the relationships between family structure and community connectedness on the one hand, as well as risk for teenage pregnancy on the other. Data were from the South African General Household Surveys 2011, 2012, 2013; N = 279 042, female teenagers = 9.9%, age range 10 to 19; black teenagers = 81%. We applied multilevel binary logistic regression to predict risk for teenage pregnancy by family structure and community affinity variables. Findings suggest teenage female teenagers from single-headed households and with higher numbers of male relatives are at elevated risk for teenage pregnancy. Similarly, female teenagers from families with parental divorce were at higher risk for pregnancy as were those from communities with lower coherence.     

Relative deprivation,national GDP and right-wing populism: A multilevel,multinational study

We conducted a secondary analysis of the 2018 (ninth round) European Social Survey dataset (N = 19,512, nested in 24 countries) to analyse the association between relative deprivation and right-wing populist voting, moderated by participants' income and their country's GDP. A multilevel moderated model showed that relative deprivation, income and GDP had no association with right-wing populist votes by themselves. However, income and GDP moderated the relation between relative deprivation and right-wing populist votes, that was significant for participants with high incomes and for those living in wealthy countries. The strengths, limitations and future developments of the study are discussed.