Modeling and Testing Differential Item Functioning in Unidimensional Binary Item Response Models with a Single Continuous Covariate: A Functional Data Analysis Approach

Differential item functioning (DIF), referring to between-group variation in item characteristics above and beyond the group-level disparity in the latent variable of interest, has long been regarded as an important item-level diagnostic. The presence of DIF impairs the fit of the single-group item response model being used, and calls for either model modification or item deletion in practice, depending on the mode of analysis. Methods for testing DIF with continuous covariates, rather than categorical grouping variables, have been developed; however, they are restrictive in parametric forms, and thus are not sufficiently flexible to describe complex interaction among latent variables and covariates. In the current study, we formulate the probability of endorsing each test item as a general bivariate function of a unidimensional latent trait and a single covariate, which is then approximated by a two-dimensional smoothing spline. The accuracy and precision of the proposed procedure is evaluated via Monte Carlo simulations. If anchor items are available, we proposed an extended model that simultaneously estimates item characteristic functions (ICFs) for anchor items, ICFs conditional on the covariate for non-anchor items, and the latent variable density conditional on the covariate—all using regression splines. A permutation DIF test is developed, and its performance is compared to the conventional parametric approach in a simulation study. We also illustrate the proposed semiparametric DIF testing procedure with an empirical example.     

Handling Missing Covariates in Conditional Mixture Models Under Missing at Random Assumptions

Mixture modeling is a popular method that accounts for unobserved population heterogeneity using multiple latent classes that differ in response patterns. Psychologists use conditional mixture models to incorporate covariates into between-class and/or within-class regressions. Although psychologists often have missing covariate data, conditional mixtures are currently fit with a conditional likelihood, treating covariates as fixed and fully observed. Under this exogenous-x approach, missing covariates are handled primarily via listwise deletion. This sacrifices efficiency and does not allow missingness to depend on observed outcomes. Here we describe a modified joint likelihood approach that (a) allows inference about parameters of the exogenous-x conditional mixture even with nonnormal covariates, unlike a conventional multivariate mixture; (b) retains all cases under missing at random assumptions; (c) yields lower bias and higher efficiency than the exogenous-x approach under a variety of conditions with missing covariates; and (d) is straightforward to implement in available commercial software. The proposed approach is illustrated with an empirical analysis predicting membership in latent classes of conduct problems. Recommendations for practice are discussed.     

A joint modeling framework of responses and response times to assess learning outcomes

Abstract

A general modeling framework of response accuracy and response times is proposed to track skill acquisition and provide additional diagnostic information on the change of latent speed in a learning environment. This framework consists of two types of models: a dynamic response model that captures the response accuracy and the change of discrete latent attribute profile upon factors such as practice, intervention effects, and other latent and observable covariates, and a dynamic response time model that describes the change of the continuous response latency due to change of latent attribute profile. These two types of models are connected through a parameter, describing the change rate of the latent speed through the learning process, and a covariate defined as a function of the latent attribute profile. A Bayesian estimation procedure is developed to calibrate the model parameters and measure the latent variables. The estimation algorithm is evaluated through several simulation studies under various conditions. The proposed models are applied to a real data set collected through a spatial rotation diagnostic assessment paired with learning tools.     

The mirror effect and the spacing effect

In the mirror effect, there are fewer false negatives (misses) and false positives (false alarms) for rare (low-frequency) words than for common (high-frequency) words. In the spacing effect, recognition accuracy is positively related to the interval (spacing or lag) between two presentations of an item. These effects are related in that they are both manifestations of a leapfrog effect (a weaker item jumps over a stronger item). They seem to be puzzles for traditional strength theory and at least some current global-matching models. A computational strength-based model (EICL) is proposed that incorporates excitation, inhibition, and a closed-loop learning algorithm. The model consists of three nonlinear coupled stochastic difference equations, one each for excitation (x), inhibition (y), and context (z). Strength is the algebraic sum (i.e., s = x − y + z). These equations are used to form a toy lexicon that serves as a basis for the experimental manipulations. The model can simulate the mirror effect forcedchoice inequalities and the spacing effect for single-item recognition, all parameters are random variables, and the same parameter values are used for both the mirror and the spacing effects. No parameter values varied with the independent variables (word frequency for the mirror effect, lag for the spacing effect), so the model, not the parameters, is doing the work.     

A Propensity Score Adjustment for Multiple Group Structural Equation Modeling

In the behavioral and social sciences, quasi-experimental and observational studies are used due to the difficulty achieving a random assignment. However, the estimation of differences between groups in observational studies frequently suffers from bias due to differences in the distributions of covariates. To estimate average treatment effects when the treatment variable is binary, Rosenbaum and Rubin (1983a) proposed adjustment methods for pretreatment variables using the propensity score. However, these studies were interested only in estimating the average causal effect and/or marginal means. In the behavioral and social sciences, a general estimation method is required to estimate parameters in multiple group structural equation modeling where the differences of covariates are adjusted. We show that a Horvitz–Thompson-type estimator, propensity score weighted M estimator (PWME) is consistent, even when we use estimated propensity scores, and the asymptotic variance of the PWME is shown to be less than that with true propensity scores. Furthermore, we show that the asymptotic distribution of the propensity score weighted statistic under a null hypothesis is a weighted sum of independent χ² ₁ variables. We show the method can compare latent variable means with covariates adjusted using propensity scores, which was not feasible by previous methods. We also apply the proposed method for correlated longitudinal binary responses with informative dropout using data from the Longitudinal Study of Aging (LSOA). The results of a simulation study indicate that the proposed estimation method is more robust than the maximum likelihood (ML) estimation method, in that PWME does not require the knowledge of the relationships among dependent variables and covariates.     

A model for realism of confidence judgments: Implications for underconfidence in sensory discrimination

In a recent issue of this journal, Björkman, Juslin, and Winman (1993) presented a model of the calibration of subjective confidence judgments for sensory discrimination which they called “subjective distance theory.” They proposed that there was a robust underconfidence bias in such judgments, that the model predicted such a bias, and that two different models were needed for the calibration of subjective confidence for cognitive judgments and for sensory ones. This paper addresses issues they raised. It points out that they have not presented a new model, but rather a portion of a more general one, the “decision-variable partition model” originally proposed in Ferrell and McGoey (1980). This paper explores properties of the model and shows, contrary to Björkman, Juslin, and Winman’s hypotheses, that the model does not predict under-confidence, that the “hard-easy effect” can be observed with sensory discriminations, and that the model fits not only sensory, but also cognitive judgments.     

A profile of adolescent alcohol offenders in two rural Midwestern counties

Screening tools specifically developed for use with adolescents may be more sensitive predictors of relapse or recidivism than self-report inventories typically used to screen adults. 70 adolescents in a program for drunk drivers in two counties in southeastern Nebraska were given both the CRAFFT and the Alcohol Use Disorder Identification Test questionnaires during routine alcohol-dependency evaluations. The Michigan Alcoholism Screening Test was also given to 28 subjects selected at random. 11 boys and 6 girls did not successfully complete the program. Significant correlations obtained for AUDIT scores for both the CRAFFT (r69=.65, p<.01) and failure to complete diversion (r69=.23, p<.05). Subjects were grouped by age (18 and younger and over 18 years) and by sex. A 2 x 2 analysis of variance for scores on the AUDIT indicated significant main effects for both age (F1,66=4.86, p<.05) and sex (F1.66=5.96, p<.01). MAST and CRAFFT scores showed no age or sex differences. The AUDIT might be included in drug and alcohol assessments with similar samples of adolescents.     

协单参数单维度Rasch模型的优势与意义

相比多参数多维度IRT模型通过增加参数的方式来提升模型拟合度和解释度，Rasch模型流派强调“理论驱动研究”和“数据符合模型”，推崇单参数单维度的测量模型能最大限度地减少额外因素对真实测量目的的影响和干扰，从而保证测量的客观性和准确性。Rasch模型关注测量目标与测量工具的对应关系，它的“简单”特性有助于研究者更准确地评估和解释被测目标与测量工具间的适配性，且在将非线性数据转化为等距数据时具有天然的优势。     

Generalized graded unfolding model with structural equation for subject parameters

The generalized graded unfolding model (GGUM) is capable of analyzing polytomous scored, unfolding data such as agree‐disagree responses to attitude statements. In the present study, we proposed a GGUM with structural equation for subject parameters, which enabled us to evaluate the relation between subject parameters and covariates and/or latent variables simultaneously, in order to avoid the influence of attenuation. Additionally, an algorithm for parameter estimation is newly implemented via the Markov Chain Monte Carlo (MCMC) method, based on Bayesian statistics. In the simulation, we compared the accuracy of estimates of regression coefficients between the proposed model and a conventional method using a GGUM (where regression coefficients are estimated using estimates of θ). As a result, the proposed model performed much better than the conventional method in terms of bias and root mean squared errors of estimates of regression coefficients. The study concluded by verifying the efficacy of the proposed model, using an actual data example of attitude measurement.     

Accounting for belief bias in a mental model framework: comment on Klauer, Musch, and Naumer (2000)

K. C. Klauer, J. Musch, and B. Naumer (2000; see record 2000-02818-008) presented a general multinomial model of belief bias effects in syllogistic reasoning. They claimed to map a particular mental model account of belief bias (J. V. Oakhill, P. N. Johnson-Laird, & A. Garnham, 1989; see record 1989-38845-001)) onto this model and to show empirically that it is incorrect. The authors argue that this mental model account does not map onto the multinomial model and that it can account for the data presented by Klauer et al. (Experiments 1-4). The authors further argue that additional data Klauer et al. presented in support of a new model of their own (Experiments 5-8) are explained by this mental model account. The mental model account is, therefore, refuted neither by Klauer et al.'s theoretical analysis nor by any of the results they presented. Furthermore, the account can accommodate more recent findings on belief bias in a more satisfactory way than can alternative models that have been proposed.     

Abstract: Evaluation of Test Statistics for Robust Structural Equation Modeling With Nonnormal Missing Data

Traditional structural equation modeling (SEM) techniques have trouble dealing with incomplete and/or nonnormal data that are often encountered in practice. Yuan and Zhang (2011a) developed a two-stage procedure for SEM to handle nonnormal missing data and proposed four test statistics for overall model evaluation. Although these statistics have been shown to work well with complete data, their performance for incomplete data has not been investigated in the context of robust statistics.

Focusing on a linear growth curve model, a systematic simulation study is conducted to evaluate the accuracy of the parameter estimates and the performance of five test statistics including the naive statistic derived from normal distribution based maximum likelihood (ML), the Satorra-Bentler scaled chi-square statistic (RML), the mean- and variance-adjusted chi-square statistic (AML), Yuan-Bentler residual-based test statistic (CRADF), and Yuan-Bentler residual-based F statistic (RF). Data are generated and analyzed in R using the package rsem (Yuan & Zhang, 2011b).

Based on the simulation study, we can observe the following: (a) The traditional normal distribution-based method cannot yield accurate parameter estimates for nonnormal data, whereas the robust method obtains much more accurate model parameter estimates for nonnormal data and performs almost as well as the normal distribution based method for normal distributed data. (b) With the increase of sample size, or the decrease of missing rate or the number of outliers, the parameter estimates are less biased and the empirical distributions of test statistics are closer to their nominal distributions. (c) The ML test statistic does not work well for nonnormal or missing data. (d) For nonnormal complete data, CRADF and RF work relatively better than RML and AML. (e) For missing completely at random (MCAR) missing data, in almost all the cases, RML and AML work better than CRADF and RF. (f) For nonnormal missing at random (MAR) missing data, CRADF and RF work better than AML. (g) The performance of the robust method does not seem to be influenced by the symmetry of outliers.     

Reconsidering "evidence" for fast-and-frugal heuristics

In several recent reviews, authors have argued for the pervasive use of fast-and-frugal heuristics in human judgment. They have provided an overview of heuristics and have reiterated findings corroborating that such heuristics can be very valid strategies leading to high accuracy. They also have reviewed previous work that implies that simple heuristics are actually used by decision makers. Unfortunately, concerning the latter point, these reviews appear to be somewhat incomplete. More important, previous conclusions have been derived from investigations that bear some noteworthy methodological limitations. I demonstrate these by proposing a new heuristic and provide some novel critical findings. Also, I review some of the relevant literature often not—or only partially—considered. Overall, although some fast-and-frugal heuristics indeed seem to predict behavior at times, there is little to no evidence for others. More generally, the empirical evidence available does not warrant the conclusion that heuristics are pervasively used.     

Improving the Accuracy of Group Judgment: A Process Intervention Combining Group Facilitation,Social Judgment Analysis,and Information Technology

Interacting groups fail to make judgments as accurate as those of their most capable members due to problems associated with both interaction processes and cognitive processing. Group process techniques and decision analytic tools have been used with groups to combat these problems. While such techniques and tools do improve the quality of group judgment, they have not enabled groups to make judgments more accurate than those of their most capable members on tasks that evoke a great deal of systematic bias. A new intervention procedure that integrates group facilitation, social judgment analysis, and information technology was developed to overcome more fully the problems typically associated with interaction processes and cognitive processing. The intervention was evaluated by testing the hypothesis that groups using this new procedure can establish judgment policies for cognitive conflict tasks that are more accurate than the ones produced by any of their members. An experiment involving 16 four- and five-member groups was conducted to compare the accuracy of group judgments with the accuracy of the judgments of the most capable group member. A total of 96 participants (48 males and 48 females) completed the individual part of the task; 71 of these participants worked in groups. Results indicated that the process intervention enabled small, interacting groups to perform significantly better than their most capable members on two cognitive conflict tasks (p < .05). The findings suggest that Group Decision Support Systems that integrate facilitation, social judgment analysis, and information technology should be used to improve the accuracy of group judgment.     

Using analysis of covariance (ANCOVA) with fallible covariates

Analysis of covariance (ANCOVA) is used widely in psychological research implementing nonexperimental designs. However, when covariates are fallible (i.e., measured with error), which is the norm, researchers must choose from among 3 inadequate courses of action: (a) know that the assumption that covariates are perfectly reliable is violated but use ANCOVA anyway (and, most likely, report misleading results); (b) attempt to employ 1 of several measurement error models with the understanding that no research has examined their relative performance and with the added practical difficulty that several of these models are not available in commonly used statistical software; or (c) not use ANCOVA at all. First, we discuss analytic evidence to explain why using ANCOVA with fallible covariates produces bias and a systematic inflation of Type I error rates that may lead to the incorrect conclusion that treatment effects exist. Second, to provide a solution for this problem, we conduct 2 Monte Carlo studies to compare 4 existing approaches for adjusting treatment effects in the presence of covariate measurement error: errors-in-variables (EIV; Warren, White, & Fuller, 1974), Lord's (1960) method, Raaijmakers and Pieters's (1987) method (R&P), and structural equation modeling methods proposed by S?rbom (1978) and Hayduk (1996). Results show that EIV models are superior in terms of parameter accuracy, statistical power, and keeping Type I error close to the nominal value. Finally, we offer a program written in R that performs all needed computations for implementing EIV models so that ANCOVA can be used to obtain accurate results even when covariates are measured with error.     

A two-stage logistic regression model for analyzing inter-rater agreement

Studies of agreement commonly occur in psychiatric research. For example, researchers are often interested in the agreement among radiologists in their review of brain scans of elderly patients with dementia or in the agreement among multiple informant reports of psychopathology in children. In this paper, we consider the agreement between two raters when rating a dichotomous outcome (e.g., presence or absence of psychopathology). In particular, we consider logistic regression models that allow agreement to depend on both rater- and subject-level covariates. Logistic regression has been proposed as a simple method for identifying covariates that are predictive of agreement (Coughlin et al., 1992). However, this approach is problematic since it does not take account of agreement due to chance alone. As a result, a spurious association between the probability (or odds) of agreement and a covariate could arise due entirely to chance agreement. That is, if the prevalence of the dichotomous outcome varies among subgroups of the population, then covariates that identify the subgroups may appear to be predictive of agreement. In this paper we propose a modification to the standard logistic regression model in order to take proper account of chance agreement. An attractive feature of the proposed method is that it can be easily implemented using existing statistical software for logistic regression. The proposed method is motivated by data from the Connecticut Child Study (Zahner et al., 1992) on the agreement among parent and teacher reports of psychopathology in children. In this study, parents and teachers provide dichotomous assessments of a child's psychopathology and it is of interest to examine whether agreement among the parent and teacher reports is related to the age and gender of the child and to the time elapsed between parent and teacher assessments of the child.The authors thank the Associate Editor and the referees for their helpful comments and suggestions. We also thank Gwen Zahner for use of data from the Connecticut Child Study, which was conducted under contract to the Connecticut Department of Children and Youth Services. This research was supported by grants HL 69800, AHRQ 10871, HL52329, HL61769, GM 29745, MH 54693 and MH 17119 from the National Institutes of Health.     

Model comparison of nonlinear structural equation models with fixed covariates

Recently, it has been recognized that the commonly used linear structural equation model is inadequate to deal with some complicated substantive theory. A new nonlinear structural equation model with fixed covariates is proposed in this article. A procedure, which utilizes the powerful path sampling for computing the Bayes factor, is developed for model comparison. In the implementation, the required random observations are simulated via a hybrid algorithm that combines the Gibbs sampler and the Metropolis-Hastings algorithm. It is shown that the proposed procedure is efficient and flexible; and it produces Bayesian estimates of the parameters, latent variables, and their highest posterior density intervals as by-products. Empirical performances of the proposed procedure such as sensitivity to prior inputs are illustrated by a simulation study and a real example.This research is fully supported by a grant from the Research Grant Council of the Hong Kong Special Administrative Region, China (Project No. CUHK 4346/01H). The authors are thankful to the Editor, the Associate Editor, and anonymous reviewers for valuable comments which improve the paper significantly, and grateful to ICPSR and the relevant funding agency for allowing use of the data in the example. The assistance of Michael K.H. Leung and Esther L.S. Tam is gratefully acknowledged.     

Cognitive diagnosis models for multiple strategies

Cognitive diagnosis models (CDMs) have been used as psychometric tools in educational assessments to estimate students’ proficiency profiles. However, most CDMs assume that all students adopt the same strategy when approaching problems in an assessment, which may not be the case in practice. This study develops a generalized multiple-strategy CDM for dichotomous response data. The proposed model provides a unified framework to accommodate various condensation rules (e.g., conjunctive, disjunctive, and additive) and different strategy selection approaches (i.e., probability-matching, over-matching, and maximizing). Model parameters are estimated using the marginal maximum likelihood estimation via expectation-maximization algorithm. Simulation studies showed that the parameters of the proposed model can be adequately recovered and that the proposed model was relatively robust to some types of model misspecifications. A set of real data was analysed as well to illustrate the use of the proposed model in practice.     

Calibration of sensory and cognitive judgments: A single model for both

In a recent issue of this journal, Winman and Juslin (34 , 135–148, 1993) present a model of the calibration of subjective probability judgments for sensory discrimination tasks. They claim that the model predicts a pervasive underconfidence bias observed in such tasks, and present evidence from a training experiment that they interpret as supporting the notion that different models are needed to describe judgment of confidence in sensory and in cognitive tasks. The model is actually part of the more comprehensive decision variable partition model of subjective probability calibration that was originally proposed in Ferrell and McGoey (Organizational Behavior and Human Performance, 26 , 32–53, 1980). The characteristics of the model are described and it is demonstrated that the model does not predict underconfidence, that it is fully compatible with the overconfidence frequently found in calibration studies with cognitive tasks, and that it well represents experimental results from such studies. It is concluded that only a single model is needed for both types of task.     

Intelligent management of intelligence agencies: beyond accountability ping-pong

The intelligence community (IC) is asked to predict outcomes that may often be inherently unpredictable-and is blamed for the inevitable forecasting failures, be they false positives or false negatives. To move beyond blame games of accountability ping-pong that incentivize bureaucratic symbolism over substantive reform, it is necessary to reach bipartisan agreements on performance indicators that are transparent enough to reassure clashing elites (to whom the IC must answer) that estimates have not been politicized. Establishing such transideological credibility requires (a) developing accuracy metrics for decoupling probability and value judgments; (b) using the resulting metrics as criterion variables in validity tests of the IC's selection, training, and incentive systems; and (c) institutionalizing adversarial collaborations that conduct level-playing-field tests of clashing perspectives.     

A generalized item response tree model for psychological assessments

A new item response theory (IRT) model with a tree structure has been introduced for modeling item response processes with a tree structure. In this paper, we present a generalized item response tree model with a flexible parametric form, dimensionality, and choice of covariates. The utilities of the model are demonstrated with two applications in psychological assessments for investigating Likert scale item responses and for modeling omitted item responses. The proposed model is estimated with the freely available R package flirt (Jeon et al., 2014b).