Sample size determinations for the two rater kappa statistic

This paper gives a method for determining a sample size that will achieve a prespecified bound on confidence interval width for the interrater agreement measure,. The same results can be used when a prespecified power is desired for testing hypotheses about the value of kappa. An example from the literature is used to illustrate the methods proposed here.     

Application of the bootstrap methods in factor analysis

A Monte Carlo experiment is conducted to investigate the performance of the bootstrap methods in normal theory maximum likelihood factor analysis both when the distributional assumption is satisfied and unsatisfied. The parameters and their functions of interest include unrotated loadings, analytically rotated loadings, and unique variances. The results reveal that (a) bootstrap bias estimation performs sometimes poorly for factor loadings and nonstandardized unique variances; (b) bootstrap variance estimation performs well even when the distributional assumption is violated; (c) bootstrap confidence intervals based on the Studentized statistics are recommended; (d) if structural hypothesis about the population covariance matrix is taken into account then the bootstrap distribution of the normal theory likelihood ratio test statistic is close to the corresponding sampling distribution with slightly heavier right tail.This study was carried out in part under the ISM cooperative research program (91-ISM · CRP-85, 92-ISM · CRP-102). The authors would like to thank the editor and three reviewers for their helpful comments and suggestions which improved the quality of this paper considerably.     

Some neglected problems in IRT

The paper addresses three neglected questions from IRT. In section 1, the properties of the “measurement” of ability or trait parameters and item difficulty parameters in the Rasch model are discussed. It is shown that the solution to this problem is rather complex and depends both on general assumptions about properties of the item response functions and on assumptions about the available item universe. Section 2 deals with the measurement of individual change or “modifiability” based on a Rasch test. A conditional likelihood approach is presented that yields (a) an ML estimator of modifiability for given item parameters, (b) allows one to test hypotheses about change by means of a Clopper-Pearson confidence interval for the modifiability parameter, or (c) to estimate modifiability jointly with the item parameters. Uniqueness results for all three methods are also presented. In section 3, the Mantel-Haenszel method for detecting DIF is discussed under a novel perspective: What is the most general framework within which the Mantel-Haenszel method correctly detects DIF of a studied item? The answer is that this is a 2PL model where, however, all discrimination parameters are known and the studied item has the same discrimination in both populations. Since these requirements would hardly be satisfied in practical applications, the case of constant discrimination parameters, that is, the Rasch model, is the only realistic framework. A simple Pearsonx ² test for DIF of one studied item is proposed as an alternative to the Mantel-Haenszel test; moreover, this test is generalized to the case of two items simultaneously studied for DIF.     

On normal theory based inference for multilevel models with distributional violations

Data in social and behavioral sciences are often hierarchically organized though seldom normal, yet normal theory based inference procedures are routinely used for analyzing multilevel models. Based on this observation, simple adjustments to normal theory based results are proposed to minimize the consequences of violating normality assumptions. For characterizing the distribution of parameter estimates, sandwich-type covariance matrices are derived. Standard errors based on these covariance matrices remain consistent under distributional violations. Implications of various covariance estimators are also discussed. For evaluating the quality of a multilevel model, a rescaled statistic is given for both the hierarchical linear model and the hierarchical structural equation model. The rescaled statistic, improving the likelihood ratio statistic by estimating one extra parameter, approaches the same mean as its reference distribution. A simulation study with a 2-level factor model implies that the rescaled statistic is preferable.This research was supported by grants DA01070 and DA00017 from the National Institute on Drug Abuse and a University of North Texas faculty research grant. We would like to thank the Associate Editor and two reviewers for suggestions that helped to improve the paper.     

认知诊断模型中项目水平模型比较统计量的健壮性

使用模拟研究方法比较了以往研究中提出的基于观察信息矩阵、三明治矩阵的Wald（分别表示为W_Obs、W_Sw）、似然比（Likelihood Ratio）统计量以及新提出的基于经验交叉相乘信息矩阵的Wald统计量（W_XPD）在模型——数据失拟条件下进行项目水平上模型比较时的表现。结果显示：（1）W_Sw的一类错误控制率有很强的健壮性。（2）W_XPD在Q矩阵错误设定的大多数条件下的表现优于W_Sw。结论：模型—数据拟合良好时可以使用W_Sw进行项目水平上的模型比较,当模型与数据失拟时W_XPD可能是更好的选择。     

认知诊断模型Q矩阵修正：完整信息矩阵的作用

Q矩阵是CDM的核心元素之一,反映了测验的内部结构和内容设计,通常由领域专家根据经验进行主观界定,因此需要对可能存在的错误进行修正。本研究提出了一种新的Q矩阵修正方法——基于完整经验交叉相乘信息矩阵的Wald-XPD方法。采用Monte Carlo模拟检验了新方法的表现,并与同类方法进行了比较。研究表明：新开发的Wald-XPD方法在Q矩阵恢复率、保留正确标定属性的比例以及修正错误标定属性的比例这3个主要指标上均有较好的表现,且整体上优于其他方法,尤其是在修正错误标定的属性方面。通过实证数据展示了Wald-XPD方法在Q矩阵修正中的良好表现。总之,本研究为Q矩阵修正提供了有效的方法。     

Stepwise variable selection in factor analysis

It is very important to choose appropriate variables to be analyzed in multivariate analysis when there are many observed variables such as those in a questionnaire. What is actually done in scale construction with factor analysis is nothing but variable selection.In this paper, we take several goodness-of-fit statistics as measures of variable selection and develop backward elimination and forward selection procedures in exploratory factor analysis. Once factor analysis is done for a certain numberp of observed variables (thep-variable model is labeled the current model), simple formulas for predicted fit measures such as chi-square, GFI, CFI, IFI and RMSEA, developed in the field of the structural equation modeling, are provided for all models obtained by adding an external variable (so that the number of variables isp + 1) and for those by deleting an internal variable (so that the number isp – 1), provided that the number of factors is held constant.A programSEFA (Stepwise variable selection in Exploratory Factor Analysis) is developed to actually obtain a list of the fit measures for all such models. The list is very useful in determining which variable should be dropped from the current model to improve the fit of the current model. It is also useful in finding a suitable variable that may be added to the current model. A model with more appropriate variables makes more stable inference in general.The criteria traditionally often used for variable selection is magnitude of communalities. This criteria gives a different choice of variables and does not improve fit of the model in most cases.The URL of the programSEFA is http://koko15.hus.osaka-u.ac.jp/~harada/factor/stepwise/.     

The use of item scores and response times to detect examinees who may have benefited from item preknowledge

According to Wollack and Schoenig (2018, The Sage encyclopedia of educational research, measurement, and evaluation. Thousand Oaks, CA: Sage, 260), benefiting from item preknowledge is one of the three broad types of test fraud that occur in educational assessments. We use tools from constrained statistical inference to suggest a new statistic that is based on item scores and response times and can be used to detect examinees who may have benefited from item preknowledge for the case when the set of compromised items is known. The asymptotic distribution of the new statistic under no preknowledge is proved to be a simple mixture of two χ² distributions. We perform a detailed simulation study to show that the Type I error rate of the new statistic is very close to the nominal level and that the power of the new statistic is satisfactory in comparison to that of the existing statistics for detecting item preknowledge based on both item scores and response times. We also include a real data example to demonstrate the usefulness of the suggested statistic.     

心理学研究中的可重复性问题:从危机到契机

可重复性问题是当前科学界面临的共同问题。最近,心理学研究领域的可重复性问题也受到广泛关注,引起了研究者的积极讨论与探索。通过对2008年发表的100项研究结果进行大规模重复实验,研究者发现,心理学研究的成功重复率约为39%,但该研究仍然存在着巨大的争议,不同的研究者对其结果的解读不尽相同。针对可重复性问题,研究者通过数据模拟、元分析以及调查等多种方法来分析和探索其原因,这些研究表明,可重复性问题本质上可能是发表的研究假阳性过高,可疑研究操作是假阳性过高的直接原因,而出版偏见和过度依赖虚无假设检验则是更加深层的原因。面对可重复性问题,研究者从统计方法和研究实践两个方面提出了相应的解决方案,这些方法与实践正在成为心理学研究的新标准。然而,要解决可重复性问题,还需要心理学研究领域的多方参与,尤其是在政策上鼓励公开、透明和开放的研究取向,避免出版偏见。心理学研究者为解决可重复性问题做出的努力,不仅会加强心理学研究的可靠性,也为其他学科解决可重复问题提供了借鉴,推动科学界可重复问题的解决。     

薛学潜易统计方程式推导存在的问题探讨

民国薛学潜先生作为科学易学的代表人物,打通了传统易学与西方现代科学的隔阂,建立了一个相对缜密的薛氏科学易体系。但由于受诸多因素制约,其推导仍有许多不足,甚至有对物理模型的模糊认识,导致了一些结论不能令人信服。本文以易统计方程式为例进行探讨,发现易统计方程式总摄三种物理统计有些牵强,其推导过程也颇有缺陷。