表现性评价中评分者信度估计方法的比较研究--从相关法、百分比法到概化理论

随着素质教育的推进．表现性评价受到越来越多的重视。影响表现性评价结果信度的一个重要因素是评分者之间的不一致。文章使用模拟数据,在对比评分者一致性的相关法、一致性百分比法和概化系数等各种估计方法的基础上,提出概化理论在表现性评价中评分者信度问题上的应用是理论和实践发展的有益方向。     

On Similarity Coefficients for 2×2 Tables and Correction for Chance

This paper studies correction for chance in coefficients that are linear functions of the observed proportion of agreement. The paper unifies and extends various results on correction for chance in the literature. A specific class of coefficients is used to illustrate the results derived in this paper. Coefficients in this class, e.g. the simple matching coefficient and the Dice/Sørenson coefficient, become equivalent after correction for chance, irrespective of what expectation is used. The coefficients become either Cohen’s kappa, Scott’s pi, Mak’s rho, Goodman and Kruskal’s lambda, or Hamann’s eta, depending on what expectation is considered appropriate. Both a multicategorical generalization and a multivariate generalization are discussed.     

Coefficients Alpha,Beta, Omega,and the glb: Comments on Sijtsma

There are three fundamental problems in Sijtsma (Psychometrika, 2008): (1) contrary to the name, the glb is not the greatest lower bound of reliability but rather is systematically less than ω _t (McDonald, Test theory: A unified treatment, Erlbaum, Hillsdale, 1999), (2) we agree with Sijtsma that when considering how well a test measures one concept, α is not appropriate, but recommend ω _t rather than the glb, and (3) the end user needs procedures that are readily available in open source software.     

中文版无手机恐惧量表的修订

该研究以国外学者Caglar Yildirim和Ana-Paula Correia于2015年编制的Nomophobia量表为基础,修订出了适用于国内的中文版无手机恐惧量表。研究首先使用探索性结构方程模型对量表结构进行初探; 其次使用项目反应模型进行项目分析并根据其分析结果对项目进行修改和筛选,最终形成正式版量表; 最后对正式版量表其进行信效度检验,并再次对量表题目进行项目分析。最终量表共16道题目,包括害怕无法获得信息、害怕失去便利、害怕失去联系和害怕失去网络连接4个维度; 总量表的Cronbach α系数为0.931,四维度的α系数分别为:0.789,0.816,0.887和0.896。CFA验证性结果显示量表结构较好(χ2/df=3.91,RMSEA=0.067,TLI=0.941,CFI=0.952,SRMR=0.04); 使用手机成瘾倾向量表作为校标的结果显示二者之间相关为0.626,效度良好。最终量表项目区分度在[1.734,4.806]之间,四维度区分度均值为2.1765,2.72,3.2925和3.883,项目难度参数在[-1.830,4.806]之间。综上结果显示该研究编制的无手机恐惧量表各项指标均达到了心理测量学要求,可供后续研究使用     

高校教师教学水平评价概化理论预算限制下最佳样本量估计

使用“高校教师教学水平评价问卷”,要求566名学生对19名教师进行评价,对收集到的数据作不同的概化设计,包括t×i、（s：t）×i、（s：t）×（i：v）和（s：t）×（i：v）×o四种设计。基于概化理论,结合预算限制,统一LaGrange乘法公式,自行推导不同设计的最佳样本量公式,联合估计的方差分量,计算出不同设计的最佳样本量。结果表明：（1）LaGrange乘法统一公式表现出较强的通用性,能够适用于预算限制下各种概化设计;（2）评价场合是影响高校教师教学水平评价一个相当重要的因素;（3）（s：t）×（i：v）×o是高校教师教学水平评价概化理论预算限制下最优概化设计;（4）高校教师教学水平评价概化理论预算限制下,每位教师最佳评价学生人数为20人,每个维度最佳评价题目数为3题。     

Minimizing the number of observations: A generalization of the spearman-brown formula

A new method for determining the minimum number of observations per subject needed to achieve a specific generalizability coefficient is presented. This method, which consists of a branch-and-bound algorithm, allows for the employment of constraints specified by the investigator.     

Effect sizes in ANCOVA and difference-in-differences designs

It is common practice in both randomized and quasi-experiments to adjust for baseline characteristics when estimating the average effect of an intervention. The inclusion of a pre-test, for example, can reduce both the standard error of this estimate and—in non-randomized designs—its bias. At the same time, it is also standard to report the effect of an intervention in standardized effect size units, thereby making it comparable to other interventions and studies. Curiously, the estimation of this effect size, including covariate adjustment, has received little attention. In this article, we provide a framework for defining effect sizes in designs with a pre-test (e.g., difference-in-differences and analysis of covariance) and propose estimators of those effect sizes. The estimators and approximations to their sampling distributions are evaluated using a simulation study and then demonstrated using an example from published data.     

Psychometric properties of the Danish 16PF and EPQ

A Danish translation of the Cattell's 16PF has been used in studies evaluating the effects of prenatal drug exposure. This paper reports a psychometric analysis of the 16PF and Eysenck's EPQ based on a sample of 558 young Danes. Many 16PF scales had unacceptable psychometric properties (as indicated by coefficient alpha and item–total score correlations), but more satisfactory results were obtained with the EPQ N and E scales. A factor analysis of all 16PF and EPQ scales suggested a six factor solution that roughly corresponds to the second-order factor structure obtained by Krug and Johns (1986). It is concluded that the second-order factor structure should be the basis of interpretation of the 16PF in both practical and research contexts.     

On Association Coefficients for 2×2 Tables and Properties That Do Not Depend on the Marginal Distributions

We discuss properties that association coefficients may have in general, e.g., zero value under statistical independence, and we examine coefficients for 2×2 tables with respect to these properties. Furthermore, we study a family of coefficients that are linear transformations of the observed proportion of agreement given the marginal probabilities. This family includes the phi coefficient and Cohen’s kappa. The main result is that the linear transformations that set the value under independence at zero and the maximum value at unity, transform all coefficients in this family into the same underlying coefficient. This coefficient happens to be Loevinger’s H.