追踪研究中测验信度的估计

追踪研究中测验工具的信度是衡量追踪研究质量的重要指标。传统的信度估计方法不适用于估计追踪研究的测验信度。近年来, 研究者提出了四种估计追踪研究的测验信度, 包括估计单个时间点的测验信度系数rw和r(Sw), 以及估计整个追踪研究的测验信度系数RT和RL。本文评述了这四种信度估计方法的数学模型、前提假设及其优缺点。RT和RL既可估计追踪研究中单个时间点的测验信度, 也可估计追踪研究中整个追踪研究的测验信度, 所需要的前提假设较少, 推荐同时使用RT和RL来估计追踪研究的测验信度。     

用Delta法估计多维测验合成信度的置信区间

大量研究表明,一般情况下用合成信度可以较好地估计测验信度。对于合成信度及其置信区间的估计方法,在单维测验的情形已有不少研究。但罕有研究讨论多维测验合成信度的区间估计方法。本文用Delta法推导出计算多维测验合成信度的标准误公式,进而计算置信区间,并用一个例子说明如何编程估计多维测验合成信度及其置信区间。     

信度的再认识与信度概括化研究

本文首先对信度概念进行了明确,指出信度是评价测验结果可靠与否的一个指标,而不是测验工具的不变属性。针对测验结果的信度估计的可变性,介绍了上世纪末Vacha-Haase提出的信度概括化研究方法．即一种用来探索得分信度估计的可变性、并对引起变异的预测源进行探讨的一种元分析方法。最后通过对信度概括化研究手段的分析,指出信度概念的再认识与信度概括化研究将会给心理测验工作者带来新的启示。     

用Delta法估计误差相关测验合成信度的置信区间:以FAD为例

诸多研究显示用合成信度可以较好地估计测验信度。文献上对合成信度置信区间估计的研究都假设题目测量误差不相关,而在实证研究中,也会遇到误差相关的情况,此时α系数往往高估测验信度,使用合成信度估计测验信度比较准确。本文给出用Delta法计算一般的单维测验合成信度的标准误公式,此公式无论测验误差是否相关都适用,据此可以计算合成信度的置信区间。通过对600名青少年调查发现,中文版FAD分测验"总的功能"的反向题测验误差存在相关,演示了如何估计此分测验的合成信度及其置信区间。     

使用Bootstrap方法计算认知诊断评估中的信度

测验信度是衡量测验质量的一个重要指标,认知诊断评估中同样需要重视信度问题。现有认知诊断中计算信度的方法均有一个前提假设:被试在前后两次测验的后验概率分布和边际概率完全相同。该假设过强,未考虑两次测验间存在的随机误差。基于Bootstrap抽样,提出了两类属性信度和模式信度的指标,分别是积差相关法和修正的一致性法。通过模拟研究比较了新方法和现有方法在不同属性个数、属性间相关性和题目数量下的表现,并基于英语能力认证考试ECPE和分数减法的实证数据验证了新方法的可行性。最后,对信度估计的影响因素进行了讨论。     

单维测验合成信度三种区间估计的比较

已有许多研究建议使用合成信度来估计测验信度, 并报告其置信区间。有三种方法或途径可以计算单维测验合成信度的置信区间, 包括Bootstrap法、Delta法和直接用统计软件(如LISREL)输出的标准误进行计算。本文通过模拟研究进行比较, 发现Delta法与Bootstrap法得到的置信区间相当接近, 但用LISREL输出的标准误计算的与Bootstrap法得到的结果相差很大。推荐用Delta法估计合成信度的置信区间(使用Mplus容易实现), 但不能直接用LISREL输出的标准误来计算。举例说明了如何计算单维测验的合成信度以及用Delta法计算其置信区间。     

两种估计多维测验合成信度置信区间方法比较

有两种方法可以估计多维测验合成信度的置信区间：Bootstrap法和Delta法.本文用模拟研究比较这两种方法,结果发现,Delta法与Bootstrap法得到结果的差异很小.因为Bootstrap法得到的是实证结果,通常被认为是真值的反映,而Delta法比Bootstrap法简单得多,所以可以用Delta法估计合成信度的置信区间.举例演示如何计算多维测验的合成信度以及用Delta法计算其置信区间.     

测验信度估计：从α系数到内部一致性信度

沿用经典的测验信度定义, 简介了信度与a 系数的关系以及a系数的局限。为了推荐替代a系数的信度估计方法, 深入讨论了与a 系数关系密切的同质性信度和内部一致性信度。在很一般的条件下, 证明了a 系数和同质性信度都不超过内部一致性信度, 后者不超过测验信度, 说明内部一致性信度比较接近测验信度。总结出一个测验信度分析流程, 说明什么情况下a 系数还有参考价值; 什么情况下a 系数不再适用, 应当使用内部一致性信度(文献上也常称为合成信度)。提供了计算同质性信度和内部一致性信度的计算程序, 一般的应用工作者可以直接套用。     

标准参照测验及其等级线信度的概化理论分析

在测量工作中,误用经典测验理论方法估计标准参照性测验的整体信度和等级线决策信度的情况非常突出。如,无论测量设计是交叉的还是嵌套的,也无论测验结果是做常模参照性解释,还是做标准参照性解释,测验工作者往往只报告克龙巴赫α系数或经典测验理论中的其它少数几个信度指标,而误把整体信度作为等级线信度的现象则更加普遍,这是十分不妥的。本文借用概化理论中的可靠性指数Φ和Φ(λ)公式,分别针对交叉设计和嵌套设计,就标准参照性测验的整体信度和等级分数线决策信度的估计问题进行了探讨。用数据演示的方法比较了交叉设计与嵌套设计在估计标准参照性测验整体信度方面的差异,展示了等级决策分数线决策信度的估计方法。     

两水平研究中单维测验信度的估计

在心理、教育和管理等研究领域中,经常会碰到两水平（两层）的数据结构,如学生嵌套在班级中,员工嵌套在企业中。在两水平研究中,被试通常不是独立的,如果直接用单水平信度公式进行估计,会高估测验信度。文献上已有研究讨论如何更准确地估计两水平研究中单维测验的信度。本研究指出了现有的估计公式的不足之处,用两水平验证性因子分析推导出一个新的信度公式,举例演示如何计算,并给出简单的计算程序。     

Sample size planning for composite reliability coefficients: accuracy in parameter estimation via narrow confidence intervals

Composite measures play an important role in psychology and related disciplines. Composite measures almost always have error. Correspondingly, it is important to understand the reliability of the scores from any particular composite measure. However, the point estimates of the reliability of composite measures are fallible and thus all such point estimates should be accompanied by a confidence interval. When confidence intervals are wide, there is much uncertainty in the population value of the reliability coefficient. Given the importance of reporting confidence intervals for estimates of reliability, coupled with the undesirability of wide confidence intervals, we develop methods that allow researchers to plan sample size in order to obtain narrow confidence intervals for population reliability coefficients. We first discuss composite reliability coefficients and then provide a discussion on confidence interval formation for the corresponding population value. Using the accuracy in parameter estimation approach, we develop two methods to obtain accurate estimates of reliability by planning sample size. The first method provides a way to plan sample size so that the expected confidence interval width for the population reliability coefficient is sufficiently narrow. The second method ensures that the confidence interval width will be sufficiently narrow with some desired degree of assurance (e.g., 99% assurance that the 95% confidence interval for the population reliability coefficient will be less than W units wide). The effectiveness of our methods was verified with Monte Carlo simulation studies. We demonstrate how to easily implement the methods with easy-to-use and freely available software.     

测验同质性系数及其区间估计

在决定将多维测验分数合并成测验总分时, 应当考虑测验同质性。如果同质性太低, 合成总分没有什么意义。同质性高低可以用同质性系数来衡量。用来计算同质性系数的模型是近年来受到关注的双因子模型(既有全局因子又有局部因子), 测验的同质性系数定义为测验分数方差中全局因子分数方差所占的比例。本文用Delta法推导出计算同质性系数的标准误公式, 进而计算其置信区间。提供了简单的计算同质性系数及其置信区间的程序。用一个例子说明如何估计同质性系数及其置信区间, 通过模拟比较了用Delta法和用Bootstrap法计算的置信区间, 发现两者差异很小。     

认知诊断属性分类一致性信度区间估计三种方法

摘要：引入了三种可以估计认知诊断属性分类一致性信度置信区间的方法：Bootstrap法、平行测验法和平行测验配对法。用模拟研究验证和比较了这三种方法的表现,结果发现,平行测验法和Bootstrap法在被试量比较少、题目数量比较少的情况下,估计的标准误和置信区间较接近,但是随着被试量的增加,Bootstrap法的估计精度提高较快,在被试量大和题目数量较多时基本接近平行测验配对法的结果。Bootstrap法的所需时间最少,平行测验配对法计算过程复杂且用时较长,推荐用Bootstrap法估计认知诊断属性分类一致性信度的置信区间。     

α系数的区间估计方法比较

多数情况下,α系数可以用来评价测验信度。诸多研究建议,在报告测验信度的时候应当包括其置信区间。通过蒙特卡洛模拟研究,比较了7种α系数区间估计方法,包括Fisher法、Bonett-02法、Bonett-10法、精确Koning-Franses法、渐近ID法、渐近Koning-Franses法和ADF法。结果发现Bonett-10法和精确Koning-Franses法较好,它们的结果相差很小。这两种方法都比较简单,只需要样本的α值、测验题数、被试人数及F临界值,通过简单的运算便可得到α系数的置信区间。     

Fisher transformation based confidence intervals of correlations in fixed- and random-effects meta-analysis

Meta-analyses of correlation coefficients are an important technique to integrate results from many cross-sectional and longitudinal research designs. Uncertainty in pooled estimates is typically assessed with the help of confidence intervals, which can double as hypothesis tests for two-sided hypotheses about the underlying correlation. A standard approach to construct confidence intervals for the main effect is the Hedges-Olkin-Vevea Fisher-z (HOVz) approach, which is based on the Fisher-z transformation. Results from previous studies (Field, 2005, Psychol. Meth., 10, 444; Hafdahl and Williams, 2009, Psychol. Meth., 14, 24), however, indicate that in random-effects models the performance of the HOVz confidence interval can be unsatisfactory. To this end, we propose improvements of the HOVz approach, which are based on enhanced variance estimators for the main effect estimate. In order to study the coverage of the new confidence intervals in both fixed- and random-effects meta-analysis models, we perform an extensive simulation study, comparing them to established approaches. Data were generated via a truncated normal and beta distribution model. The results show that our newly proposed confidence intervals based on a Knapp-Hartung-type variance estimator or robust heteroscedasticity consistent sandwich estimators in combination with the integral z-to-r transformation (Hafdahl, 2009, Br. J. Math. Stat. Psychol., 62, 233) provide more accurate coverage than existing approaches in most scenarios, especially in the more appropriate beta distribution simulation model.     

Synthesizing single-case studies: A Monte Carlo examination of a three-level meta-analytic model

Numerous ways to meta-analyze single-case data have been proposed in the literature; however, consensus has not been reached on the most appropriate method. One method that has been proposed involves multilevel modeling. For this study, we used Monte Carlo methods to examine the appropriateness of Van den Noortgate and Onghena's (2008) raw-data multilevel modeling approach for the meta-analysis of single-case data. Specifically, we examined the fixed effects (e.g., the overall average treatment effect) and the variance components (e.g., the between-person within-study variance in the treatment effect) in a three-level multilevel model (repeated observations nested within individuals, nested within studies). More specifically, bias of the point estimates, confidence interval coverage rates, and interval widths were examined as a function of the number of primary studies per meta-analysis, the modal number of participants per primary study, the modal series length per primary study, the level of autocorrelation, and the variances of the error terms. The degree to which the findings of this study are supportive of using Van den Noortgate and Onghena's (2008) raw-data multilevel modeling approach to meta-analyzing single-case data depends on the particular parameter of interest. Estimates of the average treatment effect tended to be unbiased and produced confidence intervals that tended to overcover, but did come close to the nominal level as Level-3 sample size increased. Conversely, estimates of the variance in the treatment effect tended to be biased, and the confidence intervals for those estimates were inaccurate.     

Reliability Generalization Study of the Yale–Brown Obsessive–Compulsive Scale for Children and Adolescents

The Yale–Brown Obsessive–Compulsive Scale for children and adolescents (CY–BOCS) is a frequently applied test to assess obsessive–compulsive symptoms. We conducted a reliability generalization meta-analysis on the CY–BOCS to estimate the average reliability, search for reliability moderators, and propose a predictive model that researchers and clinicians can use to estimate the expected reliability of the CY–BOCS scores. A total of 47 studies reporting a reliability coefficient with the data at hand were included in the meta-analysis. The results showed good reliability and a large variability associated to the standard deviation of total scores and sample size.