计划缺失设计——通过有意缺失让研究更高效

缺失值是社会科学研究中非常普遍的现象。全息极大似然估计和多重插补是目前处理缺失值最有效的方法。计划缺失设计利用特殊的实验设计有意产生缺失值, 再用现代的缺失值处理方法来完成统计分析, 获得无偏的统计结果。计划缺失设计可用于横断面调查减少(或增加)问卷长度和纵向调查减少测量次数, 也可用于提高测量有效性。常用的计划缺失设计有三式设计和两种方法测量。     

The Relative Importance and Interaction of Contextual and Methodological Predictors of Mean rWG for Work Climate

A variety of collective phenomena are understood to exist to the extent that workers agree on their perceptions of the phenomena, such as perceptions of their organization’s climate or perceptions of their team’s mental model. Researchers conducting group-level studies of such phenomena measure individuals’ perceptions via surveys and then aggregate data to the group level if the mean within-group agreement for a sample of groups is sufficiently high. Despite this widespread practice, we know little about the factors potentially affecting mean within-group agreement. Here, focusing on work climate, we report an investigation of a number of expected contextual (social interaction) and methodological predictors of mean r_WG, a common statistic for judging within-group agreement in applied psychology and management research. We used the novel approach of meta-CART, which allowed us to assess the relative importance and possible interactions of the predictor variables. Notably, mean r_WG values are driven by both contextual (average number of individuals per group and cultural individualism-collectivism) and methodological factors (the number of items in a scale and scale reliability). Our findings are largely consistent with expectations concerning how social interaction affects within-group agreement and psychometric arguments regarding why adding more items to a scale will not necessarily increase the magnitude of an index based on a Spearman-Brown “stepped-up correction.” We discuss the key insights from our results, which are relevant to the study of multilevel phenomena relying on the aggregation of individual-level data and informative for how meta-analytic researchers can simultaneously examine multiple moderator variables.

     

认知诊断缺失数据处理方法的比较：零替换、多重插补与极大似然估计法

数据缺失在测验中经常发生, 认知诊断评估也不例外, 数据缺失会导致诊断结果的偏差。首先, 通过模拟研究在多种实验条件下比较了常用的缺失数据处理方法。结果表明：(1)缺失数据导致估计精确性下降, 随着人数与题目数量减少、缺失率增大、题目质量降低, 所有方法的PCCR均下降, Bias绝对值和RMSE均上升。(2)估计题目参数时, EM法表现最好, 其次是MI, FIML和ZR法表现不稳定。(3)估计被试知识状态时, EM和FIML表现最好, MI和ZR表现不稳定。其次, 在PISA2015实证数据中进一步探索了不同方法的表现。综合模拟和实证研究结果, 推荐选用EM或FIML法进行缺失数据处理。     

2PLM下缺失数据处理方法及其比较

项目反应理论(IRT)是用于客观测量的现代教育与心理测量理论之一,广泛用于缺失数据十分常见的大尺度测验分析。IRT中两参数逻辑斯蒂克模型(2PLM)下仅有完全随机缺失机制下缺失反应和缺失能力处理的EM算法。本研究推导2PLM下缺失反应忽略的EM 算法,并提出随机缺失机制下缺失反应和缺失能力处理的EM算法和考虑能力估计和作答反应不确定性的多重借补法。研究显示：在各种缺失机制、缺失比例和测验设计下,缺失反应忽略的EM算法和多重借补法表现理想。     

A rank test for two group concordance

A test statistic is introduced which allows one to test the hypothesis of agreement of several judges on the ranking of items within each of two groups and between the two groups. The groups of judges may be unequal in size. A normal approximation for the test statistic is developed. The relationship to existing techniques given by Kendall, Friedman, Page, Spearman, and Lyerly is discussed. A generalization of the coefficient of concordance is presented and the extension of the method to multi-group problems is suggested. Research supported in part by ONR Contract N00014-72-A-0296.     

概化理论缺失数据方差分量估计

各种心理调查、心理实验中, 数据的缺失随处可见。由于数据缺失, 给概化理论分析非平衡数据的方差分量带来一系列问题。基于概化理论框架下, 运用Matlab 7.0软件, 自编程序模拟产生随机双面交叉设计p×i×r缺失数据, 比较和探讨公式法、REML法、拆分法和MCMC法在估计各个方差分量上的性能优劣。结果表明：(1) MCMC方法估计随机双面交叉设计p×i×r缺失数据方差分量, 较其它3种方法表现出更强的优势; (2) 题目和评分者是缺失数据方差分量估计重要的影响因素。     

Modeling Concordance Correlation Coefficient for Longitudinal Study Data

Measures of agreement are used in a wide range of behavioral, biomedical, psychosocial, and health-care related research to assess reliability of diagnostic test, psychometric properties of instrument, fidelity of psychosocial intervention, and accuracy of proxy outcome. The concordance correlation coefficient (CCC) is a popular measure of agreement for continuous outcomes. In modern-day applications, data are often clustered, making inference difficult to perform using existing methods. In addition, as longitudinal study designs become increasingly popular, missing data have become a serious issue, and the lack of methods to systematically address this problem has hampered the progress of research in the aforementioned fields. In this paper, we develop a novel approach to tackle the complexities involved in addressing missing data and other related issues for performing CCC analysis within a longitudinal data setting. The approach is illustrated with both real and simulated data.     

Ethnic Identity, Self-Esteem, and Values in Dominicans, Puerto Ricans, and African Americans

This study investigates the relationship between ethnic identity, self-esteem, value orientations, and perceived value congruence in 207 minority students. It also investigates within-group concordance and cross-cultural differences in value orientations. Dilemmas were used to measure value orientations and perceived congruence between personal and group values. A version of the Multigroup Ethnic Identity Measure (Phinney, 1992) and Rosenberg's Self-Esteem Scale (1965) were used to measure ethnic identity and self-esteem, respectively. Ethnic identity was positively related to self-esteem. The perception of value congruence was not related to ethnic identity or self-esteem. There was within-group concordance in the ranking of value solutions. In addition, the groups differed in the strength of ethnic identity, perceived value congruence, and the ranking of the value solutions.     

Aggregation and beyond: Some basic issues on the prediction of behavior

Failure to appreciate the role that aggregation plays in increasing reliability and validity and in establishing the range of generalization of findings has resulted in misunderstandings about the stability of behavior across time and situations, and in the conduct of experiments that produce results that tend to be neither generalizable nor replicable. Appropriate aggregation can reduce error variance associated with the unrepresentativeness of individual stimuli, situations, occasions, judges, items of behavior, and subjects. Inappropriate aggregation can result not only in a loss of information but also in a reduction in reliability as well as validity. Different approaches to prediction with single items of behavior are discussed, and it is concluded that single items tend to be too unreliable and too narrow in scope to measure broad dispositions such as traits. A major emphasis is that behavior is often so highly situationally specific that unless this is taken into account by procedures such as aggregation over situations and/or occasions, or by the investigation of events that are so highly ego-involving that experimental effects dominate situation-ally unique effects, results will tend to be unreplicable or ungeneralizable, no matter what their level of statistical significance.     

Comparison of Modern Methods for Analyzing Repeated Measures Data With Missing Values

Missing data are a pervasive problem in many psychological applications in the real world. In this article we study the impact of dropout on the operational characteristics of several approaches that can be easily implemented with commercially available software. These approaches include the covariance pattern model based on an unstructured covariance matrix (CPM-U) and the true covariance matrix (CPM-T), multiple imputation-based generalized estimating equations (MI-GEE), and weighted generalized estimating equations (WGEE). Under the missing at random mechanism, the MI-GEE approach was always robust. The CPM-T and CPM-U methods were also able to control the error rates provided that certain minimum sample size requirements were met, whereas the WGEE was more prone to inflated error rates. In contrast, under the missing not at random mechanism, all evaluated approaches were generally invalid. Our results also indicate that the CPM methods were more powerful than the MI-GEE and WGEE methods and their superiority was often substantial. Furthermore, we note that little or no power was sacrificed by using CPM-U method in place of CPM-T, although both methods have less power in situations where some participants have incomplete data. Some aspects of the CPM-U and MI-GEE methods are illustrated using real data from 2 previously published data sets. The first data set comes from a randomized study of AIDS patients with advanced immune suppression, the second from a cohort of patients with schizotypal personality disorder enrolled in a prevention program for psychosis.     

Imputation methods for missing data in educational diagnostic evaluation

In the diagnostic evaluation of educational systems, self-reports are commonly used to collect data, both cognitive and orectic. For various reasons, in these self-reports, some of the students' data are frequently missing. The main goal of this research is to compare the performance of different imputation methods for missing data in the context of the evaluation of educational systems. On an empirical database of 5,000 subjects, 72 conditions were simulated: three levels of missing data, three types of loss mechanisms, and eight methods of imputation. The levels of missing data were 5%, 10%, and 20%. The loss mechanisms were set at: Missing completely at random, moderately conditioned, and strongly conditioned. The eight imputation methods used were: listwise deletion, replacement by the mean of the scale, by the item mean, the subject mean, the corrected subject mean, multiple regression, and Expectation-Maximization (EM) algorithm, with and without auxiliary variables. The results indicate that the recovery of the data is more accurate when using an appropriate combination of different methods of recovering lost data. When a case is incomplete, the mean of the subject works very well, whereas for completely lost data, multiple imputation with the EM algorithm is recommended. The use of this combination is especially recommended when data loss is greater and its loss mechanism is more conditioned. Lastly, the results are discussed, and some future lines of research are analyzed.     

Grouping in short-term memory: do oscillators code the positions of items?

According to several current models of short-term memory, items are retained in order by associating them with positional codes. The models differ as to whether temporal oscillators provide those codes. The authors examined errors in recall of sequences comprising 2 groups of 4 consonants. A critical manipulation was the precise timing of items within the groups, whereby temporal position (time from group onset) and ordinal position (number of items from group onset) were partially unconfounded. Errors that involve items migrating across groups should preserve within-group temporal position according to oscillator models, but should preserve within-group ordinal position according to nonoscillator models. Results from the intergroup errors strongly favored preservation of ordinal rather than temporal position.     

Neuropsychological changes in children with cancer: The treatment of missing data in longitudinal studies

Researchers conducting longitudinal studies with children or adults are inevitably confronted with problems of attrition and missing data. Missing data in longitudinal studies is frequently handled by excluding from analyses those cases for whom data are incomplete. This approach to missing data is not optimal. On the one hand, if data are missing at random, then dropping incomplete cases ignores information collected on those cases that could be used to improve estimates of population parameters (e.g., means, variances, covariances, and growth rates) and improve the power of significance tests of statistical hypotheses. On the other hand, if data are not missing at random, then dropping incomplete cases leads to biased parameter estimates and hypothesis tests that may be internally and externally invalid. This study uses three years of follow-up data from a longitudinal investigation of neuropsychological outcomes of cancer in children to demonstrate the problems presented by missing data in repeated measures designs and some solutions. In evaluating potential biasing effects of attrition, the study extends previous research on neuropsychological outcomes in pediatric cancer by inclusion of patients whose disease had relapsed, and by comparison of surviving and nonsurviving patients. Although the data presented have specific relevance to the study of neuropsychological outcome in pediatric cancer, the problems of missing data and the solutions presented are relevant to a wide variety of diseases and conditions of interest to researchers in child and adult neuropsychology.     

An ordinal coefficient of relational agreement for multiple judges

In a recent article, Fagot proposed a generalized family of coefficients of relational agreement for multiple judges, focusing on the concept of empirically meaningful relationships. In this paper an ordinal coefficient of relational agreement, based on ranking data, is presented as a special case of the generalized family. It is shown that the proposed ordinal coefficient encompasses other ordinal coefficients, such as the Kendall coefficient of concordance, the average Spearman rank-order coefficient, and intraclass correlation based on ranks. It is also shown that the Kendall coefficient of concordance, corrected for chance agreement, is equivalent to the ordinal coefficient proposed in this paper.     

认知诊断测验中基于信息矩阵的多群组DIF检验

基于改进的Wald统计量,将适用于两群组的DIF检测方法拓展至多群组的项目功能差异（DIF）检验;改进的Wald统计量将分别通过计算观察信息矩阵（Obs）和经验交叉相乘信息矩阵（XPD）而得到。模拟研究探讨了此二者与传统计算方法在多个群组下的DIF检验情况,结果表明：（1）Obs和XPD的一类错误率明显低于传统方法,DINA模型估计下Obs和XPD的一类错误率接近理论水平;（2）样本量和DIF量较大时,Obs和XPD具有与传统Wald统计量大体相同的统计检验力。     

Agreement between Two Independent Groups of Raters

We propose a coefficient of agreement to assess the degree of concordance between two independent groups of raters classifying items on a nominal scale. This coefficient, defined on a population-based model, extends the classical Cohen’s kappa coefficient for quantifying agreement between two raters. Weighted and intraclass versions of the coefficient are also given and their sampling variance is determined by the Jackknife method. The method is illustrated on medical education data which motivated the research.     

Power and Personality Perception in Real‐Life Hierarchical Relationships

The present study examined how Big Five personality ratings of the same target individuals differ as a function of the power relation between the target and the judge. Our targets were 37 employees with leadership duties from two large organizations. The targets' subordinates (N = 352), peers (N = 186), and superiors (N = 62) constituted our groups of judges. The targets and judges also provided self‐reports of personality. Subordinate judges showed higher consensus but not higher self‐other agreement than peer or superior judges. Furthermore, the targets were judged as more extraverted, more emotionally stable, less agreeable, and less open to experience by their subordinates than by their superiors. The results suggest that (i) observer consensus, but not self‐other agreement or assumed similarity varies as a function of real‐life power; (ii) the effects of power on mean trait scores are mostly congruent with the previously observed effects of power on behaviour and on stereotypes. Copyright © 2012 John Wiley & Sons, Ltd.     

2PL模型的两种马尔可夫蒙特卡洛缺失数据处理方法比较

马尔科夫蒙特卡洛（MCMC）是项目反应理论中处理缺失数据的一种典型方法。文章通过模拟研究比较了在不同被试人数,项目数,缺失比例下两种MCMC方法（M-H within Gibbs和DA-T Gibbs）参数估计的精确性,并结合了实证研究。研究结果表明,两种方法是有差异的,项目参数估计均受被试人数影响很大,受缺失比例影响相对更小。在样本较大缺失比例较小时,M-H within Gibbs参数估计的均方误差（RMSE）相对略小,随着样本数的减少或缺失比例的增加,DA-T Gibbs方法逐渐优于M-H within Gibbs方法     

The Missing Data Assumptions of the NEAT Design and their Implications for Test Equating

The Non-Equivalent groups with Anchor Test (NEAT) design involves missing data that are missing by design. Three nonlinear observed score equating methods used with a NEAT design are the frequency estimation equipercentile equating (FEEE), the chain equipercentile equating (CEE), and the item-response-theory observed-score-equating (IRT OSE). These three methods each make different assumptions about the missing data in the NEAT design. The FEEE method assumes that the conditional distribution of the test score given the anchor test score is the same in the two examinee groups. The CEE method assumes that the equipercentile functions equating the test score to the anchor test score are the same in the two examinee groups. The IRT OSE method assumes that the IRT model employed fits the data adequately, and the items in the tests and the anchor test do not exhibit differential item functioning across the two examinee groups. This paper first describes the missing data assumptions of the three equating methods. Then it describes how the missing data in the NEAT design can be filled in a manner that is coherent with the assumptions made by each of these equating methods. Implications on equating are also discussed.