Estimating Between-Person and Within-Person Subscore Reliability with Profile Analysis

Subscores are of increasing interest in educational and psychological testing due to their diagnostic function for evaluating examinees' strengths and weaknesses within particular domains of knowledge. Previous studies about the utility of subscores have mostly focused on the overall reliability of individual subscores and ignored the fact that subscores should be distinct and have added value over the total score. This study introduces a profile reliability approach that partitions the overall subscore reliability into within-person and between-person subscore reliability. The estimation of between-person reliability and within-person reliability coefficients is demonstrated using subscores from number-correct scoring, unidimensional and multidimensional item response theory scoring, and augmented scoring approaches via a simulation study and a real data study. The effects of various testing conditions, such as subtest length, correlations among subscores, and the number of subtests, are examined. Results indicate that there is a substantial trade-off between within-person and between-person reliability of subscores. Profile reliability coefficients can be useful in determining the extent to which subscores provide distinct and reliable information under various testing conditions.     

The design of the MBT‐G adherence and quality scale

Few group psychotherapy studies focus on therapists' interventions, and instruments that can measure group psychotherapy treatment fidelity are scarce. The aim of the present study was to evaluate the reliability of the Mentalization‐based Group Therapy Adherence and Quality Scale (MBT‐G‐AQS), which is a 19‐item scale developed to measure adherence and quality in mentalization‐based group therapy (MBT‐G). Eight MBT groups and eight psychodynamic groups (a total of 16 videotaped therapy sessions) were rated independently by five raters. All groups were long‐term, outpatient psychotherapy groups with 1.5 hours weekly sessions. Data were analysed by a Generalizability Study (G‐study and D‐study). The generalizability models included analyses of reliability for different numbers of raters. The global (overall) ratings for adherence and quality showed high to excellent reliability for all numbers of raters (the reliability by use of five raters was 0.97 for adherence and 0.96 for quality). The mean reliability for all 19 items for a single rater was 0.57 (item range 0.26–0.86) for adherence, and 0.62 (item range 0.26–0.83) for quality. The reliability for two raters obtained mean absolute G‐coefficients on 0.71 (item range 0.41–0.92 for the different items) for adherence and 0.76 (item range 0.42–0.91) for quality. With all five raters the mean absolute G‐coefficient for adherence was 0.86 (item range 0.63–0.97) and 0.88 for quality (item range 0.64–0.96). The study demonstrates high reliability of ratings of MBT‐G‐AQS. In models differentiating between different numbers of raters, reliability was particularly high when including several raters, but was also acceptable for two raters. For practical purposes, the MBT‐G‐AQS can be used for training, supervision and psychotherapy research.     

社会公正对权威合法性的影响:社会阶层的调节作用

社会层面中的权威合法性研究是国内外学者和管理实践者近来关注的重要问题。而社会公正的两个维度——分配公正和程序公正对权威合法性的交互作用的结果并不一致。本研究基于解释水平理论,提出社会阶层能调节分配公正和程序公正对权威合法性感知的交互作用,并通过实验室研究和情境启动两种方法进行验证。结果发现对低阶层者来说,无论是否程序公正,分配公正能显著提高个体的权威合法性感知;在分配公正和多得不公条件下,程序公正显著降低权威合法性感知。对高阶层者来说,分配公正能显著提高程序公正时的权威合法性感知;程序公正能显著提高分配公正时的权威合法性感知。研究结果启示社会管理者在推行依法治国时应针对不同阶层民众的思维方式采取管理策略。     

Reminiscence functions over time: consistency of self functions and variation of prosocial functions

The current study examines the temporal stability of the tripartite model of reminiscence functions in which eight separate reminiscence functions map onto three second-order factors which contribute significantly to measurement of an overarching reminiscence latent construct. We collected online responses from 411 adults 50+ years of age. Confirmatory factor analytic models were computed at three points of data collection over 16 months. Invariance analyses were next undertaken to simultaneously compare the measurement properties to assess within-person stability of reminiscence functions over time. The tripartite structure of reminiscence functions was replicated at each point of data collection. As hypothesised, self-positive and self-negative functions are consistent across points of data collection, whereas prosocial functions vary over time. The temporal stability of the self functions may be attributed to enduring characteristics of the individual such as personality traits and life attitudes, as well as their solitary nature. Previous research indicates that consistency of self-positive reminiscence functions has ensuing benefits for physical health and psychological well-being; the opposite is true for self-negative functions. The temporal variation of prosocial functions may be due to the varying availability of others to share memories and their responsiveness to the emotional context.     

大五人格测验在中国应用的信度概化分析

对过去20年(1994~2013年)间国内有关大五人格测验的研究文献进行信度概化分析。结果表明:(1)检索到的文献中约68.15%存在"信度引入"现象;(2)未加权估计中,A和O的均值最低,N和C的均值最高,国内所得结果均略低于国外(O除外),而后者的变异性略大(E除外);采用α系数效果量方法,在随机效应模型中,N的估计值最高,O和A的估计值最低;(3)回归分析显示,分数均值、量表来源和南北地域差异是N维度信度的预测变量;量表来源、文章专业类型、测验版本和测验记分对E维度信度具有预测作用;样本量、文章专业类型和量表来源是O维度信度的预测变量;量表来源、文章专业类型、项目数和样本类型对A维度信度具有预测作用;量表来源、项目数、文章专业类型和测验记分是C维度信度的预测变量。     

Why do residents accept a demanding rule?: Fairness and social benefit as determinants of approval of a recycling system1,2

Abstract: This study examined why people accepted a demanding rule in a recycling system that was newly introduced in Nagoya City. We focused on two social psychological topics: social dilemmas and fairness. While the new system succeeded in reducing waste, it imposed a burden on citizens without providing incentives and sanctions. In a research survey, 1442 responses from a sample of 3000 (48% response rate) were obtained using a stratified sampling method. The results showed that the new recycling system was approved despite the demands placed on citizens, with a preference for more strict rules such as penalties and surveillance for noncompliance. The main determinants of approval of the new recycling system were social benefit and procedural fairness, whereas the main determinant of preference for strict rules was outcome fairness. We argue that (a) social benefit should be emphasized to facilitate cooperative behavior in a social dilemma situation and (b) the government should ensure sufficient discussion with citizens and acknowledgment of their opinions.     

表现性评价中评分者信度估计方法的比较研究--从相关法、百分比法到概化理论

随着素质教育的推进．表现性评价受到越来越多的重视。影响表现性评价结果信度的一个重要因素是评分者之间的不一致。文章使用模拟数据,在对比评分者一致性的相关法、一致性百分比法和概化系数等各种估计方法的基础上,提出概化理论在表现性评价中评分者信度问题上的应用是理论和实践发展的有益方向。     

一个基于综合印象评分法的作文分事后调整模型

对大学英语四、六级考试作文评分进行了详细的描述,重点介绍了作文分的事后调整的原理和方法,并且给出了一个基于线性等值原理的数学推导。认为其数学模型主要是一个运用了“极大似然估计法”和“正态分”概念的统计模型。“评分标准的制定”、“用参照样卷来校准阅卷员对作文评分标准的掌握尺度”以及“阅卷员的培训和考核”构成了大学英语四、六级考试作文信度的基石;而作文分调整中基于“随机分发作文卷”、“客观题分数与作文分相关”以及“评分前后一致性”的统计方法则对出现系统性误差的阅卷员的评分结果进行了事后的校正。还提出如果拥有往次考试总体作文分均值数据,使用“加权移动平均法”可以实现对作文分的跨考次平衡。     

质性研究中编码者信度的多种方法考察

质性研究中检验编码者信度的方法有归类一致性指数、编码信度系数、相关系数、中位数检验、概化系数等。基于教师胜任力访谈数据集,对编码者信度考察结果表明,归类一致性指数和编码信度系数受相同编码数影响而不稳定,相关系数受数据类型制约,中位数检验受研究设计影响,概化系数则受编码者和编码项目的数量影响。研究中须合理选用。