Rater Model Using Signal Detection Theory for Latent Differential Rater Functioning

Differential rater functioning (DRF) occurs when raters show evidence of exercising differential severity or leniency when scoring examinees within different subgroups. Previous studies of DRF have examined rater bias using manifest variables (e.g., use of covariates) to determine the subgroups. These manifest variables include gender and the ethnicity of the examinee. For example, a rater may score males more severely. Ideally, each rater’s severity should be invariant across subgroups. This study examines DRF in the context of latent subgroups that classify possible sources of DRF based on raters’ scoring behavior rather than manifest factors. An extension of the latent class signal detection theory (LC-SDT) model for identifying DRF is proposed and examined using real-world data and simulations. Results from real-world data show that the signal detection approach leads to an effective method to identify latent DRF. Simulations with varying sample sizes and conditions of rater precision were shown to recover parameters at an adequate level, supporting its use to identify latent DRF in large-scale data. These findings suggest that the DRF extension of the LC-SDT can be a useful model to examine characteristics of raters and add information that can aid rater training.     

Simple Structure Detection Through Bayesian Exploratory Multidimensional IRT Models

In modern validity theory, a major concern is the construct validity of a test, which is commonly assessed through confirmatory or exploratory factor analysis. In the framework of Bayesian exploratory Multidimensional Item Response Theory (MIRT) models, we discuss two methods aimed at investigating the underlying structure of a test, in order to verify if the latent model adheres to a chosen simple factorial structure. This purpose is achieved without imposing hard constraints on the discrimination parameter matrix to address the rotational indeterminacy. The first approach prescribes a 2-step procedure. The parameter estimates are obtained through an unconstrained MCMC sampler. The simple structure is, then, inspected with a post-processing step based on the Consensus Simple Target Rotation technique. In the second approach, both rotational invariance and simple structure retrieval are addressed within the MCMC sampling scheme, by introducing a sparsity-inducing prior on the discrimination parameters. Through simulation as well as real-world studies, we demonstrate that the proposed methods are able to correctly infer the underlying sparse structure and to retrieve interpretable solutions.     

To hangover or not: trajectories of job satisfaction in adolescent workforce newcomers

The development of job satisfaction during the first months on the job often indicates a honeymoon hangover, with high levels of job satisfaction gradually declining. This effect is often explained by disappointed expectations that are informed by previous job experiences. However, research has not established whether a hangover pattern could also be observed in individuals without previous work experience. We explored the development of job satisfaction with 4 assessment points across the first 4 months after starting vocational training among 357 Swiss adolescents. On average, a hangover pattern in job satisfaction was confirmed. Using person-centred growth mixture modelling, we identified two groups with distinct trajectories. Although a majority showed a hangover pattern, a third of participants showed stable, high job satisfaction. We presumed that adolescents with more contextual and personal resources (i.e., perceived social support, occupational self-efficacy, core self-evaluations, and perceived person–job fit) would be more likely to avoid a hangover pattern. Results confirmed that the two groups differed significantly in all these resources, with the high stable satisfaction group showing higher resources. The results illustrate the importance of a diverse set of resources to facilitate a positive trajectory of job satisfaction at the beginning of work life.     

Assessment of Differential Rater Functioning in Latent Classes with New Mixture Facets Models

Multifaceted data are very common in the human sciences. For example, test takers' responses to essay items are marked by raters. If multifaceted data are analyzed with standard facets models, it is assumed there is no interaction between facets. In reality, an interaction between facets can occur, referred to as differential facet functioning. A special case of differential facet functioning is the interaction between ratees and raters, referred to as differential rater functioning (DRF). In existing DRF studies, the group membership of ratees is known, such as gender or ethnicity. However, DRF may occur when the group membership is unknown (latent) and thus has to be estimated from data. To solve this problem, in this study, we developed a new mixture facets model to assess DRF when the group membership is latent and we provided two empirical examples to demonstrate its applications. A series of simulations were also conducted to evaluate the performance of the new model in the DRF assessment in the Bayesian framework. Results supported the use of the mixture facets model because all parameters were recovered fairly well, and the more data there were, the better the parameter recovery.     

Estimating Between-Person and Within-Person Subscore Reliability with Profile Analysis

Subscores are of increasing interest in educational and psychological testing due to their diagnostic function for evaluating examinees' strengths and weaknesses within particular domains of knowledge. Previous studies about the utility of subscores have mostly focused on the overall reliability of individual subscores and ignored the fact that subscores should be distinct and have added value over the total score. This study introduces a profile reliability approach that partitions the overall subscore reliability into within-person and between-person subscore reliability. The estimation of between-person reliability and within-person reliability coefficients is demonstrated using subscores from number-correct scoring, unidimensional and multidimensional item response theory scoring, and augmented scoring approaches via a simulation study and a real data study. The effects of various testing conditions, such as subtest length, correlations among subscores, and the number of subtests, are examined. Results indicate that there is a substantial trade-off between within-person and between-person reliability of subscores. Profile reliability coefficients can be useful in determining the extent to which subscores provide distinct and reliable information under various testing conditions.     

Likert量表分析中不同IRT模型的有效性

5级Likert量表可直接分析,也可以转化为3级评分,或转化为2级评分。前二者可以采用等级IRT模型,后者可以采用2级IRT模型。研究表明2级IRT模型中的2参数模型是最适合的模型。多级评分模型与数据拟合也很好,而且等级越多测量精度越大。     

群体水平领域分数及其估计方法

对人才的需求已经引起各国政府和国际组织对教育的高度重视,纷纷在国家和地区层面进行大规模的教育评估。在大尺度教育评估中,如何向政府、管理者和公众报告学生表现是不可避免的重要问题。报告学生表现有多种方式,领域分数作为管理者和公众最容易理解和接受的分数报告工具之一,在近些年受到研究者和实践者的关注,因此也成为了大型教育评价项目的必然选择。文中将介绍群体领域分数的起源和定义,并重点介绍群体领域分数的估计方法和相关研究,最后对未来开展进一步研究进行展望。     

四参数模型对被试作答异常现象的拟合与纠正

心理与教育测验中存在着被试作答异常现象(能力测验中的猜测现象和睡眠现象, 人格测验中的非0下渐近线现象和非1上渐近线现象), 会导致被试能力或人格特征的测量偏差。在能力测验中, 研究者已提出了多种方法来纠正猜测现象和睡眠现象, 这些方法往往需要调整或删除被试作答信息, 而四参数模型不需要改变被试作答信息而能有效纠正被试能力高估或低估现象。在人格测验中存在着非0下渐近线和非1上渐近线现象, 四参数模型能增强测验项目拟合性能, 提高人格测验的准确性。     

群体水平IRT模型及其应用——兼与IRT的比较

传统上,群体评估都是以个体的评估结果的平均值为基础进行的。而群体水平IRT理论则可以避开对个体的评估,直接实现对群体的评估,它具有许多传统方法难以企及的优点。本文将群体水平IRT模型应用于2007年某省高考英语阅读理解的410所学校的能力评估,评估结果发现：410所学校的英语阅读理解能力几乎都在[-1,1]区间内,没有能力极高或极低的学校。对这些学校而言,测验中所有项目的难度较易,区分度适中。所有的评估结果与IRT模型的评估结果在 的水平上相关显著,表明GIRT模型在实践中是可以选择的一种群体评估方法。     

Categories and Dimensions

ABSTRACT— The distinction between categories and dimensions has important consequences for basic and applied science in many areas of psychological research. Decisions as to whether individuals should be assigned to groups or located along one or more continua often are based on personal preferences or discipline-specific measurement traditions, which can lead to the creation, use, or reification of spurious categories or dimensions. Methods for evaluating the latent structure of psychological constructs, using powerful and informative tests between competing models, are available. Rather than choosing on a priori grounds, investigators can perform structural research to evaluate the strength and consistency with which results tease apart categorical and dimensional models. Here, we review why researchers should make this distinction empirically, briefly discuss methods available for doing so, and describe the breadth of areas ripe for exploiting the largely untapped potential of structural research.