ITC Guidelines on Quality Control in Scoring,Test Analysis,and Reporting of Test Scores

The Quality Control (QC) Guidelines are intended to increase the efficiency, precision, and accuracy of the scoring, analysis, and reporting process of testing. The QC Guidelines focus on large-scale testing operations where multiple forms of tests are created for use on set dates. However, they may also be used for a wide variety of other testing situations and assessment techniques and for almost any situation in which assessment occurs. The QC Guidelines are applicable in any form of test administration, including paper and pencil tests and the ever-increasing computerized assessments via the Internet or offline.     

Multilevel Factor Analysis: Reporting Guidelines and a Review of Reporting Practices

We provide reporting guidelines for multilevel factor analysis (MFA) and use these guidelines to systematically review 72 MFA applications in journals across a range of disciplines (e.g., education, health/nursing, management, and psychology) published between 1994 and 2014. Results are organized in terms of the (a) characteristics of the MFA application (e.g., construct measured), (b) purpose (e.g., measurement validation), (c) data source (e.g., number of cases at Level 1 and Level 2), (d) statistical approach (e.g., maximum likelihood), and (e) results reported (e.g., intraclass correlations for indicators and latent variables, standardized factor loadings, fit indices). Results from this review have implications for applied researchers interested in expanding their approaches to psychometric analyses and construct validation within a multilevel framework and for methodologists using Monte Carlo methods to explore technical and methodological issues grounded in realistic research design conditions.     

Dynamic Structure in Diagnostic Structured Interviewing: A Comparative Test of Accuracy and Efficiency

The incorporation of Bayesian logic into diagnostic interviewing may assist with empirically based diagnostic assessment strategies in practice settings, balancing cost effectiveness, administration demands, and accuracy, yet few demonstrations of such a system have been undertaken in the context of mental health diagnosis. The present study represented an initial feasibility demonstration of whether a simplified Bayesian approach offered comparative advantages in interview accuracy and efficiency against a standard assessment procedure. Two different diagnostic algorithms were compared targeting three selected diagnoses: generalized anxiety disorder (GAD), major depressive disorder (MDD), and social phobia (SP). The first algorithm was from a standard semi-structured diagnostic interview, and the second was from a dynamic system using diagnostic base rate information to select interview content. The dynamic algorithm reduced administration time and uniformly matched or improved accuracy over standard procedures. Preparation of this article was supported in part by National Institute of Mental Health Grant R03 MH60134, an award from the University of Hawai‘i Research Council, and awards from the Hawaii Departments of Health and Education to the first author.     

The Concept of Personal Initiative: An Overview of Validity Studies

Construct validity of an interview measure of personal initiative (PI) is examined in two parts. The first part assembles the results from 11 samples, showing that PI is meaningfully related to a nomological network of variables, based on environmental supports; knowledge, skills, and cognitive abilities; personality variables and orientations; and behavior and performance, confirming our hypotheses. In the second part, the article presents a new analysis that looks at the influence of motivational parameters (control aspiration, self-efficacy, and change orientation) and cognitive ability on PI within a longitudinal study in East Germany.     

认知诊断测验蓝图的设计

通常认为由属性和项目关联阵（即Q矩阵）的列对应的项目充任认知诊断测验中行为样本,其实这种做法不能有效防止理想反应模式的误判。如在测验之前便可确定欲测之属性及层级关系,找到可达阵,可证明可达阵的各个列对应的项目类在认知诊断测验中必不可少,否则在理想反应模式下就一定有一些被试会被误判。本文给出充分必要Q矩阵的概念,以区别Tatsuoka(1995,2009) 讨论过的充分Q矩阵概念。充分必要Q矩阵才能有效指导测验的编制。     

The Challenge of Information Technology for Literary Studies

ABSTRACT

This paper identifies the challenge and shows how research done on the Kierkegaard corpus is meeting the challenge. It presents eighteen studies, tracing the development of computer based Kierkegaard research, and suggesting that the computer might be used to produce more than word counts and locations of words or strings of texts. The study introduces the idea of a multi-dimensional concordance as evidenced by recent research on Fear and Trembling.     

Information Distortion and Voting Choices: The Origins and Effects of Factual Beliefs in Initiative Elections

To account for voter decision making in initiative elections, we integrate theory and research on public opinion, misinformation, and motivated reasoning. Heuristic and motivated reasoning literatures suggest that voters' preexisting values interact with political sophistication such that politically knowledgeable voters develop systematically distorted empirical beliefs relevant to the initiatives on their ballots. These beliefs, in turn, can predict voting preferences even after controlling for underlying values, regardless of one's political sophistication. These hypotheses were tested using a 2003 voter survey conducted prior to a statewide initiative election that repealed a workplace safety regulation. Results showed that only those voters knowledgeable of key endorsements had initiative-specific beliefs that lined up with their underlying antiregulation values. Also, voters' empirical beliefs had an effect on initiative support even after controlling for prior values, and political sophistication did not moderate this effect.     

Reporting Diagnostic Scores in Educational Testing: Temptations,Pitfalls, and Some Solutions

Diagnostic scores are of increasing interest in educational testing due to their potential remedial and instructional benefit. Naturally, the number of educational tests that report diagnostic scores is on the rise, as are the number of research publications on such scores. This article provides a critical evaluation of diagnostic score reporting in educational testing. The existing methods for diagnostic score reporting are discussed. A recent method (Haberman, 2008a Haberman, S. J. 2008a. When can subscores have value?. Journal of Educational and Behavioral Statistics, 33: 204–229. [Crossref], [Web of Science ®] , [Google Scholar]) that examines if diagnostic scores are worth reporting is reviewed. It is demonstrated, using results from operational and simulated data, that diagnostic scores have to be based on a sufficient number of items and have to be sufficiently distinct from each other to be worth reporting and that several operationally reported subscores are actually not worth reporting. Several recommendations are made for those interested to report diagnostic scores for educational tests.     

The Szondi Test in a Psychological Battery: Two Case Studies

This study investigated MMPI Characteristics of male and female adolescent inpatients with diagnoses of borderline personality disorder (n = 28) in contrast to adolescent inpatients receiving principal diagnoses of conduct disorder (n = 21), dysthymic disorder (n = 50), other personality disorders (n = 17), and other diagnoses (n = 30). The borderline group has significantly higher elevations than comparison groups on MMPI Scales F, Hs, D, Pd, Pa, Pt, Sc, and Ma. A stepwise discriminant analysis resulted in 82.1%. accuracy in correctly classifying borderline patients and 78.0% accuracy in identifying, nonborderline patients. Findings are discussed in terms of potential uses and limitations in identifying borderline personality disorder with the MMPI.     

认知诊断测验中基于信息矩阵的多群组DIF检验

基于改进的Wald统计量,将适用于两群组的DIF检测方法拓展至多群组的项目功能差异（DIF）检验;改进的Wald统计量将分别通过计算观察信息矩阵（Obs）和经验交叉相乘信息矩阵（XPD）而得到。模拟研究探讨了此二者与传统计算方法在多个群组下的DIF检验情况,结果表明：（1）Obs和XPD的一类错误率明显低于传统方法,DINA模型估计下Obs和XPD的一类错误率接近理论水平;（2）样本量和DIF量较大时,Obs和XPD具有与传统Wald统计量大体相同的统计检验力。     

The Effect of Incentives upon Information Gathering in Diagnostic Problem Solving

Tasks reflecting both Level I and Level II abilities as defined by Jensen (6) were performed with more accuracy by preschool children identified in the upper SES level. This performance trend remained the same even after the variable of IQ was controlled for by covariance for the SES levels involved in the study.

These results may reflect a general state of cognitive deprivation for children in the lower SES level as opposed to a specific Level II deficit. However, the performance on Subtest 2 was not significantly different for the two socioeconomic groups involved. This subtest involves choosing, from an array of four pictures of objects, the picture that is conceptually similar to a stimulus picture presented to the youngster. This is supposedly a Level II task. Therefore, some doubt is cast upon the notion of the generic differences between Level I and Level II abilities. At least for the sample in this study the Level I-Level II dichotomy has not been substantiated, and the corollary Arthur Jensen (6) hypotheses have equivocal substantiation.     

Studies of Family Relations Test Patterns: 1. Test Inhibition

An empirical investigation of Bene and Anthony's (1957) “tenderness vs. toughness” hypothesis of inhibition in boys was conducted. Examination of the Family Relations Test (FRT) protocols of 217 boys (age range, 7 years 2 months to 12 years 10 months; IQ range, 80 to 132) referred to Calgary School Board psychologists, showed Bene and Anthony's hypothesis to be valid in this sample. Evidence is given to suggest that each of the eight scoring categories should be viewed separately for inhibition trends and not summed over any of the three dimensions, intensity, direction, and valence. The relation of FRT Inhibition to reason for referral was examined but only in the eight-year-old group was any significant relationship found.     

基于项目自动生成的认知诊断测验开发

认知诊断、项目自动生成是现代心理测量领域的重要发展领域,二者的结合更是心理测量领域亟待开展的重要课题。本研究以小学数学问题解决认知诊断项目自动生成为例,探讨认知诊断领域的项目生成技术及算法。研究发现：(1)计算机自生成的项目参数与原模板参数具有较高的一致性。(2)同一项目模板下生成的不同试题的测量学特征基本不变。(3)同一批被试在自动生成的两份试卷的前、后测的能力( )值高度相关(r=0.811),前、后两次对被试诊断结果的一致性高达86.5%。这表明本文所设计的认知诊断测验项目的自动生成技术及其算法基本可行,小学数学问题解决认知诊断项目的自动生成效果较好。这也为其它认知诊断领域的项目自动生成提供了技术借鉴和支持。