A review of the observational data-collection and reliability procedures reported in The Journal of Applied Behavior Analysis

The research published in the Journal of Applied Behavior Analysis (1968 to 1975) was surveyed for three basic elements: data-collection methods, reliability procedures, and reliability scores. Three-quarters of the studies reported observational data. Most of these studies' observational methods were variations of event recording, trial scoring, interval recording, or time-sample recording. Almost all studies reported assessment of observer reliability, usually total or point-by-point percentage agreement scores. About half the agreement scores were consistently above 90%. Less than one-quarter of the studies reported that reliability was assessed at least once per condition.     

单维测验合成信度三种区间估计的比较

已有许多研究建议使用合成信度来估计测验信度, 并报告其置信区间。有三种方法或途径可以计算单维测验合成信度的置信区间, 包括Bootstrap法、Delta法和直接用统计软件(如LISREL)输出的标准误进行计算。本文通过模拟研究进行比较, 发现Delta法与Bootstrap法得到的置信区间相当接近, 但用LISREL输出的标准误计算的与Bootstrap法得到的结果相差很大。推荐用Delta法估计合成信度的置信区间(使用Mplus容易实现), 但不能直接用LISREL输出的标准误来计算。举例说明了如何计算单维测验的合成信度以及用Delta法计算其置信区间。     

Critical issues in psychological autopsy studies

This paper reviews research based on the psychological autopsy (PA) method applied to the study of suicide. It evidences the presence of a number of methodological problems. Shortcomings concern sampling biases in the selection of control subjects, confounding influences of extraneous variables, and reliability of the assessment instruments. The absence of homogeneity among studies in the procedure employed, as well as the lack of defined guidelines for performing this type of inquiry are emphasized. Questions needing empirical investigation in the future are pointed out. It is concluded that the validity and reliability of findings emerging from the use of this method of investigation would benefit from a standardization of its application.     

Reliability of Summed Item Scores Using Structural Equation Modeling: An Alternative to Coefficient Alpha

A method is presented for estimating reliability using structural equation modeling (SEM) that allows for nonlinearity between factors and item scores. Assuming the focus is on consistency of summed item scores, this method for estimating reliability is preferred to those based on linear SEM models and to the most commonly reported estimate of reliability, coefficient alpha.     

Towards an epistemology of religious traditions

Summary Starting from the acceptance of the Egalitarian Principle I exhibited a version which I considered too lax (BEP) and one I considered too strict (NEP), arriving at a version (MEP) which allows that there can be tolerance-limiting reasons for adhering to traditions but only if they are based on unreasoned knowledge claims. In fact, I hold that the situation most of us find ourselves in restricts such claims on religious topics to very general ones. Hence the choice between NEP and MEP is not significant. It follows that we should take up one of two positions concerning religious traditions: either we argue from the shared assumptions of a variety of traditions without genuine participation in any of them; or we justify participation in one of them by noting various marks of reliability, such as serendipitous understanding. A version of this paper was read at the Faith and Reason Conference held at the Catholic Institute in Strathfield, October 5 and 6, 1996. I would like to thank all who participated in the discussion of my paper on that occasion. I would also like to thank the anonymous referees for Sophia for their helpful comments.     

An investigation of the reliability of the Rosenzweig Picture-Frustration (P-F) Study, Children's Form

Presents data on the retest and split-half reliability of the Rosenzweig Picture-Frustration (P-F) Study, Children's Form, for two groups of subjects (aged 10-11 and 12-13 years, respectively), each group tested twice at an interval of three months. It was found that the retest reliability for all scoring categories (except O-D) were statistically significant (.01 or .05 level). GCR proved stable for the younger group (r = .53) but not for the older group. The difference here may be due to the less comfortable retest conditions prevailing for the older subjects. Reliability by retest was consistently higher than by the less appropriate split-half method, but the latter proved, nevertheless, to be in the main favorable.     

A Recursive Partitioning Method for the Prediction of Preference Rankings Based Upon Kemeny Distances

Preference rankings usually depend on the characteristics of both the individuals judging a set of objects and the objects being judged. This topic has been handled in the literature with log-linear representations of the generalized Bradley-Terry model and, recently, with distance-based tree models for rankings. A limitation of these approaches is that they only work with full rankings or with a pre-specified pattern governing the presence of ties, and/or they are based on quite strict distributional assumptions. To overcome these limitations, we propose a new prediction tree method for ranking data that is totally distribution-free. It combines Kemeny’s axiomatic approach to define a unique distance between rankings with the CART approach to find a stable prediction tree. Furthermore, our method is not limited by any particular design of the pattern of ties. The method is evaluated in an extensive full-factorial Monte Carlo study with a new simulation design.     

Meta-interpretive reliability of computer-based test interpretations: the Karson Clinical Report.

Meta-interpretive reliability is a new method to evaluate the accuracy with which personality trait scores are communicated via interpretive statements in a computer-based test interpretation (CBTI). The prototypic experimental design is based on a two-way repeated measures analysis of variance (ANOVA); the two effects are personality traits and randomly chosen CBTI protocols. In this application, 101 psychologists read four examples of the Karson Clinical Report (KCR, Karson & O'Dell, 1975) and estimated the original trait scores from the Sixteen Personality Factor Questionnaire (16PF; Cattell, Eber, & Tatsuoka, 1970) on which the KCR is based. Estimated trait score variance was significantly related to the Trait x Protocol interaction and the main effects for personality trait and differences among protocols (omega 2 = .55). The total effect size corresponded to a multiple correlation of .74, suggesting that the KCR had acceptable meta-interpretive reliability. The protocol effect denoted a context effect created by the juxtaposition of several interpretive statements. Additional analyses showed that individual differences among raters contributed to less than 1% of the estimated standard ten (sten) score variance. Meta-interpretive reliability is proposed as an index of the upper limit of validity for CBTIs.     

A method for coping with overall productivity when making between-group category comparisons

A straightforward method for dealing with the effect of response productivity when comparing individual scoring categories between groups was presented along with an example. It was contended that percentage comparisons based on group data can circumvent the reliability problems associated with percentage scores derived from single protocols. Other problems connected with ratio and percentage scores were discussed.     

New aspects of the probabilistic evaluation of hypotheses and experience

The probabilistic corroboration of two or more hypotheses or series of observations may be performed additively or multiplicatively . For additive corroboration (e.g. by Laplace's rule of succession), stochastic independence is needed. Inferences, based on overwhelming numbers of observations without unexplained counterinstances permit hyperinduction , whereby extremely high probabilities, bordering on certainty for all practical purposes may be achieved. For multiplicative corroboration, the error probabilities (1 - Pr) of two (or more) hypotheses are multiplied. The probabilities, obtained by reconverting the product, are valid for both of the hypotheses and indicate the gain by corroboration.. This method is mathematically correct, no probabilities > 1 can result (as in some conventional methods) and high probabilities with fewer observations may be obtained, however, semantical independence is a prerequisite. The combined method consists of (1) the additive computation of the error probabilities (1 - Pr) of two or more single hypotheses, whereby arbitrariness is avoided or at least reduced and (2) the multiplicative procedure . The high reliability of Empirical Counterfactual Statements is explained by the possibility of multiplicative corroboration of “all-no” statements due to their strict semantical independence.     

类别学习的SUSTAIN模型

SUSTAIN模型是关于类别学习的类群或群集规则的选择模型,它强调多层次的类别子结构和模型的结构搜索功能.模型的运行以类别相似性为基础,从最简单的规则开始,有灵活的参数及其运算过程,与其他主要的类别学习模型相比有更多的优势.因此,SUSTAIN模型是迄今解释人类类别学习的最优模型.     

两水平研究中单维测验信度的估计

在心理、教育和管理等研究领域中,经常会碰到两水平（两层）的数据结构,如学生嵌套在班级中,员工嵌套在企业中。在两水平研究中,被试通常不是独立的,如果直接用单水平信度公式进行估计,会高估测验信度。文献上已有研究讨论如何更准确地估计两水平研究中单维测验的信度。本研究指出了现有的估计公式的不足之处,用两水平验证性因子分析推导出一个新的信度公式,举例演示如何计算,并给出简单的计算程序。     

The Psychometric Evaluation of the Satisfaction with Life Scale Using a Nationally Representative Sample of China

This study examined the psychometric properties of the Satisfaction with Life Scale (SWLS) in terms of internal consistency reliability, factorial validity, and measurement invariance across gender, age, residential region (metropolitan area/county town/rural area), educational level, and household income with a nationally representative sample of 4,795 participants in China. The percentages of male (47.0%) and female (53.0%) participants were about equal, and the mean age was 38.56 years (SD = 11.87). The results show that the SWLS has a high internal consistency reliability, a one-factor structure (with a wording effect on Items 4 and 5), strict invariance across gender, partial strict invariance across education (Items 1, 2 and 3), and partial strong invariance across age, income (Items 1, 2 and 3) and residential region (Items 1 and 2). The non-invariance of some items across age and region is discussed in terms of the rapid economic and social transitions of China in the past three decades.     

The Psychodynamic Diagnostic Manual (PDM): rationale, conception, and structure

The declaredly atheoretical DSM-III (and its successors), the diagnostic manual of the American Psychiatric Association, was created to enhance diagnostic reliability for research, epidemiological survey, and governmental and insurance categorization and reimbursement purposes. It has, however, exhibited many inadequacies for psychodynamic diagnosis and case formulation for treatment planning and outcome assessment, and its claimed diagnostic reliability has turned out to be less than originally projected. The psychoanalytically sponsored Psychodynamic Diagnostic Manual (PDM) marks a return to a theoretically (psychoanalytically) based diagnostic frame and was created as a supplement to, or replacement for, DSM (depending on the precise clinical need), for use in psychodynamic diagnosis and treatment planning.     

使用Bootstrap方法计算认知诊断评估中的信度

测验信度是衡量测验质量的一个重要指标,认知诊断评估中同样需要重视信度问题。现有认知诊断中计算信度的方法均有一个前提假设:被试在前后两次测验的后验概率分布和边际概率完全相同。该假设过强,未考虑两次测验间存在的随机误差。基于Bootstrap抽样,提出了两类属性信度和模式信度的指标,分别是积差相关法和修正的一致性法。通过模拟研究比较了新方法和现有方法在不同属性个数、属性间相关性和题目数量下的表现,并基于英语能力认证考试ECPE和分数减法的实证数据验证了新方法的可行性。最后,对信度估计的影响因素进行了讨论。     

A Direct Method for Obtaining Approximate Standard Error and Confidence Interval of Maximal Reliability for Composites With Congeneric Measures

Unlike a substantial part of reliability literature in the past, this article is concerned with weighted combinations of a given set of congeneric measures with uncorrelated errors. The relationship between maximal coefficient alpha and maximal reliability for such composites is initially dealt with, and it is shown that the former is a lower bound of the latter. A direct method for obtaining approximate standard error and confidence interval for maximal reliability is then outlined. The procedure is based on a second-order Taylor series approximation and is readily and widely applicable in empirical research via use of covariance structure modeling. The described method is illustrated with a numerical example.     

Exploring trust of mobile applications based on user behaviors: an empirical study

This paper explores trust of mobile applications based on users' behaviors. It proposes a trust behavior construct through principal component analysis, reliability analysis, and confirmatory factor analysis based on the data collected from a questionnaire survey with more than 1,500 participants. It is indicated that a user's trust behavior is composed of three principal constructs: using behavior, reflection behavior, and correlation behavior. They are further delineated into 12 measurable sub‐constructs and relate to a number of external factors. The data analysis showed that the questionnaire has positive psychometric properties with respect to construct validity and reliability. We also discuss the practical significance and limitations of our work toward usable trust management.     

The persuasiveness puzzle about bootstrapping

This paper aims at resolving a puzzle about the persuasiveness of bootstrapping. On the one hand, bootstrapping is not a persuasive method of settling questions about the reliability of a source. On the other hand, our beliefs that our sense apparatus is reliable is based on other empirically formed beliefs, that is, they are acquired via a presumably complex bootstrapping process. I will argue that when we doubt the reliability of a source, bootstrapping is not a persuasive method for coming to believe that the source is reliable. However, when being initially unaware of a source and its reliability, as in the case of forming beliefs about our sense apparatus, bootstrapping can be eventually persuasive.