Criterion-related construct validity

Established results on latent variable models are applied to the study of the validity of a psychological test. When the test predicts a criterion by measuring a unidimensional latent construct, not only must the total score predict the criterion, but the joint distribution of criterion scores and item responses must exhibit a certain pattern. The presence of this population pattern may be tested with sample data using the stratified Wilcoxon rank sum test. Often, criterion information is available only for selected examinees, for instance, those who are admitted or hired. Three cases are discussed: (i) selection at random, (ii) selection based on the current test, and (iii) selection based on other measures of the latent construct. Discriminant validity is also discussed.This work was supported in part by Grant SES-87-01890 from the Measurement Methods and Data Improvement Program of the U.S. National Science Foundation.     

驾驶经验和危险类型对驾驶员危险检测的影响

结合反应时和信号检测方法,考察不同危险类型下驾驶员危险检测的特点。35名新手和35名有经验驾驶员依次完成一个反应时测试和一个信号检测任务。结果发现：新手对明显和隐藏危险的反应时间比有经验驾驶员长。与隐藏危险相比,驾驶员对明显危险的敏感性高、判断标准低。驾驶员对两类危险的反应时间长是因为他们的判断标准严格。研究结果表明,驾驶员危险检测的特点随着危险类型不同而变化。     

Does inconsistency always lead to negative affect? The influence of need for closure on affective reactions to cognitive inconsistency

We present an experiment showing that need for closure (NFC)—defined as the epistemic desire for certainty—can moderate individuals' affective reactions to cognitive inconsistency. Informed by Kruglanski and colleagues' new theory, that cognitive inconsistency elicits negative affect particularly under certain circumstances, we find that NFC (i.e. the desire for certain, stable and unambiguous knowledge) influences the strength of consistency effects and resulting negative affect. More specifically, we find that individuals who are high on NFC experience more negative affect upon encountering an inconsistent (vs. consistent) cognition. However, when individuals are low on NFC, inconsistency is irrelevant, and their affect depends on whether the ultimate outcome of the cognition is positive or negative. We discuss the theoretical and practical implications of this research.     

Modeling Asymmetry in the Time–Distance Relation of Ordinal Personality Items

In analyzing responses and response times to personality questionnaire items, models have been proposed which include the so-called “inverted-U effect.” These models predict that response times to personality test items decrease as the latent trait value of a given person gets closer to the attractiveness of an item. Initial studies into these models have focused on dichotomous personality items, and more recently, models for Likert-type scale items have been proposed. In all these models, it is assumed that the inverted-U effect is symmetrical around 0, while, as will be explained in this article, there are substantive and statistical reasons to study this assumption. Therefore, in this article, a general inverted-U model is proposed which accommodates two sources of asymmetry between the response times and the attractiveness of the items. The viability of this model is demonstrated in a simulation study, and the model is applied to the responses and response times of the Temperament and Character Inventory–Revised, covering a broad range of personality dimensions.     

EARLY RESPONSE DISTRIBUTION AND OUTCOMES OF RESPONSE-RESTRICTION ANALYSES

We examined whether the results of a response‐restriction analysis (RRA) could be predicted on the basis of response distribution in early sessions, when these sessions indicated interaction with multiple items. Four preschool‐aged children participated. For 3 of the 4 participants, the results from sessions conducted prior to restriction of the first item corresponded closely with results of the full RRA.     

THE EFFECTS OF FIXED-TIME REINFORCEMENT SCHEDULES ON FUNCTIONAL RESPONSE CLASSES: A TRANSLATIONAL STUDY

Research on functional response classes has applied significance because less severe forms of problem behavior have been found to co‐occur with more severe forms. In addition, the most severe forms of problem behavior are sometimes targeted for intervention without monitoring other less severe forms. In such cases, it is unknown whether and how untreated forms of problem behavior covary with the targeted behaviors. The present study employed a translational procedure (with button presses as the target behavior) to investigate response covariation under noncontingent reinforcement with typically developing preschoolers. The results indicated that noncontingent reinforcement was generally effective in decreasing all response class members when only one member was targeted.     

INFLUENCES OF RESPONSE RATE AND DISTRIBUTION ON THE CALCULATION OF INTEROBSERVER RELIABILITY SCORES

We examined the effects of several variations in response rate on the calculation of total, interval, exact‐agreement, and proportional reliability indices. Trained observers recorded computer‐generated data that appeared on a computer screen. In Study 1, target responses occurred at low, moderate, and high rates during separate sessions so that reliability results based on the four calculations could be compared across a range of values. Total reliability was uniformly high, interval reliability was spuriously high for high‐rate responding, proportional reliability was somewhat lower for high‐rate responding, and exact‐agreement reliability was the lowest of the measures, especially for high‐rate responding. In Study 2, we examined the separate effects of response rate per se, bursting, and end‐of‐interval responding. Response rate and bursting had little effect on reliability scores; however, the distribution of some responses at the end of intervals decreased interval reliability somewhat, proportional reliability noticeably, and exact‐agreement reliability markedly.     

Multilevel Reliability Measures of Latent Scores Within an Item Response Theory Framework

Abstract

This paper evaluated multilevel reliability measures in two-level nested designs (e.g., students nested within teachers) within an item response theory framework. A simulation study was implemented to investigate the behavior of the multilevel reliability measures and the uncertainty associated with the measures in various multilevel designs regarding the number of clusters, cluster sizes, and intraclass correlations (ICCs), and in different test lengths, for two parameterizations of multilevel item response models with separate item discriminations or the same item discrimination over levels. Marginal maximum likelihood estimation (MMLE)-multiple imputation and Bayesian analysis were employed to evaluate the accuracy of the multilevel reliability measures and the empirical coverage rates of Monte Carlo (MC) confidence or credible intervals. Considering the accuracy of the multilevel reliability measures and the empirical coverage rate of the intervals, the results lead us to generally recommend MMLE-multiple imputation. In the model with separate item discriminations over levels, marginally acceptable accuracy of the multilevel reliability measures and empirical coverage rate of the MC confidence intervals were found in a limited condition, 200 clusters, 30 cluster size, .2 ICC, and 40 items, in MMLE-multiple imputation. In the model with the same item discrimination over levels, the accuracy of the multilevel reliability measures and the empirical coverage rate of the MC confidence intervals were acceptable in all multilevel designs we considered with 40 items under MMLE-multiple imputation. We discuss these findings and provide guidelines for reporting multilevel reliability measures.     

An IRT analysis of motive questionnaires: The Unified Motive Scales

Multiple inventories claiming to assess the same explicit motive (achievement, power, or affiliation) show only mediocre convergent validity. In three studies (N = 1685) the structure, nomological net, and content coverage of multiple existing motive scales was investigated with exploratory factor analyses. The analyses revealed four approach factors (achievement, power, affiliation, and intimacy) and a general avoidance factor with a facet structure. New scales (the Unified Motive Scales; UMS) were developed using IRT, reflecting these underlying dimensions. In comparison to existing questionnaires, the UMS have the highest measurement precision and provide short (six-item) and ultra-short (three-item) scales. In a fourth study (N = 96), the UMS demonstrated incremental validity over existing motive scales with respect to several outcome criteria.