Psychometric examination of the short version of the learning climate questionnaire using item response theory

Self-determination theory proposes that autonomy support in the classroom is critical for students’ optimal motivation and performance. However, the literature has not adequately demonstrated the psychometric qualities of the most popular measurement for autonomy-supportive classrooms, the Learning Climate Questionnaire (LCQ) and its short version. Using the graded response model in item response theory (IRT), the current study evaluates the short version of the LCQ with a large sample (N?=?13570). IRT and classic psychometric analyses show that the scale is generally satisfactory in measuring latent learning climate, with the exceptions that Item 4 appears to be inadequate and that the scale is relatively weak in distinguishing highly autonomy-supportive classrooms. We provide suggestions for future studies, such as dropping Item 4 and including more items that tap into instructional practices located on the higher end of the latent autonomy support spectrum. Implications of the current findings for the conceptualization of autonomy support are also discussed.     

A dual process item response theory model for polytomous multidimensional forced-choice items

The use of multidimensional forced-choice (MFC) items to assess non-cognitive traits such as personality, interests and values in psychological tests has a long history, because MFC items show strengths in preventing response bias. Recently, there has been a surge of interest in developing item response theory (IRT) models for MFC items. However, nearly all of the existing IRT models have been developed for MFC items with binary scores. Real tests use MFC items with more than two categories; such items are more informative than their binary counterparts. This study developed a new IRT model for polytomous MFC items based on the cognitive model of choice, which describes the cognitive processes underlying humans' preferential choice behaviours. The new model is unique in its ability to account for the ipsative nature of polytomous MFC items, to assess individual psychological differentiation in interests, values and emotions, and to compare the differentiation levels of latent traits between individuals. Simulation studies were conducted to examine the parameter recovery of the new model with existing computer programs. The results showed that both statement parameters and person parameters were well recovered when the sample size was sufficient. The more complete the linking of the statements was, the more accurate the parameter estimation was. This paper provides an empirical example of a career interest test using four-category MFC items. Although some aspects of the model (e.g., the nature of the person parameters) require additional validation, our approach appears promising.     

Further examination of the psychometric properties of the comprehensive effects of alcohol questionnaire

Although the expectancies component of the Comprehensive Effects of Alcohol Questionnaire has previously been shown to be factorially valid, the factor structure of its valuations component has not previously been examined. The aims of this paper were: (i) to replicate the factor structure of the expectancies items; (ii) to explore the factor structure of the valuations items; and (iii) to investigate the utility of using the Comprehensive Effects of Alcohol Questionnaire to predict drinking behavior. The questionnaire was administered to 1004 university students along with measures of quantity and frequency of alcohol consumption. Fromme, Stroot, and Kaplan's (1993) factor structure of the expectancies scales was replicated. The factor structures of the negative valuations scales were characterized by 2 rather than 3 factors. Negative expectancies improved upon the prediction of drinking quantity and frequency over-and-above positive expectancies, and valuations further improved prediction over-and-above expectancies. Theoretical and clinical implications are discussed.     

A note on monotonicity of item response functions for ordered polytomous item response theory models

A monotone relationship between a true score (τ) and a latent trait level (θ) has been a key assumption for many psychometric applications. The monotonicity property in dichotomous response models is evident as a result of a transformation via a test characteristic curve. Monotonicity in polytomous models, in contrast, is not immediately obvious because item response functions are determined by a set of response category curves, which are conceivably non-monotonic in θ. The purpose of the present note is to demonstrate strict monotonicity in ordered polytomous item response models. Five models that are widely used in operational assessments are considered for proof: the generalized partial credit model (Muraki, 1992, Applied Psychological Measurement, 16, 159), the nominal model (Bock, 1972, Psychometrika, 37, 29), the partial credit model (Masters, 1982, Psychometrika, 47, 147), the rating scale model (Andrich, 1978, Psychometrika, 43, 561), and the graded response model (Samejima, 1972, A general model for free-response data (Psychometric Monograph no. 18). Psychometric Society, Richmond). The study asserts that the item response functions in these models strictly increase in θ and thus there exists strict monotonicity between τ and θ under certain specified conditions. This conclusion validates the practice of customarily using τ in place of θ in applied settings and provides theoretical grounds for one-to-one transformations between the two scales.     

An item response model with internal restrictions on item difficulty

An IRT model based on the Rasch model is proposed for composite tasks, that is, tasks that are decomposed into subtasks of different kinds. There is one subtask for each component that is discerned in the composite tasks. A component is a generic kind of subtask of which the subtasks resulting from the decomposition are specific instantiations with respect to the particular composite tasks under study. The proposed model constrains the difficulties of the composite tasks to be linear combinations of the difficulties of the corresponding subtask items, which are estimated together with the weights used in the linear combinations, one weight for each kind of subtask. Although the model does not belong to the exponential family, its parameters can be estimated using conditional maximum likelihood estimation. The approach is demonstrated with an application to spelling tasks. We thank Eric Maris for his helpful comments.     

Measuring change for a multidimensional test using a generalized explanatory longitudinal item response model

Even though many educational and psychological tests are known to be multidimensional, little research has been done to address how to measure individual differences in change within an item response theory framework. In this paper, we suggest a generalized explanatory longitudinal item response model to measure individual differences in change. New longitudinal models for multidimensional tests and existing models for unidimensional tests are presented within this framework and implemented with software developed for generalized linear models. In addition to the measurement of change, the longitudinal models we present can also be used to explain individual differences in change scores for person groups (e.g., learning disabled students versus non‐learning disabled students) and to model differences in item difficulties across item groups (e.g., number operation, measurement, and representation item groups in a mathematics test). An empirical example illustrates the use of the various models for measuring individual differences in change when there are person groups and multiple skill domains which lead to multidimensionality at a time point.     

Initial psychometric properties of the experiences questionnaire: validation of a self-report measure of decentering

Decentering is defined as the ability to observe one's thoughts and feelings as temporary, objective events in the mind, as opposed to reflections of the self that are necessarily true. The Experiences Questionnaire (EQ) was designed to measure both decentering and rumination but has not been empirically validated. The current study investigated the factor structure of the EQ in both undergraduate and clinical populations. A single, unifactorial decentering construct emerged using 2 undergraduate samples. The convergent and discriminant validity of this decentering factor was demonstrated in negative relationships with measures of depression symptoms, depressive rumination, experiential avoidance, and emotion regulation. Finally, the factor structure of the EQ was replicated in a clinical sample of individuals in remission from depression, and the decentering factor evidenced a negative relationship to concurrent levels of depression symptoms. Findings from this series of studies offer initial support for the EQ as a measure of decentering.     

Preliminary examination of the psychometric properties of the Psychiatric Scepticism Scale

Despite much commentary on "psychiatric skepticism", there remains a dearth of appropriate psychometric scales for the measurement of this concept. To overcome this limitation, the present study examined the psychometric properties and correlates of the recently developed Psychiatric Scepticism Scale (PSS). A total of 564 individuals from the community in London, England, completed the PSS along with measures of anti-scientific attitudes, attitudes to authority, knowledge of mental health disorders, and demographics. Results showed that the PSS has a one-dimensional factor structure with very good internal consistency. In addition, it showed adequate convergent (anti-scientific attitudes, knowledge of mental health disorders) and construct validity (attitudes to authority, religiosity). Results also showed that there were small but significant differences in psychiatric skepticism by ethnicity and education, but not sex or previous diagnosis of mental health disorder. These results confirm that the PSS has adequate psychometric properties for the measurement of psychiatric skepticism.     

Diagnosing item score patterns on a test using item response theory-based person-fit statistics

Person-fit statistics have been proposed to investigate the fit of an item score pattern to an item response theory (IRT) model. The author investigated how these statistics can be used to detect different types of misfit. Intelligence test data were analyzed using person-fit statistics in the context of the G. Rasch (1960) model and R. J. Mokken's (1971, 1997) IRT models. The effect of the choice of an IRT model to detect misfitting item score patterns and the usefulness of person-fit statisticsfor diagnosis of misfit are discussed. Results showed that different types of person-fit statistics can be used to detect different kinds of person misfit. Parametric person-fit statistics had more power than nonparametric person-fit statistics.     

多级计分测验中基于残差统计量的被试拟合研究

本文提出一种多级计分项目下的个人拟合统计量R, 考察它在检测6种常见的异常作答模式(作弊、猜测、随机、粗心、创新作答、混合异常)下的表现, 并与标准化对数似然统计量l_zp进行比较。结果表明：(1) 在异常作答覆盖率较低并且异常作答类型为作弊和猜测时, R的检测率显著高于l_zp; (2) 随着测验长度和被试异常程度的增加, 两种统计量的检测率都会上升; (3) 在一些条件下, R与l_zp检测效果接近。实证数据分析进一步展示了R统计量的使用方法和过程, 结果也表明R统计量具有较好的应用前景。     

An item and radial parcel factor analysis of the 16PF questionnaire

An extensive series of analyses were carried out on a sample of data from 491 undergraduate university students who completed Form A of Cattell's 16PF questionnaire. The data was item analysed, factored using both principal component and image analyses, and radial parcelled. However, even though five different factor solutions were rotated to a maximum simple structure, the 16 factors did not emerge as expected. Radial parcelling also yielded equivocal results. Using only psychometric criteria to guide the analysis, three new factor scales were generated that satisfied the test of high factor validity and high coefficient alpha simultaneously for each scale. The overall solution yielded seven factored scales. Additionally, results were reported of a scale factoring of the 16 scales yielding a replicable 4-factor solution. An alternative 7-factor solution was not replicable among subsamples taken from the total data set.     

An examination of the psychometric properties of scores on the Family Strain Index (FSI) in a general community sample

Given the potentially harmful effects of parenting stress on parents, children, and their relationship, it is critical to have a reliable and valid measure of parenting stress in clinical and community samples. The Family Strain Index (FSI) is a brief questionnaire designed to measure stress and demand on parents of children with ADHD. The present study is the first to evaluate the psychometric properties of scores on the FSI in a general community sample. Parents (89% mothers) of 550 preschool children (aged 2–5 years; 50% boys) sampled through 17 kindergartens located in Danish cities and villages completed the FSI, the ADHD Rating Scale (RS)‐IV Preschool Version, and a background questionnaire. FSI scores were characterized by restricted range and floor effects. The scale's construct validity was not supported and the measurement repeatability after 1 month was low. The scale did have convergent validity as levels of parenting stress were associated with perceived ADHD behavior in off‐spring, but overall, results did not encourage the use of the FSI as a measure of parenting stress in the general population. Measures that include more normative events may be more appropriate when attempting to capture parenting stress in general community samples.     

An item response tree model with not-all-distinct end nodes for non-response modelling

The non-response model in Knott et al. (1991, Statistician, 40, 217) can be represented as a tree model with one branch for response/non-response and another branch for correct/incorrect response, and each branch probability is characterized by an item response theory model. In the model, it is assumed that there is only one source of non-responses. However, in questionnaires or educational tests, non-responses might come from different sources, such as test speededness, inability to answer, lack of motivation, and sensitive questions. To better accommodate such more realistic underlying mechanisms, we propose a a tree model with four end nodes, not all distinct, for non-response modelling. The Laplace-approximated maximum likelihood estimation for the proposed model is suggested. The validation of the proposed estimation procedure and the advantage of the proposed model over traditional methods are demonstrated in simulations. For illustration, the methodologies are applied to data from the 2012 Programme for International Student Assessment (PISA). The analysis shows that the proposed tree model has a better fit to PISA data than other existing models, providing a useful tool to distinguish the sources of non-responses.     

The child PTSD Symptom Scale: a preliminary examination of its psychometric properties

Reports on the development and preliminary validation of the Child PTSD Symptom Scale (CPSS) for children and adolescents. The CPSS is a new instrument that was developed to assess the severity of Diagnostic and Statistical Manual of Mental Disorders (4th ed.; American Psychiatric Association, 1994) posttraumatic stress disorder symptoms in children exposed to trauma. The CPSS was administered to 75 school-age children approximately 2 years after the 1994 Northridge, California, earthquake. The psychometric properties of the CPSS show high internal consistency and test-retest reliability for both the total score and the three subscales. Convergent validity with the Child Post-Traumatic Stress Disorder Reaction Index (CPTSD-RI) was established. As expected, the correlations of the CPSS with depression and anxiety measures were lower than those with the CPTSD-RI, providing some support for discriminant validity of the CPSS. These results suggest that the CPSS is a useful tool for the assessment of posttraumatic stress disorder (PTSD) severity and for the screening of PTSD diagnosis among traumatized children.     

Analysis of residuals for the multionmial item response model

Using the item response model as developed on the multinomial distribution, asymptotic variances are obtained for residuals associated with response patterns and first-, and second-order marginal frequencies of manifest variables. When the model does not fit well, an examination of these residuals may reveal the source of the poor fit. Finally, a limited-information test of fit for the model is developed by using residuals defined for the first-, and second-order marginals. Model evaluation based on residuals for these marginals is particularly useful when the response pattern frequencies are sparse.The author would like to thank Yasuo Amemiya and Joseph Lucke for helpful suggestions. This research was supported by a Research Incentive Grant from Arizona State University.     

Multilevel multidimensional item response model with a multilevel latent covariate

In a pre‐test–post‐test cluster randomized trial, one of the methods commonly used to detect an intervention effect involves controlling pre‐test scores and other related covariates while estimating an intervention effect at post‐test. In many applications in education, the total post‐test and pre‐test scores, ignoring measurement error, are used as response variable and covariate, respectively, to estimate the intervention effect. However, these test scores are frequently subject to measurement error, and statistical inferences based on the model ignoring measurement error can yield a biased estimate of the intervention effect. When multiple domains exist in test data, it is sometimes more informative to detect the intervention effect for each domain than for the entire test. This paper presents applications of the multilevel multidimensional item response model with measurement error adjustments in a response variable and a covariate to estimate the intervention effect for each domain.     

Estimation of a four‐parameter item response theory model

We explore the justification and formulation of a four‐parameter item response theory model (4PM) and employ a Bayesian approach to recover successfully parameter estimates for items and respondents. For data generated using a 4PM item response model, overall fit is improved when using the 4PM rather than the 3PM or the 2PM. Furthermore, although estimated trait scores under the various models correlate almost perfectly, inferences at the high and low ends of the trait continuum are compromised, with poorer coverage of the confidence intervals when the wrong model is used. We also show in an empirical example that the 4PM can yield new insights into the properties of a widely used delinquency scale. We discuss the implications for building appropriate measurement models in education and psychology to model more accurately the underlying response process.     

An item factor analysis and item response theory-based revision of the Everyday Discrimination Scale

The Everyday Discrimination Scale (EDS), a widely used measure of daily perceived discrimination, is purported to be unidimensional, to function well among African Americans, and to have adequate construct validity. Two separate studies and data sources were used to examine and cross-validate the psychometric properties of the EDS. In Study 1, an exploratory factor analysis was conducted on a sample of African American law students (N = 589), providing strong evidence of local dependence, or nuisance multidimensionality within the EDS. In Study 2, a separate nationally representative community sample (N = 3,527) was used to model the identified local dependence in an item factor analysis (i.e., bifactor model). Next, item response theory (IRT) calibrations were conducted to obtain item parameters. A five-item, revised-EDS was then tested for gender differential item functioning (in an IRT framework). Based on these analyses, a summed score to IRT-scaled score translation table is provided for the revised-EDS. Our results indicate that the revised-EDS is unidimensional, with minimal differential item functioning, and retains predictive validity consistent with the original scale.