Efficient nonparametric approaches for estimating the operating characteristics of discrete item responses

Rationale and the actual procedures of two nonparametric approaches, called Bivariate P.D.F. Approach and Conditional P.D.F. Approach, for estimating the operating characteristic of a discrete item response, or the conditional probability, given latent trait, that the examinee's response be that specific response, are introduced and discussed. These methods are featured by the facts that: (a) estimation is made without assuming any mathematical forms, and (b) it is based upon a relatively small sample of several hundred to a few thousand examinees.Some examples of the results obtained by the Simple Sum Procedure and the Differential Weight Procedure of the Conditional P.D.F. Approach are given, using simulated data. The usefulness of these nonparametric methods is also discussed.This research was mostly supported by the Office of Naval Research (N00014-77-C-0360, N00014-81-C-0569, N00014-87-K-0320, N00014-90-J-1456).     

Profiling movement quality and gait characteristics according to body-mass index in children (9–11 y)

Obese children move less and with greater difficulty than their normal-weight counterparts. Whilst the effect of high BMI on cardiovascular fitness is well known, the effect on movement quality characteristics during a standardised fitness test has not been investigated. The aims of this study were, to characterise the movement quality of children performing the multi-stage fitness test (MSFT), and, report how movement quality characteristics cluster according to weight status. One hundred and three children (10.3 ± 0.6 y, 1.42 ± 0.08 m, 37.8 ± 9.3 kg, BMI; 18.5 ± 3.3 kg m²) performed the MSFT whilst wearing an ankle mounted accelerometer. BMI groups were used to classify children as underweight (UW), normal weight (NW), overweight (OW) and obese (OB). Characteristics of movement were profiled using a clustering algorithm. Spearman’s rho was used to assess relationship with BMI group, and a Mann-Whitney U test was used to assess differences between BMI groups. Obese children had significantly lower spectral purity than every other group and significantly lower time to exhaustion (TTE) than UW and NW children (P < 0.05). BMI was clustered with stride profile and TTE with spectral purity. Significant negative correlations (P < 0.05) were found between BMI and TTE (r = −0.25), spectral purity (r = −0.24), integrated acceleration (r = −0.22), stride angle (r = −0.23) and stride variability (r = −0.22). This was the first study to report the spectral purity of children’s gait. Further analysis unveiled key performance characteristics that differed between BMI groups. These were (i) representative of children’s performance during the MSFT and, (ii) significantly negatively correlated with BMI.     

The Development and Testing of a New Version of the Cognitive Reflection Test Applying Item Response Theory (IRT)

The cognitive reflection test (CRT) is a short measure of a person's ability to resist intuitive response tendencies and to produce a normatively correct response, which is based on effortful reasoning. Although the CRT is a very popular measure, its psychometric properties have not been extensively investigated. A major limitation of the CRT is the difficulty of the items, which can lead to floor effects in populations other than highly educated adults. The present study aimed at investigating the psychometric properties of the CRT applying item response theory analyses (a two‐parameter logistic model) and at developing a new version of the scale (the CRT‐long), which is appropriate for participants with both lower and higher levels of cognitive reflection. The results demonstrated the good psychometric properties of the original, as well as the new scale. The validity of the new scale was also assessed by measuring correlations with various indicators of intelligence, numeracy, reasoning and decision‐making skills, and thinking dispositions. Moreover, we present evidence for the suitability of the new scale to be used with developmental samples. Finally, by comparing the performance of adolescents and young adults on the CRT and CRT‐long, we report the first investigation into the development of cognitive reflection. Copyright © 2015 John Wiley & Sons, Ltd.     

Test of memory malingering with children: The utility of Trial 1 and TOMMe10 as screeners of test validity

There is a growing body of research suggesting that the shorter versions of the Test of Memory Malingering (TOMM) may provide an accurate assessment of effort in children. During neuropsychological evaluations, some circumstances result in only one completed trial of the TOMM or partial completion of a trial. Research suggests that a cut-score of 40 or 41 on Trial1 is highly predictive of passing the TOMM overall. In the current study, 194 school-age children with academic and/or behavioral problems were used to compare the accuracy of TOMM1 and TOMMe10 (errors on the first 10 items of TOMM1) in predicting passing/failing of TOMM2. For the children in this sample, a score of < 40 items correct (≥ 10 errors) on TOMM1 was highly accurate in predicting a passing performance on the TOMM2 (sensitivity = .80, specificity = .91) with a Negative Predictive Value = .98 at the malingering base rate of 7% (TOMM2 failure in our sample). A score of 2 errors (8 items correct) on the TOMMe10 was slightly less sensitive than that of the TOMM1 (specificity = .96, sensitivity = .53) but with a similar Negative Predictive Value (.96). Consistent with the research from adult populations, TOMM1 and TOMMe10 appear to be quite accurate in predicting performance on the standard administration of the TOMM and may be useful screeners. However, compared to that found in adult samples, slight differences in suggested cutoffs for TOMM1 and TOMMe10 may be warranted for children.     

Unique designs,errors and strategies in the Five-Point Test: The contribution of age,phonemic fluency and visuospatial abilities in Italian children aged 6–11 years

Performances on the Five-Point Test of 161 Italian children aged 6 to 11 years were investigated, along with phonemic fluency, visual-motor integration, visual perception, motor coordination, visuospatial memory, and fluid intelligence. Five-Point Test accuracy was significantly related to phonemic fluency and visual-motor integration, while phonemic fluency was linked to motor coordination. The two fluency measures increased linearly with age, but the developmental progression of Five-Point Test accuracy was less influenced by age. Different age effects were also found on the relationship between fluid intelligence and the two fluency measures. The inspection of qualitative aspects of Five-Point Test performance (errors and strategies) suggested that strategy usage enhanced both productivity and accuracy in children; age-related changes were observed in the relationship between the number of errors and the total number of designs produced.     

Developing Pairwise Preference-Based Personality Test and Experimental Investigation of Its Resistance to Faking Effect by Item Response Model

Recent years have shown increased awareness of the importance of personality tests in educational, clinical, and occupational settings, and developing faking-resistant personality tests is a very pragmatic issue for achieving more precise measurement. Inspired by Stark (2002) and Stark, Chernyshenko, and Drasgow (2005), we develop a pairwise preference-based personality test that aims to measure multidimensional personality traits using a large-scale statement bank. An experiment compares the resistance of the developed personality test to faking with that of rating scale-based personality tests in the item response theory model framework. Results show that latent traits estimated from the personality test based on the rating scale method are severely biased, and that faking effect can be pragmatically ignored in the personality test developed based on the pairwise preference method.     

A multitrait-multimethod model with minimal assumptions

A new model of confirmatory factor analysis (CFA) for multitrait-multimethod (MTMM) data sets is presented. It is shown that this model can be defined by only three assumptions in the framework of classical psychometric test theory (CTT). All other properties of the model, particularly the uncorrelated-ness of the trait with the method factors are logical consequences of the definition of the model. In the model proposed there are as many trait factors as different traits considered, but the number of method factors is one fewer than the number of methods included in an MTMM study. The covariance structure implied by this model is derived, and it is shown that this model is identified even under conditions under which other CFA-MTMM models are not. The model is illustrated by two empirical applications. Furthermore, its advantages and limitations are discussed with respect to previously developed CFA models for MTMM data.     

心理学视野下的社会变迁研究：研究设计与分析方法

近年来,社会变迁导致的文化、心理与行为变化及其相互建构已经成为心理学的研究热点。社会变迁研究通常涉及和变迁相关的时间效应、时代效应以及和变迁无关的年龄效应。在考察变迁趋势及其影响因素时,常用的研究设计有跨时间比较、跨代际比较、跨地区比较(历史重构)等方法;常用的分析方法有传统的相关和回归分析和现代的时间序列分析(比如交叉滞后相关和格兰杰因果检验)等。由于每一种设计都有其优点和不足,在具体研究中,研究者要根据研究的问题和可操作性选择合适的方法;如果可能,最好同时采用多种方法,以寻求基于不同方法的聚合证据。     

Ordinal consistency and ordinal true scores

This paper argues that test data are ordinal, that latent trait scores are only determined ordinally, and that test data are used largely for ordinal purposes. Therefore it is desirable to develop a test theory based only on ordinal assumptions. A set of ordinal assumptions is presented, including an ordinal version of local independence. From these assumptions it is first shown that the gamma-correlation between two tests is the product of their gamma-correlations with the true latent order. The theory is generalized to allow for heterogeneous tests by defining a weighted average local independence. The tau-correlations between total score and the latent order can be found in both homogeneous and heterogeneous cases, and a system of differential item weighting to maximize the tau-correlation between weighted items and the latent order is provided. Thus a purely ordinal test theory seems possible.Part of this work was done while the author was a Visiting Fellow at Macquarrie University. The paper has benefitted from discussions with Professors Thomas J. Reynolds and Roderick P. McDonald and from the comments of several anonymous reviewers.     

An empirical Q-matrix validation method for the sequential generalized DINA model

As a core component of most cognitive diagnosis models, the Q-matrix, or item and attribute association matrix, is typically developed by domain experts, and tends to be subjective. It is critical to validate the Q-matrix empirically because a misspecified Q-matrix could result in erroneous attribute estimation. Most existing Q-matrix validation procedures are developed for dichotomous responses. However, in this paper, we propose a method to empirically detect and correct the misspecifications in the Q-matrix for graded response data based on the sequential generalized deterministic inputs, noisy ‘and’ gate (G-DINA) model. The proposed Q-matrix validation procedure is implemented in a stepwise manner based on the Wald test and an effect size measure. The feasibility of the proposed method is examined using simulation studies. Also, a set of data from the Trends in International Mathematics and Science Study (TIMSS) 2011 mathematics assessment is analysed for illustration.