Developing Pairwise Preference-Based Personality Test and Experimental Investigation of Its Resistance to Faking Effect by Item Response Model

Recent years have shown increased awareness of the importance of personality tests in educational, clinical, and occupational settings, and developing faking-resistant personality tests is a very pragmatic issue for achieving more precise measurement. Inspired by Stark (2002) and Stark, Chernyshenko, and Drasgow (2005), we develop a pairwise preference-based personality test that aims to measure multidimensional personality traits using a large-scale statement bank. An experiment compares the resistance of the developed personality test to faking with that of rating scale-based personality tests in the item response theory model framework. Results show that latent traits estimated from the personality test based on the rating scale method are severely biased, and that faking effect can be pragmatically ignored in the personality test developed based on the pairwise preference method.     

Optimal designs for the generalized partial credit model

Analysing ordinal data is becoming increasingly important in psychology, especially in the context of item response theory. The generalized partial credit model (GPCM) is probably the most widely used ordinal model and has found application in many large-scale educational assessment studies such as PISA. In the present paper, optimal test designs are investigated for estimating persons’ abilities with the GPCM for calibrated tests when item parameters are known from previous studies. We find that local optimality may be achieved by assigning non-zero probability only to the first and last categories independently of a person's ability. That is, when using such a design, the GPCM reduces to the dichotomous two-parameter logistic (2PL) model. Since locally optimal designs require the true ability to be known, we consider alternative Bayesian design criteria using weight distributions over the ability parameter space. For symmetric weight distributions, we derive necessary conditions for the optimal one-point design of two response categories to be Bayes optimal. Furthermore, we discuss examples of common symmetric weight distributions and investigate under what circumstances the necessary conditions are also sufficient. Since the 2PL model is a special case of the GPCM, all of these results hold for the 2PL model as well.     

Using multidimensional item response theory to evaluate how response styles impact measurement

Multidimensional item response theory (MIRT) models for response style (e.g., Bolt, Lu, & Kim, 2014, Psychological Methods, 19, 528; Falk & Cai, 2016, Psychological Methods, 21, 328) provide flexibility in accommodating various response styles, but often present difficulty in isolating the effects of response style(s) from the intended substantive trait(s). In the presence of such measurement limitations, we consider several ways in which MIRT models are nevertheless useful in lending insight into how response styles may interfere with measurement for a given test instrument. Such a study can also inform whether alternative design considerations (e.g., anchoring vignettes, self-report items of heterogeneous content) that seek to control for response style effects may be helpful. We illustrate several aspects of an MIRT approach using real and simulated analyses.     

Rater Model Using Signal Detection Theory for Latent Differential Rater Functioning

Differential rater functioning (DRF) occurs when raters show evidence of exercising differential severity or leniency when scoring examinees within different subgroups. Previous studies of DRF have examined rater bias using manifest variables (e.g., use of covariates) to determine the subgroups. These manifest variables include gender and the ethnicity of the examinee. For example, a rater may score males more severely. Ideally, each rater’s severity should be invariant across subgroups. This study examines DRF in the context of latent subgroups that classify possible sources of DRF based on raters’ scoring behavior rather than manifest factors. An extension of the latent class signal detection theory (LC-SDT) model for identifying DRF is proposed and examined using real-world data and simulations. Results from real-world data show that the signal detection approach leads to an effective method to identify latent DRF. Simulations with varying sample sizes and conditions of rater precision were shown to recover parameters at an adequate level, supporting its use to identify latent DRF in large-scale data. These findings suggest that the DRF extension of the LC-SDT can be a useful model to examine characteristics of raters and add information that can aid rater training.     

Item response theory analyses of the Delis-Kaplan Executive Function System card sorting subtest

In the current study, we examined the dimensionality of the 16-item Card Sorting subtest of the Delis-Kaplan Executive Functioning System assessment in a sample of 264 native English-speaking children between the ages of 9 and 15 years. We also tested for measurement invariance for these items across age and gender groups using item response theory (IRT). Results of the exploratory factor analysis indicated that a two-factor model that distinguished between verbal and perceptual items provided the best fit to the data. Although the items demonstrated measurement invariance across age groups, measurement invariance was violated for gender groups, with two items demonstrating differential item functioning for males and females. Multigroup analysis using all 16 items indicated that the items were more effective for individuals whose IRT scale scores were relatively high. A single-group explanatory IRT model using 14 non-differential item functioning items showed that for perceptual ability, females scored higher than males and that scores increased with age for both males and females; for verbal ability, the observed increase in scores across age differed for males and females. The implications of these findings are discussed.     

Same face,same place,different memory: manner of presentation modulates the associative deficit in older adults

One of the more severe and consequential memory impairments experienced by older adults is the loss of the ability to form and remember associations. Although the associative deficit is often assumed to be unitary, memory episodes may contain different types of associations (e.g., item–item, item–context). Research in younger adults suggests that these different association types may involve different neural mechanisms. This raises the possibility that different association types are not equally affected by aging. In order to investigate this, the current study directly compared memory across item–item and item–context associations in younger and older adults by manipulating the manner of presentation of the associations. Results indicate that the associative deficit in aging is not uniform and that aging has a greater impact on item–context compared to item–item associations. The results have implications for theories of associative memory, age-related cognitive decline, and the functional organization of the medial temporal lobe in aging.     

Item Parameter Drift in Context Questionnaires from International Large-Scale Assessments

The purpose of the current study was to examine the impact of item parameter drift (IPD) occurring in context questionnaires from an international large-scale assessment and determine the most appropriate way to address IPD. Focusing on the context of psychometric and educational research where scores from context questionnaires composed of polytomous items were employed for the classification of examinees, the current research investigated the impacts of IPD on the estimation of questionnaire scores and classification accuracy with five manipulated factors: the length of a questionnaire, the proportion of items exhibiting IPD, the direction and magnitude of IPD, and three decisions about IPD. The results indicated that the impact of IPD occurring in a short context questionnaire on the accuracy of score estimation and classification of examinees was substantial. The accuracy in classification considerably decreased especially at the lowest and highest categories of a trait. Unlike the recommendation from literature in educational testing, the current study demonstrated that keeping items exhibiting IPD and removing them only for transformation were appropriate when IPD occurred in relatively short context questionnaires. Using 2011 TIMSS data from Iran, an applied example demonstrated the application of provided guidance in making appropriate decisions about IPD.     

Reliability and validity of the Basic Depression Questionnaire

Background/ObjectiveDepressive disorders have a high prevalence around the world. They present a great comorbidity with other disorders like anxiety, thereby making a differential diagnosis very difficult. The Basic Depression Questionnaire was designed to palliate this issue by isolating specific depression symptoms. Our aim is to study the reliability, factorial structure, and differential item functioning of this questionnaire. Method: The sample consisted of 1,397 adults without psychological problems (M_age = 29.76, SD = 11.25, 64.78% women) who completed the CBD. Results: We observed that none of the items presented differential functioning. A monofactorial structure was established. In this model a good fit was obtained by confirmatory factor analysis and a strict invariance by sex. The ordinal alpha was used to check the reliability and it fetched an index of .95. Conclusions: The Basic Depression Questionnaire has adequate psychometric properties. The absence of differential item functioning and the invariance by sex are guarantees of an adequate use to diagnose depression for men and women. So, its clinical use can help to differentiate between the specific diagnoses for depressive disorders and anxiety disorders.     

心理与教育测验中反应时模型应用的优势与举例

基于计算机形式的测验使得收集作答反应时信息成为可能,这些信息的有效利用对心理与教育测验的理论研究和实际应用产生了重大影响。首先,归纳并总结了测验中使用反应时信息的五大优势。其次,分别介绍了4种不同取向下较典型的反应时模型与模型特征,并分别进行评价。再次,较系统地梳理了反应时模型在实践中的应用,使读者了解反应时信息在测验中所发挥的作用。最后,探讨了未来将反应时应用于心理与教育测量领域的几个研究方向。