Item Characteristic Curve Estimation of Signal Detection Theory-Based Personality Data: A Two-Stage Approach to Item Response Modeling

Signal Detection Theory (SDT; MacMillan & Creelman, 1991) is a method of data collection that has been used for several years, which describes the decision-making strategies of individuals. However, its use has been largely restricted to experiments involving sensation and perception. The Overclaiming Questionnaire (OCQ; Paulhus & Bruce, 1990) is a scale that has been developed to measure intellectual ability and personality, using SDT as a guideline. Although the scale has been successful in measuring human characteristics such as narcissism and intelligence, it is still unclear how to measure the characteristics of the various stimuli used (e.g., item difficulty, item discrimination, etc.). In some ways, this is a direct consequence of the general lack of research involved in item parameter estimation in the field of SDT. Using the OCQ, this article presents a graphical and nonparametric form of item response modeling to address this issue. In many ways, the approach is influenced by and structured around item response theory (IRT; Hambleton, Swaminathan, & Rogers, 1991). The general features of both SDT and IRT are described. Results suggest that this method is indeed a reasonable approach to describing item functioning, and there are several advantages to using this method over traditional IRT methods. Furthermore, SDT appears to be a fruitful approach to assessing intelligence, ability, and other psychological constructs, with advantages over traditional approaches. Overall, the results provide interesting implications for item selection and test development in several scientific and academic fields.     

International Semiotics: Item Difficulty and the Complexity of Science Item Illustrations in the PISA-2009 International Test Comparison

We examined multimodality (the representation of information in multiple semiotic modes) in the context of international test comparisons. Using Program of International Student Assessment (PISA)-2009 data, we examined the correlation of the difficulty of science items and the complexity of their illustrations. We observed statistically significant mean correlation coefficient differences favoring higher-ranking over lower-ranking jurisdictions and non-Western over Western jurisdictions. We examined this correlation as a function of the variety of forms of representation of information used in illustrations. For the highest-ranking jurisdictions, the correlation tended to be positive and its magnitude tended to increase with the number of forms of representation. For the lowest-ranking jurisdictions, the correlation tended to be negative and its magnitude tended to increase with the number of forms of representation. Increased illustration complexity appears to be an affordance in making sense of science items for students from the highest-ranking jurisdictions and a challenge for students from the lowest-ranking jurisdictions. Our findings support the notion that integrating information provided in multiple semiotic modes is critical to science achievement and underscore the importance for science standards and international assessment frameworks to address this skill as intrinsic to (rather than an aspect of) science proficiency.     

Group Learning Curves: The Effects of Turnover and Task Complexity on Group Performance1

This study examined the effects of turnover and task complexity on group performance. Two hundred and forty subjects arranged into three-person groups performed a production-type task for six experimental periods. The design was a 2 (Turnover vs. No Turnover) × 2 (Simple vs. Complex Task) × 2 (Male vs. Female) × 6 (Periods) factorial with repeated measures on the last factor. The analysis revealed that group performance improved markedly as groups gained experience with the task. Groups which did not experience turnover (closed groups) produced significantly more products than did groups which experienced turnover (open groups). The superior performance of closed over open groups was amplified over periods. Groups produced more of the simple than of the complex product, and this difference was also amplified over periods. The gap in the performance of closed versus open groups increased over periods, and the increase in the gap was greater for the simple than for the complex task. The lesser impact of turnover on the complex task is consistent with an innovation hypothesis, according to which increases in the production of complex tasks are due more to innovation than to repetition. Supportive analyses of innovation data are reported.     

Perceptual Bias in Speech Error Data Collection: Insights from Spanish Speech Errors

This paper studies the reliability and validity of naturalistic speech errors as a tool for language production research. Possible biases when collecting naturalistic speech errors are identified and specific predictions derived. These patterns are then contrasted with published reports from Germanic languages (English, German and Dutch) and one Romance language (Spanish). Unlike findings in the Germanic languages, Spanish speech errors show many patterns which run contrary to those expected from bias: (1) more phonological errors occur between words than within word; (2) word-initial consonants are less likely to participate in errors than word-medial consonants, (3) errors are equally likely in stressed and in unstressed syllables, (4) perseverations are more frequent than anticipations, and (5) there is no trace of a lexical bias. We present a new corpus of Spanish speech errors collected by many theoretically naïve observers (whereas the only corpus available so far was collected by two highly trained theoretically informed observers), give a general overview of it, and use it to replicate previous reports. In spite of the different susceptibility of these methods to bias, results were remarkably similar in both corpora and again contrary to predictions from bias. As a result, collecting speech errors “in the wild” seems to be free of bias to a reasonable extent even when using a multiple-collector method. The observed contrasting patterns between Spanish and Germanic languages arise as true cross-linguistic differences.     

Adapting Group Therapy to Address Real World Problems: Insights from Groups Offered in the Bahamas

This article presents a cultural adaptation of a group therapeutic approach that is being offered in the Bahamas. “The Family: People Helping People” project was designed as an intervention to improve socialization in New Providence, the Bahamian capital and its most heavily populated city. “The Family” group model offers support and training to improve communication in relationships and to encourage constructive emotional expression. This article will provide an overview of “The Family,” address key elements of this approach that are culturally tailored, and offer clinical examples and note implications for group therapy training. This cultural adaptation offers helpful insights for addressing community problems, such as violence and societal fragmentation, and may inform the development of community-based group interventions in other settings.     

Background Data and Autobiographical Memory: Effects of Item Types and Task Characteristics

Background data, or life history measures, ask people to recall and report prior behavior and experiences. Although these measures have been found to be effective predictors of performance, little is known about the characteristics of items contributing to accurate reporting and good recall. Accordingly, 153 undergraduates were asked to work on a complex, computer-based ‘in-basket’ task. In working through this task, objective indices of their activities were obtained. After completing this task, subjects were asked to answer a set of background data items describing their behavior and experiences during task performance. Additionally, they were asked to return one week later and, again, answer the background data questions. Indices of accuracy and recall consistency were obtained for various item types. In accordance with the literature of autobiographical memory, it was found that people could consistency recall and accurately report general summaries of activities and their reactions to the task. More specific actions or behaviors that were not linked to goals were not accurately and consistently recalled. The implications of these findings for the development and application of background data measures are discussed.     

Item Response Modeling of Paired Comparison and Ranking Data

The comparative format used in ranking and paired comparisons tasks can significantly reduce the impact of uniform response biases typically associated with rating scales. Thurstone's (1927, 1931) model provides a powerful framework for modeling comparative data such as paired comparisons and rankings. Although Thurstonian models are generally presented as scaling models, that is, stimuli-centered models, they can also be used as person-centered models. In this article, we discuss how Thurstone's model for comparative data can be formulated as item response theory models so that respondents' scores on underlying dimensions can be estimated. Item parameters and latent trait scores can be readily estimated using a widely used statistical modeling program. Simulation studies show that item characteristic curves can be accurately estimated with as few as 200 observations and that latent trait scores can be recovered to a high precision. Empirical examples are given to illustrate how the model may be applied in practice and to recommend guidelines for designing ranking and paired comparisons tasks in the future.     

Ethics in Publishing: Complexity Science and Human Factors Offer Insights to Develop a Just Culture

While ethics in publishing has been increasingly debated, there seems to be a lack of a theoretical framework for making sense of existing rules of behavior as well as for designing, managing and enforcing such rules. This letter argues that systems-oriented disciplines, such as complexity science and human factors, offer insights into new ways of dealing with ethics in publishing. Some examples of insights are presented. Also, a call is made for empirical studies that unveil the context and details of both retracted papers and the process of writing and publishing academic papers. This is expected to shed light on the complexity of the publication system as well as to support the development of a just culture, in which all participants are accountable.     

Socialization: Insights from Social Cognition

Socialization is the process by which individuals are assisted to become members of their social groups. Findings from social cognition and cross-cultural psychology offer two major insights into the socialization process. First, basic social cognitive principles imply that the immediate environment functions as a socialization agent by activating and inhibiting knowledge structures and thereby shaping cognition and behavior. Second, because the immediate environment factors into cognition and behavior, socialization efforts should involve the modification of the environment for optimal effect. We discuss various examples of socialization through the configuration of the immediate environment, such as rituals and use of physical artifacts. Our review links basic social cognitive mechanisms to socialization processes, which are customarily treated at higher levels of analysis.     

Parent Ratings of the Oppositional Defiant Disorder Symptoms: Item Response Theory Analyses of Cross-National and Cross-Racial Invariance

Item response theory (IRT) based differential item functioning (DIF) was used to examine the construct and normative invariance of the DSM-IV oppositional defiant disorder (ODD) symptoms for ratings across Malaysian and Australian children, and Malaysian Malay and Malaysian Chinese children. To accomplish these goals, parents completed the Disruptive Behavior Rating Scale, which includes the eight DSM-IV ODD symptoms. Although the comparisons involving Malaysian and Australian children indicated DIF for five symptoms, only the symptom for “touchy” showed notable DIF. This was also the only symptom that showed DIF for the comparisons involving Malay and Chinese children. There were also minimal differences in the latent mean scores across Australian and Malaysian children and also Malay and Chinese children. These results indicate good support for the construct and normative invariance of the ODD symptoms for the samples compared.     

Modeling and Testing Differential Item Functioning in Unidimensional Binary Item Response Models with a Single Continuous Covariate: A Functional Data Analysis Approach

Differential item functioning (DIF), referring to between-group variation in item characteristics above and beyond the group-level disparity in the latent variable of interest, has long been regarded as an important item-level diagnostic. The presence of DIF impairs the fit of the single-group item response model being used, and calls for either model modification or item deletion in practice, depending on the mode of analysis. Methods for testing DIF with continuous covariates, rather than categorical grouping variables, have been developed; however, they are restrictive in parametric forms, and thus are not sufficiently flexible to describe complex interaction among latent variables and covariates. In the current study, we formulate the probability of endorsing each test item as a general bivariate function of a unidimensional latent trait and a single covariate, which is then approximated by a two-dimensional smoothing spline. The accuracy and precision of the proposed procedure is evaluated via Monte Carlo simulations. If anchor items are available, we proposed an extended model that simultaneously estimates item characteristic functions (ICFs) for anchor items, ICFs conditional on the covariate for non-anchor items, and the latent variable density conditional on the covariate—all using regression splines. A permutation DIF test is developed, and its performance is compared to the conventional parametric approach in a simulation study. We also illustrate the proposed semiparametric DIF testing procedure with an empirical example.     

Modeling Intensive Polytomous Time-Series Eye-Tracking Data: A Dynamic Tree-Based Item Response Model

This paper presents a dynamic tree-based item response (IRTree) model as a novel extension of the autoregressive generalized linear mixed effect model (dynamic GLMM). We illustrate the unique utility of the dynamic IRTree model in its capability of modeling differentiated processes indicated by intensive polytomous time-series eye-tracking data. The dynamic IRTree was inspired by but is distinct from the dynamic GLMM which was previously presented by Cho, Brown-Schmidt, and Lee (Psychometrika 83(3):751–771, 2018). Unlike the dynamic IRTree, the dynamic GLMM is suitable for modeling intensive binary time-series eye-tracking data to identify visual attention to a single interest area over all other possible fixation locations. The dynamic IRTree model is a general modeling framework which can be used to model change processes (trend and autocorrelation) and which allows for decomposing data into various sources of heterogeneity. The dynamic IRTree model was illustrated using an experimental study that employed the visual-world eye-tracking technique. The results of a simulation study showed that parameter recovery of the model was satisfactory and that ignoring trend and autoregressive effects resulted in biased estimates of experimental condition effects in the same conditions found in the empirical study.

     

Additive Multilevel Item Structure Models with Random Residuals: Item Modeling for Explanation and Item Generation

An additive multilevel item structure (AMIS) model with random residuals is proposed. The model includes multilevel latent regressions of item discrimination and item difficulty parameters on covariates at both item and item category levels with random residuals at both levels. The AMIS model is useful for explanation purposes and also for prediction purposes as in an item generation context. The parameters can be estimated with an alternating imputation posterior algorithm that makes use of adaptive quadrature, and the performance of this algorithm is evaluated in a simulation study.     

Evaluating Word-Reading Models at the Item Level: Matching the Grain of Theory and Data

Spieler and Balota (1997) showed that connectionist models of reading account for relatively little item-specific variance. In assessing this finding, it is important to recognize two factors that limit how much variance such models can possibly explain. First, item means are affected by several factors that are not addressed in existing models, including processes involved in recognizing letters and producing articulatory output. These limitations point to important areas for future research but have little bearing on existing theoretical claims. Second, the item data include a substantial amount of error variance that would be inappropriate to model. Issues concerning comparisons between simulation data and human performance are discussed with an emphasis on the importance of evaluating models at a level of specificity ("grain") appropriate to the theoretical issues being addressed.     

A Doubly Latent Space Joint Model for Local Item and Person Dependence in the Analysis of Item Response Data

Item response theory (IRT) is one of the most widely utilized tools for item response analysis; however, local item and person independence, which is a critical assumption for IRT, is often violated in real testing situations. In this article, we propose a new type of analytical approach for item response data that does not require standard local independence assumptions. By adapting a latent space joint modeling approach, our proposed model can estimate pairwise distances to represent the item and person dependence structures, from which item and person clusters in latent spaces can be identified. We provide an empirical data analysis to illustrate an application of the proposed method. A simulation study is provided to evaluate the performance of the proposed method in comparison with existing methods.

     

A Procedure for Dimensionality Analyses of Response Data from Various Test Designs

In some popular test designs (including computerized adaptive testing and multistage testing), many item pairs are not administered to any test takers, which may result in some complications during dimensionality analyses. In this paper, a modified DETECT index is proposed in order to perform dimensionality analyses for response data from such designs. It is proven in this paper that under certain conditions, the modified DETECT can successfully find the dimensionality-based partition of items. Furthermore, the modified DETECT index is decomposed into two parts, which can serve as indices of the reliability of results from the DETECT procedure when response data are judged to be multidimensional. A simulation study shows that the modified DETECT can successfully recover the dimensional structure of response data under reasonable specifications. Finally, the modified DETECT procedure is applied to real response data from two-stage tests to demonstrate how to utilize these indices and interpret their values in dimensionality analyses.     

Cross-modal attentional entrainment: Insights from magicians

Recently, performance magic has become a source of insight into the processes underlying awareness. Magicians have highlighted a set of variables that can create moments of visual attentional suppression, which they call “off-beats.” One of these variables is akin to the phenomenon psychologists know as attentional entrainment. The current experiments, inspired by performance magic, explore the extent to which entrainment can occur across sensory modalities. Across two experiments using a difficult dot probe detection task, we find that the mere presence of an auditory rhythm can bias when visual attention is deployed, speeding responses to stimuli appearing in phase with the rhythm. However, the extent of this cross-modal influence is moderated by factors such as the speed of the entrainers and whether their frequency is increasing or decreasing. In Experiment 1, entrainment occurred for rhythms presented at .67 Hz, but not at 1.5 Hz. In Experiment 2, entrainment only occurred for rhythms that were slowing from 1.5 Hz to .67 Hz, not speeding. The results of these experiments challenge current models of temporal attention.