An Illustration of Diagnostic Classification Modeling in Student Learning Outcomes Assessment

The assessment of higher-education student learning outcomes is an important component in understanding the strengths and weaknesses of academic and general education programs. This study illustrates the application of diagnostic classification models, a burgeoning set of statistical models, in assessing student learning outcomes. To facilitate understanding and future applications of diagnostic modeling, the log-linear cognitive diagnosis model used in this study is presented in a didactic manner. The model is applied in a context where undergraduate students were assessed along four learning outcomes related to psychosocial research across two time points. Results focus on implications and methods to aid stakeholders’ interpretation of the analyses. Contrasts to traditional measurement models and potential future applications are also discussed.     

Using Automated Essay Scores as an Anchor When Equating Constructed Response Writing Tests

Assessments consisting of only a few extended constructed response items (essays) are not typically equated using anchor test designs as there are typically too few essay prompts in each form to allow for meaningful equating. This article explores the idea that output from an automated scoring program designed to measure writing fluency (a common objective of many writing prompts) can be used in place of a more traditional anchor. The linear-logistic equating method used in this article is a variant of the Tucker linear equating method appropriate for the limited score range typical of essays. The procedure is applied to historical data. Although the procedure only results in small improvements over identity equating (not equating prompts), it does produce a viable alternative, and a mechanism for checking that the identity equating is appropriate. This may be particularly useful for measuring rater drift or equating mixed format tests.     

Guidelines Versus Practices in Cross-Lingual Assessment: A Disconcerting Disconnect

The International Test Commission's Guidelines for Translating and Adapting Tests (2010) provide important guidance on developing and evaluating tests for use across languages. These guidelines are widely applauded, but the degree to which they are followed in practice is unknown. The objective of this study was to perform a systematic review of published articles to determine whether the methodologies related to test adaptation and evaluation of cross-lingual measurement bias have improved since the initial publication of the Guidelines. Results demonstrated that although reporting reliability estimates and statistically analyzing construct equivalence improved since publication of the Guidelines, the majority of test development and validation practices in the published literature have not embraced the recommendations put forth by the International Test Commission. Overall, these findings call into question many of the inferences derived from cross-linguistic comparisons and clearly signal a need for better dissemination mechanisms of the Guidelines.     

Measuring slips and lapses when they occur – Ambulatory assessment in application to cognitive failures

Cognitive failures are lapses in attention, cognition, and actions that everybody experiences in everyday life. Self-reports are mainly used for assessment but those instruments are memory-biased and more related to personality aspects than to actual behavior. Ambulatory assessment is already used for capturing emotions or addictive behavior, but not yet for cognitive failures. The newly developed Questionnaire for Cognitive Failures in Everyday Life (KFA) was applied via mobile phones (electronic KFA) wherein an acoustic signal asked participants (N = 91, 60–76 years) 4 times daily to answer 13 questions for 1 week. The new instrument showed satisfying reliability and was compared with a self-report method (Cognitive Failures Questionnaire; Broadbent, Cooper, Fitzgerald, and Parkes, 1982) in terms of correlations with cognitive abilities (working memory capacity, short-term memory, switching ability, and reasoning), personality traits, and demographical aspects. Although further validation is needed, first results are promising and eKFA enriches cognitive failures research.     

The testing effect with authentic educational materials: A cautionary note

Despite considerable evidence that testing benefits subsequent retrieval of information, it remains uncertain whether this effect extends to topically related information with authentic classroom materials. In the current study we first profile the way in which quizzing is used in the classroom through a survey of introductory psychology instructors. The survey results indicate that, instructors frequently use related but different questions on quizzes and tests unlike many laboratory experiments that use identical questions. In two subsequent experiments, participants studied information from a college biology textbook, were quizzed twice, and given a final test. The items on the final test were either identical to or were related but different than the quiz items. Experiment 1 showed that testing produced the typical robust testing effect for repeated items, but there was no significant effect of testing for topically related items. In Experiment 2, participants could use their quizzes to guide restudy, and there was still no positive effect of testing for topically related information.     

The development of the Short Muslim Practice and Belief Scale

The majority of the measures of religious practice and belief found in the literature are for persons of the Christian faith; such measures for Muslims are scarce. As examining the role of religion in relation to alcohol consumption is difficult because of the lack of appropriate measures, in the current study, a brief measure of practice and belief for persons of the Islamic faith was developed. Arab Muslims living the United Arab Emirates and Oman (n = 611) and Asian Muslims living in Malaysia and Indonesia (n = 303) were surveyed. The Short Muslim Practice and Belief scale (Short-MPBS) was subjected to exploratory factor analysis and confirmatory factor analysis. These analyses indicated a 9-item measure with a two-factor structure was a good fit of the data. Internal consistency (α = 0.83) and validity were good. Participants who scored higher on the measure were likely to be lifelong alcohol abstainers.     

Exploring the Application of Actuarial Criminology in Southern Africa

In response to the declared intent of the White Paper on Corrections (Department of Corrections, 2005) to transform prisons into needs-based centers of reform, the Self-appraisal Questionnaire (SAQ) was used to explore its potential as a reliable and valid prediction scale for recidivism in the diverse cultural context of Southern Africa. A total of 269 incarcerated males agreed to be anonymously assessed. Furthermore, 38 offenders with a history of maladjustment and disciplinary problems were identified for comparison. An alpha score of.90 as well as moderate to high or high to excellent coefficient alphas for six out of the seven subscales confirm the SAQ to be a reliable assessment instrument within the Southern African context. All correlation coefficients (criminal tendencies, antisocial personality problems, conduct problems, criminal history, alcohol and drug abuse, antisocial associates and anger) also measured significantly at the 0.01 level while the scores of the 38 isolated cases suggested higher scores in relation to conduct disorder, antisocial personality trends, anger and consequently criminal history.     

Perceptions of the utility of personality assessment for personnel selection in the South African context: An exploratory study

This study explored South African industrial psychologists’ perceptions of the utility of personality assessment for personnel selection. Eleven industrial psychologists (2 male, 9 female) from the Johannesburg area, South Africa were the participants. They participated in semi-structured interviews exploring their understanding of the place of personality assessment in personnel selection within organisational settings in South Africa and the common practices employed. Thematic analysis was used to analyse the data. Four dominant themes emerged from the data: person-environment fit; specific uses of personality tests; commonly used tests and validity and ethical concerns. Based on the findings, personality assessment use in South African industry settings appears multi-layered and consistently evolving.     

Four fatal flaws of brain injury assessment in the cross-cultural context

This analytical review serves to highlight pitfalls clinicians in multicultural contexts may encounter when using normed ability tests for the assessment of cognitive dysfunction due to brain injury. Four sources of erroneous practice are: (i) Failure to recognize within-race group heterogeneity on cognitive test performance in association with factors such as quality of education; (ii) Lack of rigorous attention being paid to the exact mode of test administration used in a derived normative base to be used for assessment purposes; (iii) Estimating an expected level of performance for an individual on diverse cognitive tests based on an IQ score; (iv) Adopting a thoroughly rule-bound actuarial approach to test interpretation, over a contextually based clinical approach. Even with locally derived norms, the validity of actuarially based interpretive conclusions in the absence of a conceptually coherent case analysis is not supported.