Modeling item responses when different subjects employ different solution strategies

A model is presented for item responses when different subjects employ different strategies, but only responses, not choice of strategy, can be observed. Using substantive theory to differentiate the likelihoods of response vectors under a fixed set of strategies, we model response probabilities in terms of item parameters for each strategy, proportions of subjects employing each strategy, and distributions of subject proficiency within strategies. The probabilities that an individual subject employed the various strategies can then be obtained, along with a conditional estimate of proficiency under each. A conceptual example discusses response strategies for spatial rotation tasks, and a numerican example resolves a population of subjects into subpopulations of valid responders and random guessers.The first author's work was supported by Contract No. N00014-85-K-0683, project designation NR 150-539, from the Cognitive Science Program, Cognitive and Neural Sciences Division, Office of Naval Research. We are grateful to Murray Aitkin, Isaac Bejar, Neil Dorans, Frederiksen, and Marklyn Wingersky for their comments and suggestions, and to Alison Gooding, Maxine Kingston, Donna Lembeck, Joling Liang, and Kentaro Yamamoto for their assistance with Example 2.     

Higher-order latent trait models for cognitive diagnosis

Higher-order latent traits are proposed for specifying the joint distribution of binary attributes in models for cognitive diagnosis. This approach results in a parsimonious model for the joint distribution of a high-dimensional attribute vector that is natural in many situations when specific cognitive information is sought but a less informative item response model would be a reasonable alternative. This approach stems from viewing the attributes as the specific knowledge required for examination performance, and modeling these attributes as arising from a broadly-defined latent trait resembling theϑ of item response models. In this way a relatively simple model for the joint distribution of the attributes results, which is based on a plausible model for the relationship between general aptitude and specific knowledge. Markov chain Monte Carlo algorithms for parameter estimation are given for selected response distributions, and simulation results are presented to examine the performance of the algorithm as well as the sensitivity of classification to model misspecification. An analysis of fraction subtraction data is provided as an example. This research was funded by National Institute of Health grant R01 CA81068. We would like to thank William Stout and Sarah Hartz for many useful discussions, three anonymous reviewers for helpful comments and suggestions, and Kikumi Tatsuoka and Curtis Tatsuoka for generously sharing data.     

Characterizing the manifest probabilities of latent trait models

The problem of characterizing the manifest probabilities of a latent trait model is considered. The item characteristic curve is transformed to the item passing-odds curve and a corresponding transformation is made on the distribution of ability. This results in a useful expression for the manifest probabilities of any latent trait model. The result is then applied to give a characterization of the Rasch model as a log-linear model for a 2 ^J -contingency table. Partial results are also obtained for other models. The question of the identifiability of “guessing” parameters is also discussed. The research reported here is collaborative in every respect and the order of authorship is alphabetical. Dr. Cressie was a Visiting Research Scientist at ETS during the Fall of 1980. His current address is: School of Mathematical Sciences, The Flinders University of South Australia, Bedford Park SA, 5042, AUSTRALIA. The preparation of this paper was supported, in part, by the Program Statistics Research Project in the Research Statistics Group at ETS.     

The role of collateral information about examinees in item parameter estimation

Standard procedures for estimating item parameters in item response theory (IRT) ignore collateral information that may be available about examinees, such as their standing on demographic and educational variables. This paper describes circumstances under which collateral information about examineesmay be used to make inferences about item parameters more precise, and circumstances under which itmust be used to obtain correct inferences.This work was supported by Contract No. N00014-85-K-0683, project designation NR 150-539, from the Cognitive Science Program, Cognitive and Neural Sciences Division, Office of Naval Research. Reproduction in whole or in part is permitted for any purpose of the United States Government. We are indebted to Tim Davey, Eugene Johnson, and three anonymous referees for their comments on earlier versions of the paper.     

The effect of uncertainty of item parameter estimation on ability estimates

The conventional method of measuring ability, which is based on items with assumed true parameter values obtained from a pretest, is compared to a Bayesian method that deals with the uncertainties of such items. Computational expressions are presented for approximating the posterior mean and variance of ability under the three-parameter logistic (3PL) model. A 1987 American College Testing Program (ACT) math test is used to demonstrate that the standard practice of using maximum likelihood or empirical Bayes techniques may seriously underestimate the uncertainty in estimated ability when the pretest sample is only moderately large.This work was partially supported under contract No. N00014-85-K-0113, NR150-535, from the Cognitive Science Program, Office of Naval Research. The authors wish to thank Mark D. Reckase for providing the ACT data used in the illustration and two referees, Asociate Editor and Editor for helpful suggestions.     

Specifying optimum examinees for item parameter estimation in item response theory

Information functions are used to find the optimum ability levels and maximum contributions to information for estimating item parameters in three commonly used logistic item response models. For the three and two parameter logistic models, examinees who contribute maximally to the estimation of item difficulty contribute little to the estimation of item discrimination. This suggests that in applications that depend heavily upon the veracity of individual item parameter estimates (e.g. adaptive testing or text construction), better item calibration results may be obtained (for fixed sample sizes) from examinee calibration samples in which ability is widely dispersed.This work was supported by Contract No. N00014-83-C-0457, project designation NR 150-520, from Cognitive Science Program, Cognitive and Neural Sciences Division, Office of Naval Research and Educational Testing Service through the Program Research Planning Council. Reproduction in whole or in part is permitted for any purpose of the United States Government. The author wishes to acknowledge the invaluable assistance of Maxine B. Kingston in carrying out this study, and to thank Charles Lewis for his many insightful comments on earlier drafts of this paper.     

Evidence and inference in educational assessment

Educational assessment concerns inference about students' knowledge, skills, and accomplishments. Because data are never so comprehensive and unequivocal as to ensure certitude, test theory evolved in part to address questions of weight, coverage, and import of data. The resulting concepts and techniques can be viewed as applications of more general principles for inference in the presence of uncertainty. Issues of evidence and inference in educational assessment are discussed from this perspective.Probability isn't really about numbers; it's about the structure of reasoning.Presidential address to the Psychometric Society, presented June 25, 1994, in Champaign, Illinois.Supported by (1) Contract No. N00014-91-J-4101, R&T 4421573-01, from the Cognitive Science Program, Cognitive and Neural Sciences Division, Office of Naval Research, (2) the National Center for Research on Evaluation, Standards, Student Testing (CRESST), Educational Research and Development Program, cooperative agreement number R117G10027 and CFDA catalog number 84.117G, as administered by the Office of Educational Research and Improvement, U.S. Department of Education, and (3) the Statistical and Psychometric Research Division of Educational Testing Service. I am grateful for comments and suggestions from Henry Braun, Drew Gitomer, Richard Patz, Jonathan Troper, and Howard Wainer.     

Nonparametric Calibration of Item-by-Attribute Matrix in Cognitive Diagnosis

A nonparametric technique based on the Hamming distance is proposed in this research by recognizing that once the attribute vector is known, or correctly estimated with high probability, one can determine the item-by-attribute vectors for new items undergoing calibration. We consider the setting where Q is known for a large item bank, and the q-vectors of additional items are estimated. The method is studied in simulation under a wide variety of conditions, and is illustrated with the Tatsuoka fraction subtraction data. A consistency theorem is developed giving conditions under which nonparametric Q calibration can be expected to work.     

Full-information item bi-factor analysis

A plausibles-factor solution for many types of psychological and educational tests is one that exhibits a general factor ands − 1 group or method related factors. The bi-factor solution results from the constraint that each item has a nonzero loading on the primary dimension and at most one of thes − 1 group factors. This paper derives a bi-factor item-response model for binary response data. In marginal maximum likelihood estimation of item parameters, the bi-factor restriction leads to a major simplification of likelihood equations and (a) permits analysis of models with large numbers of group factors; (b) permits conditional dependence within identified subsets of items; and (c) provides more parsimonious factor solutions than an unrestricted full-information item factor analysis in some cases. Supported by the Cognitive Science Program, Office of Naval Research, Under grant #N00014-89-J-1104. We would like to thank Darrell Bock for several helpful suggestions.     

Does adaptive testing violate local independence?

Item response theory posits local independence, or conditional independence of item responses given item parameters and examinee proficiency parameters. The usual definition of local independence, however, addresses the context of fixed tests, and initially appears to yield incorrect response-pattern probabilities in the context of adaptive testing. The paradox is resolved by introducing additional notation to deal with the item selection mechanism.We are grateful to Charlie Lewis, Ming-Mei Wang, and Pao-Kuei Wu for discussions on this topic, and to the Editor, the reviewers, and Howard Wainer for helpful comments on an earlier version of the paper. The first author's work was supported in part by the National Center for Research on Evaluation, Standards, Student Testing (CRESST), Educational Research and Development Program, cooperative agreement number R117G10027 and CFDA catalog number 84.117G, as administered by the Office of Educational Research and Improvement, U.S. Department of Education.     

Incidental learning of abstract rules for non-dominant word orders

One way in which adult second language learners may acquire a word order that differs from their native language word order is through exposure-based incidental learning, but little is known about that process and what constrains it. The current studies examine whether a non-dominant word order can be learned incidentally, and if so, whether the rule can be generalized to new words not previously seen in the non-dominant order. Two studies examined the incidental learning of rules underlying the order of nouns and verbs in three-word strings. The self-timed reading speeds of native English speakers decreased as a result of practice with a non-dominant rule (words ordered either as “verb noun noun” or “noun noun verb”). The same pattern of results was also found for new words ordered according to the previously encountered rule, suggesting learning generalized beyond the specific instances encountered. A second experiment showed such rule learning could also occur when the nouns were replaced with pronounceable pseudowords. Learning was therefore possible in the absence of any pre-existing relationships between the items. Theoretical and educational implications are discussed. Gwen Schmidt is now with the Center for Cognitive Neuroscience, University of Pennsylvania. Tom Carr has returned to the Department of Psychology, Michigan State University. Ben Clegg’s contribution to this research was supported in part by Army Research Office Grant W9112NF-05-1-0153.     

An IRT Model with a Parameter-Driven Process for Change

An IRT model with a parameter-driven process for change is proposed. Quantitative differences between persons are taken into account by a continuous latent variable, as in common IRT models. In addition, qualitative interindividual differences and autodependencies are accounted for by assuming within-subject variability with respect to the parameters of the IRT model. In particular, the parameters of the IRT model are governed by an unobserved or “hidden'” homogeneous Markov process. The model includes the mixture linear logistic test model (Mislevy & Verhelst, 1990), the mixture Rasch model (Rost, 1990), and the Saltus model (Wilson, 1989) as specific instances. The model is applied to a longitudinal experiment on discontinuity in conservation acquisition (van der Maas, 1993). Frank Rijmen was supported by the Fund for Scientific Research Flanders (FWO), the GOA/2000/02 granted by the Katholieke Universiteit Leuven to Paul De Boeck and Iven Van Mechelen, and the PDM/02/067 granted by the Katholieke Universiteit Leuven to Paul De Boeck.     

Research on cognitive, social and cultural processes of written communication

This article compiles the investigations carried out by a Research Group of the University of Granada, Spain. Its different projects on writing’s cognitive social and cultural processes have been supported by the Spanish Government. This line of research joined together linguistic, psychological, social and cultural contributions to the development of writing from the 1970s. Currently, this line of research develops in collaboration with other European Universities: (a) Interuniversity Centre for Research On Cognitive Processing in Natural and Artificial Systems (ECONA), “La Sapienza” University of Rome (Italy); (b) Anadolu University, (Eskisehir, Turkey); (c) Coimbra University (Portugal); (d) University of Zaragoza (Spain); (e) the Institute of Education of the University of London (United Kingdom). The aforementioned collaboration is materializing into projects like the International Master on Multilingual Writing: Cognitive, Intercultural and Technological Processes of Written Communication () and the International Congress: Writing in the twenty-first Century: Cognition, Multilinguisim and Technologies, held in Granada (). This research line is focussed on the development of strategies in writing development, basic to train twenty-first century societies’ citizens. In these societies, participation in production media, social exchange and the development of multilingual written communication skills through new computer technologies spread multicultural values. In order to fulfil the social exigencies, it is needed to have the collaboration of research groups for designing and applying international research projects. International Master. Multilingual Writing Web: , .     

Robust and powerful nonorthogonal analyses

Numerous types of analyses for factorial designs having unequal cell frequencies have been discussed in the literature. These analyses test either weighted or unweighted marginal means which, in turn, correspond to different model comparisons. Previous research has indicated, however, that these analyses result in biased (liberal or conservative) tests when cell variances are heterogeneous. We show how to obtain a generally robust and powerful analysis with any of the recommended nonorthogonal solutions by adapting a modification of the Welch-James procedure for comparing means when population variances are heterogeneous.This research was supported by a Social Sciences and Humanities Research Council (SSHRC) grant (# 410-92-0430) to the first author, a Manitoba Health Research Council Scholar Award and a grant from Natural Sciences and Engineering Research Council to the second author, and a SSHRC Doctoral Fellowship (# 752-92-1628) to the third author. The authors would like to express their gratitude to Joanne Keselman and three anonymous reviewers for their many helpful substantive comments on earlier drafts of this paper.     

The Dutch Identity: A new tool for the study of item response models

The Dutch Identity is a useful way to reexpress the basic equations of item response models that relate the manifest probabilities to the item response functions (IRFs) and the latent trait distribution. The identity may be exploited in several ways. For example: (a) to suggest how item response models behave for large numbers of items—they are approximate submodels of second-order loglinear models for 2 ^J tables; (b) to suggest new ways to assess the dimensionality of the latent trait—principle components analysis of matrices composed of second-order interactions from loglinear models; (c) to give insight into the structure of latent class models; and (d) to illuminate the problem of identifying the IRFs and the latent trait distribution from sample data.This research was supported in part by contract number N00014-87-K-0730 from the Cognitive Science Program of the Office of Naval Research. I realized the usefulness of the identity in Theorem 1 while lecturing in the Netherlands during October, 1986. Because this was in no small part due to the stimulating psychometric atmosphere there, I call the result the Dutch Identity.     

Specificity of learning through memory retrieval practice: The case of addition and subtraction

The identical elements (IE) model (Rickard, Healy, & Bourne, Learning, Memory, and Cognition 32:734–748, 1994) of fact representation predicts that, in both verbal and numerical domains, performance gains with retrieval practice on multielement items will be specific to the practiced stimulus–response combinations, failing to transfer even to altered stimulus–response mappings of practiced items. In the case of arithmetic, the model predicts no transfer across either complementary operations (e.g., 4 × 7 to 28 / 4) or complementary division or subtraction problems (e.g., 28 / 4 to 28 / 7). Although that model has successfully described transfer effects in the domains of multiplication–division and episodic cued recall, it is challenged by a recent demonstration of positive cross-operation transfer for addition and subtraction (Campbell & Agnew, Psychonomic Bulletin & Review 16:938–944, 2009). We report results of a new addition–subtraction transfer experiment, the design of which closely matched that of a prior multiplication–division experiment that supported the model. The transfer results were consistent with the IE model. A two-component model of memory retrieval practice effects is proposed to account for the discrepant experimental results for addition and subtraction and to guide future work.     

The Perceived Stress Scale: Factor structure and relation to depression symptoms in a psychiatric sample

The present study sought to examine the factor structure and psychometric properties of the Perceived Stress Scale (PSS) when administered to psychiatric patients. We also examined predictive validity of the PSS by assessing the association between the Perceived Stress Scale and the Beck Depression Inventory. A heterogeneous sample of 96 psychiatric patients (48 men, 48 women) completed the Perceived Stress Scale (PSS) and the Beck Depression Inventory. Factor analysis of the PSS established that the scale consisted of two factors. The first factor was comprised primarily of items reflecting adaptational symptoms. In contrast, the second factor consisted of items reflecting coping ability. Both factors had an adequate degree of internal consistency. Finally, a series of regression analyses predicting depression found that both factors accounted for unique variance in depression scores in women, but only the first factor accounted for unique variance in men. It is concluded that the PSS is a multidimensional and internally consistent measure of perceived stress.This research was supported by Grant 410-91-1690 from the social sciences and Humanities Research Council of Canada as well as by grants from the Research and Program Evaluation Committee Brockville Psychiatric Hospital.     

Borderline and schizophrenic patients: A comparison using the MCMI-II

In the present study we examined the Millon Clinical Multiaxial Inventory-II scores of psychiatric patients with a diagnosis of either borderline personality disorder (BPD;n=13) or schizophrenia (n=13). The analyses revealed that the BPD patients were significantly elevated on a number of scales in addition to the Borderline scale. The second goal of this study was to assess the discriminative validity of the Borderline and Thought Disorder Scales with both base rate (BR) cutoffs and the number of endorsed prototypic items. The Borderline scale demonstrated the most diagnostic power when using BR cutoffs, whereas the Thought Disorder scale performed poorly, no better than chance levels, with all cutoffs.This research was supported by Social Sciences and Humanities Research Council of Canada Grant 410-89-0335 and by a grant from the Department of Research and Programme Evaluation, Brockville Psychiatric Hospital. The authors thank Renate Simmons for her assistance.     

A model-based standardization approach that separates true bias/DIF from group ability differences and detects test bias/DTF as well as item bias/DIF

A model-based modification (SIBTEST) of the standardization index based upon a multidimensional IRT bias modeling approach is presented that detects and estimates DIF or item bias simultaneously for several items. A distinction between DIF and bias is proposed. SIBTEST detects bias/DIF without the usual Type 1 error inflation due to group target ability differences. In simulations, SIBTEST performs comparably to Mantel-Haenszel for the one item case. SIBTEST investigates bias/DIF for several items at the test score level (multiple item DIF called differential test functioning: DTF), thereby allowing the study of test bias/DIF, in particular bias/DIF amplification or cancellation and the cognitive bases for bias/DIF.This research was partially supported by Office of Naval Research Cognitive and Neural Sciences Grant N0014-90-J-1940, 4421-548 and National Science Foundation Mathematics Grant NSF-DMS-91-01436. The research reported here is collaborative in every respect and the order of authorship is alphabetical. The assistance of Hsin-hung Li and Louis Roussos in conducting the simulation studies was of great help. Discussions with Terry Ackerman, Paul Holland, and Louis Roussos were very helpful.     

Information technology and quality of work life: Comparing users with non-users

This paper is concerned with the potential future effects of information technology on office-type work and job attitudes. An argument from labour process theory has suggested that such technology will simplify and deskill jobs, thereby reducing the quality of work life. This proposition was explored within several government agencies. Over one hundred users' and nonusers' attitudes concerning information technology's future effects upon their jobs and quality of work life were assessed and compared. Data indicates that respondents did not anticipate uniform deskilling effects in the future. Differences between computer users and nonusers were limited, although significant differences between the two groups did exist for communication, job complexity and accountability/feedback scales. Effects of socio-demographic variables upon attitudes were also assessed and the results suggest that salary, age and education may be responsible for attitude differences that occur within each group. Implications for practitioners and researchers are outlined.The authors would like to thank Nancy Carter, Aaron Cohen, Cathy Kirchmeyer and Richard Long for their insightful comments made on an earlier draft of this paper as well as Brenda McPhail's and Hj Shawyer's editorial assistance. Usual disclaimers apply. Financial support for this research project was provided by the Social Sciences Research Council of Canada, Contract No. 494-86-0003 and by a grant from the Alberta Manpower Program, Contract No. 083080 and the Faculty of Management, The University of Lethbridge. The views expressed in this paper are the author's own and are not necessarily shared by these organizations.At the time this research was conducted the second author was an undergraduate student at the University of Lethbridge.