Generating items during testing: Psychometric issues and models

On-line item generation is becoming increasingly feasible for many cognitive tests. Item generation seemingly conflicts with the well established principle of measuring persons from items with known psychometric properties. This paper examines psychometric principles and models required for measurement from on-line item generation. Three psychometric issues are elaborated for item generation. First, design principles to generate items are considered. A cognitive design system approach is elaborated and then illustrated with an application to a test of abstract reasoning. Second, psychometric models for calibrating generating principles, rather than specific items, are required. Existing item response theory (IRT) models are reviewed and a new IRT model that includes the impact on item discrimination, as well as difficulty, is developed. Third, the impact of item parameter uncertainty on person estimates is considered. Results from both fixed content and adaptive testing are presented.This article is based on the Presidential Address Susan E. Embretson gave on June 26, 1999 at the 1999 Annual Meeting of the Psychometric Society held at the University of Kansas in Lawrence, Kansas. —Editor     

Additive conjoint isotonic probabilistic models (ADISOP)

The ISOP-model or model of twodimensional or bi-isotonicity (Scheiblechner, 1995) postulates that the probabilities of ordered response categories increase isotonically in the order of subject ability and item easiness. Adding a conventional cancellation axiom for the factors of subjects and items gives the ADISOP model where the c.d.f.s of response categories are functions of an additive item and subject parameter and an ordinal category parameter. Extending cancellation to the interactions of subjects and categories as well as of items and categories (independence axiom of the category factor from the subject and item factor) gives the CADISOP model (completely additive model) in which the parallel c.d.f.s are functions of the sum of subject, item and category parameters. The CADISOP model is very close to the unidimensional version of the polytomous Rasch model with the logistic item/category characteristic(s) replaced by nonparametric axioms and statistics. The axioms, representation theorems and algorithms for model fitting of the additive models are presented.     

Reinforcement rate and interresponse time differentiation

Reinforcement rate and differential reinforcement of IRTs were independently manipulated to assess their relative contribution to the control of interresponse times (IRTs). Modified percentile reinforcement schedules (Platt, 1973) allowed control of reinforcement rate while longest or shortest IRTs were selectively reinforced. In the absence of differential IRT reinforcement, mean IRT decreased with increasing reinforcement rate. Compared to this small effect of reinforcement rate, reinforcement of long IRTs produced large changes in mean IRT at constant reinforcement rates. No interaction of reinforcement rate and IRT reinforcement was detected. The demonstration of large IRT changes in the absence of reinforcement-rate changes indicates the precedence of IRT reinforcement over molar reinforcement-rate correlations in the determination of IRTs in these procedures.     

Measuring individual differences in sensitivities to basic emotions in faces

The assessment of individual differences in facial expression recognition is normally required to address two major issues: (1) high agreement level (ceiling effect) and (2) differential difficulty levels across emotions. We propose a new assessment method designed to quantify individual differences in the recognition of the six basic emotions, 'sensitivities to basic emotions in faces.' We attempted to address the two major assessment issues by using morphing techniques and item response theory (IRT). We used morphing to create intermediate, mixed facial expression stimuli with various levels of recognition difficulty. Applying IRT enabled us to estimate the individual latent trait levels underlying the recognition of respective emotions (sensitivity scores), unbiased by stimulus properties that constitute difficulty. In a series of two experiments we demonstrated that the sensitivity scores successfully addressed the two major assessment issues and their concomitant individual variability. Intriguingly, correlational analyses of the sensitivity scores to different emotions produced orthogonality between happy and non-happy emotion recognition. To our knowledge, this is the first report of the independence of happiness recognition, unaffected by stimulus difficulty.     

Selecting the number of classes under latent class regression: a factor analytic analogue

Recently, the regression extension of latent class analysis (RLCA) model has received much attention in the field of medical research. The basic RLCA model summarizes shared features of measured multiple indicators as an underlying categorical variable and incorporates the covariate information in modeling both latent class membership and multiple indicators themselves. To reduce complexity and enhance interpretability, one usually fixes the number of classes in a given RLCA. Often, goodness of fit methods comparing various estimated models are used as a criterion to select the number of classes. In this paper, we propose a new method that is based on an analogous method used in factor analysis and does not require repeated fitting. Two ideas with application to many settings other than ours are synthesized in deriving the method: a connection between latent class models and factor analysis, and techniques of covariate marginalization and elimination. A Monte Carlo simulation study is presented to evaluate the behavior of the selection procedure and compare to alternative approaches. Data from a study of how measured visual impairments affect older persons’ functioning are used for illustration.This work was supported by National Institute on Aging (NIA) Program Project P01-AG-10184-03. The author wishes to thank Dr. Karen Bandeen-Roche for her stimulating comments and helpful discussions, and Drs. Gary Rubin and Sheila West for kindly making the Salisbury Eye Evaluation data available.     

Does covenant marriage predict latent trajectory groups of newlywed couples?

Covenant marriage is a legally distinct marriage license available in Arizona, Louisiana, and Arkansas in the United States. This study revisited one of the largest longitudinal dyadic studies on covenant versus standard newlywed couples. Using this dataset of 677 different-sex couples, we explored whether being in a covenant marriage could predict how marital satisfaction developed over the first five marital years. We applied the Growth Mixture Modeling (GMM) method to identify groups that differed in their initial marital satisfaction and trajectory. The results revealed three different trajectory groups—one group showing high and stable marital satisfaction, which we named High Stable, and two groups showing declines in marital satisfaction, one being medium and the other one being low in satisfaction at the beginning of the marriage, and we named them Medium Declining and Low Declining respectively. Spouses with lower initial marital satisfaction experienced a faster decline and suffered the highest divorce rate. Actor-Partner Interdependence Model (APIM) analysis revealed that one's group membership was predicted by their partner's membership, suggesting a mutual influence on marital development. Being in a covenant marriage was able to predict husbands' membership as covenant husbands were found to be more likely in the high stable group.     

Modelling multiple problem-solving strategies and strategy shift in cognitive diagnosis for growth

Problem-solving strategies, defined as actions people select intentionally to achieve desired objectives, are distinguished from skills that are implemented unintentionally. In education, strategy-oriented instructions that guide students to form problem-solving strategies are found to be more effective for low-achieving students than the skill-oriented instructions designed for enhancing their skill implementation ability. Although the existing longitudinal cognitive diagnosis models (CDMs) can model the change in students' dynamic skill mastery status over time, they are not designed to model the shift in students' problem-solving strategies. This study proposes a longitudinal CDM that considers both between-person multiple strategies and within-person strategy shift. The model, separating the strategy choice process from the skill implementation process, is intended to provide diagnostic information on strategy choice as well as skill mastery status. A simulation study is conducted to evaluate the parameter recovery of the proposed model and investigate the consequences of ignoring the presence of multiple strategies or strategy shift. Further, an empirical data analysis is conducted to illustrate the use of the proposed model to measure strategy shift, growth in skill implementation ability and skill mastery status.     

Flexible Bayesian modelling in dichotomous item response theory using mixtures of skewed item curves

Most item response theory (IRT) models for dichotomous responses are based on probit or logit link functions which assume a symmetric relationship between the probability of a correct response and the latent traits of individuals taking a test. This assumption restricts the use of those models to the case in which all items behave symmetrically. On the other hand, asymmetric models proposed in the literature impose that all the items in a test behave asymmetrically. This assumption is inappropriate for great majority of tests which are, in general, composed of both symmetric and asymmetric items. Furthermore, a straightforward extension of the existing models in the literature would require a prior selection of the items' symmetry/asymmetry status. This paper proposes a Bayesian IRT model that accounts for symmetric and asymmetric items in a flexible but parsimonious way. That is achieved by assigning a finite mixture prior to the skewness parameter, with one of the mixture components being a point mass at zero. This allows for analyses under both model selection and model averaging approaches. Asymmetric item curves are designed through the centred skew normal distribution, which has a particularly appealing parametrization in terms of parameter interpretation and computational efficiency. An efficient Markov chain Monte Carlo algorithm is proposed to perform Bayesian inference and its performance is investigated in some simulated examples. Finally, the proposed methodology is applied to a data set from a large-scale educational exam in Brazil.     

Marginal maximum likelihood estimation for a psychometric model of discontinuous development

Item response theory models posit latent variables to account for regularities in students' performances on test items. Wilson's “Saltus” model extends the ideas of IRT to development that occurs in stages, where expected changes can be discontinuous, show different patterns for different types of items, or even exhibit reversals in probabilities of success on certain tasks. Examples include Piagetian stages of psychological development and Siegler's rule-based learning. This paper derives marginal maximum likelihood (MML) estimation equations for the structural parameters of the Saltus model and suggests a computing approximation based on the EM algorithm. For individual examinees, empirical Bayes probabilities of learning-stage are given, along with proficiency parameter estimates conditional on stage membership. The MML solution is illustrated with simulated data and an example from the domain of mixed number subtraction. The authors' names appear in alphabetical order. We would like to thank Karen Draney for computer programming, Kikumi Tatsuoka for allowing us to use the mixed-number subtraction data, and Eric Bradlow, Chan Dayton, Kikumi Tatsuoka, and four anonymous referees for helpful suggestions. The first author's work was supported by Contract No. N00014-88-K-0304, R&T 4421552, from the Cognitive Sciences Program, Cognitive and Neural Sciences Division, Office of Naval Research, and by the Program Research Planning Council of Educational Testing Service. The second author's work was supported by a National Academy of Education Spencer Fellowship and by a Junior Faculty Research Grant from the Committee on Research, University of California at Berkeley. A copy of the Saltus computer program can be obtained from the second author.     

Item bias detection using loglinear irt

A method is proposed for the detection of item bias with respect to observed or unobserved subgroups. The method uses quasi-loglinear models for the incomplete subgroup × test score × Item 1 × ... × itemk contingency table. If subgroup membership is unknown the models are Haberman's incomplete-latent-class models.The (conditional) Rasch model is formulated as a quasi-loglinear model. The parameters in this loglinear model, that correspond to the main effects of the item responses, are the conditional estimates of the parameters in the Rasch model. Item bias can then be tested by comparing the quasi-loglinear-Rasch model with models that contain parameters for the interaction of item responses and the subgroups.The author thanks Wim J. van der Linden and Gideon J. Mellenbergh for comments and suggestions and Frank Kok for empirical data.