On the assessment of procedural knowledge: From problem spaces to knowledge spaces

By generalizing and completing the work initiated by Stefanutti and Albert (2003, Journal of Universal Computer Science, 9, 1455), this article provides the mathematical foundations of a theoretical approach whose primary goal is to construct a bridge between problem solving, as initially conceived by Newell and Simon (1972, Human problem solving. Englewood Cliffs, NJ: Prentice-Hall.), and knowledge assessment (Doignon and Falmagne, 1985, International Journal of Man-Machine Studies, 23, 175; Doignon and Falmagne, 1999, Knowledge spaces. Berlin, Germany: Springer-Verlag.; Falmagne et al., 2013, Knowledge spaces: Applications in education. New York, NY: Springer-Verlag; Falmagne and Doignon, 2011, Learning spaces: Interdisciplinary applied mathematics. Berlin, Germany: Springer-Verlag.). It is shown that the collection of all possible knowledge states for a given problem space is a learning space. An algorithm for deriving a learning space from a problem space is illustrated. As an example, the algorithm is used to derive the learning space of a neuropsychological test whose problem space is well known: the Tower of London (TOL; Shallice, 1982, Philosophical Transactions of the Royal Society of London B: Biological Sciences, 298, 199). The derived learning space could then be used for adaptively assessing individual planning skills with the TOL.     

Residualanalysis in the polytomous rasch model

Residuals for check of model fit in the polytomous Rasch model are examined. Comparisons are made between using counts for all response pattern and using item totals for score groups for the construction of the residuals. Comparisons are also, for the residuals based on score group totals, made between using as basis the item totals, or using the estimated item parameters. The developed methods are illustrated by two examples, one from a psychiatric rating scale, one from a Danish Welfare Study.     

多级属性Q矩阵的验证与估计

多级属性是将诊断测验中传统的二值(即两种水平, 通常定义为0和1)属性定义为多值(多个水平可以为0, 1, …), 它不但可以描述学生对于知识属性是否掌握, 而且可以描述学生在属性上的掌握程度, 这样使得诊断测验能提供给被试更丰富的知识掌握详情。本文将适用于二级属性Q矩阵的统计量(S统计量)拓展到多级属性下的Q矩阵验证和估计, 在两种常见的条件下, 设计了两种估计算法：联合估计算法和在线估计算法。模拟实验结果表明：联合估计算法适用于对专家界定的初始Q矩阵进行验证, 当初始Q矩阵中包含较少的错误时, 通过联合估计算法有很大可能恢复正确的Q矩阵; 在线估计算法适用于对“新项目”进行属性向量和项目参数的在线标定, 基于一定数量的“基础项目”, 在线估计算法对于新项目的估计也能达到较满意的成功率。实证数据分析则进一步展示了该方法的使用。     

The assessment of knowledge and learning in competence spaces: The gain–loss model for dependent skills

The gain–loss model (GaLoM) is a formal model for assessing knowledge and learning. In its original formulation, the GaLoM assumes independence among the skills. Such an assumption is not reasonable in several domains, in which some preliminary knowledge is the foundation for other knowledge. This paper presents an extension of the GaLoM to the case in which the skills are not independent, and the dependence relation among them is described by a well‐graded competence space. The probability of mastering skill s at the pretest is conditional on the presence of all skills on which s depends. The probabilities of gaining or losing skill s when moving from pretest to posttest are conditional on the mastery of s at the pretest, and on the presence at the posttest of all skills on which s depends. Two formulations of the model are presented, in which the learning path is allowed to change from pretest to posttest or not. A simulation study shows that models based on the true competence space obtain a better fit than models based on false competence spaces, and are also characterized by a higher assessment accuracy. An empirical application shows that models based on pedagogically sound assumptions about the dependencies among the skills obtain a better fit than models assuming independence among the skills.     

The asymptotic posterior normality of the latent trait for polytomous IRT models

Chang and Stout (1993) presented a derivation of the asymptotic posterior normality of the latent trait given examinee responses under nonrestrictive nonparametric assumptions for dichotomous IRT models. This paper presents an extention of their results to polytomous IRT models in a fairly straightforward manner. In addition, a global information function is defined, and the relationship between the global information function and the currently used information functions is discussed. An information index that combines both the global and local information is proposed for adaptive testing applications.This research was partially supported by Educational Testing Service Allocation Project No. 79424. The author wishes to thank Charles Davis, Xuming He, Frank Jenkins, Spence Swinton, William Stout, Ming-Mai Wang, and Zhiliang Ying for their helpful comments and discussions. The author particularly wishes to thank the Editor, Shizuhiko Nishisato, the Associate Editor, and three anonymous reviewers for their thoroughness and thoughtful suggestions.     

Analysis of structural equation model with ignorable missing continuous and polytomous data

The main purpose of this article is to develop a Bayesian approach for structural equation models with ignorable missing continuous and polytomous data. Joint Bayesian estimates of thresholds, structural parameters and latent factor scores are obtained simultaneously. The idea of data augmentation is used to solve the computational difficulties involved. In the posterior analysis, in addition to the real missing data, latent variables and latent continuous measurements underlying the polytomous data are treated as hypothetical missing data. An algorithm that embeds the Metropolis-Hastings algorithm within the Gibbs sampler is implemented to produce the Bayesian estimates. A goodness-of-fit statistic for testing the posited model is presented. It is shown that the proposed approach is not sensitive to prior distributions and can handle situations with a large number of missing patterns whose underlying sample sizes may be small. Computational efficiency of the proposed procedure is illustrated by simulation studies and a real example.The work described in this paper was fully supported by a grant from the Research Grants Council of the HKSAR (Project No. CUHK 4088/99H). The authors are greatly indebted to the Editor and anonymous reviewers for valuable comments in improving the paper; and also to D. E. Morisky and J.A. Stein for the use of their AIDS data set.     

Stochastic ordering using the latent trait and the sum score in polytomous IRT models

In a restricted class of item response theory (IRT) models for polytomous items the unweighted total score has monotone likelihood ratio (MLR) in the latent trait. MLR implies two stochastic ordering (SO) properties, denoted SOM and SOL, which are both weaker than MLR, but very useful for measurement with IRT models. Therefore, these SO properties are investigated for a broader class of IRT models for which the MLR property does not hold.In this study, first a taxonomy is given for nonparametric and parametric models for polytomous items based on the hierarchical relationship between the models. Next, it is investigated which models have the MLR property and which have the SO properties. It is shown that all models in the taxonomy possess the SOM property. However, counterexamples illustrate that many models do not, in general, possess the even more useful SOL property.Hemker's research was supported by the Netherlands Research Council, Grant 575-67-034. Junker's research was supported in part by the National Institutes of Health, Grant CA54852, and by the National Science Foundation, Grant DMS-94.04438.     

A person-fit index for polytomous rasch models,latent class models,and their mixture generalizations

A normally distributed person-fit index is proposed for detecting aberrant response patterns in latent class models and mixture distribution IRT models for dichotomous and polytomous data.This article extends previous work on the null distribution of person-fit indices for the dichotomous Rasch model to a number of models for categorical data. A comparison of two different approaches to handle the skewness of the person-fit index distribution is included.Major parts of this paper were written while the first author worked at the Institute for Science Education, Kiel, Germany. Any opinions expressed in this paper are those of the authors and not necessarily of Educational Testing Service. The results presented in this paper were improved by valuable comments from J. Rost, K. Yamamoto, N.D. Verhelst, E. Bedrick and two anonymous reviewers.     

Essential independence and likelihood-based ability estimation for polytomous items

A definition ofessential independence is proposed for sequences of polytomous items. For items satisfying the reasonable assumption that the expected amount of credit awarded increases with examinee ability, we develop a theory ofessential unidimensionality which closely parallels that of Stout. Essentially unidimensional item sequences can be shown to have a unique (up to change-of-scale) dominant underlying trait, which can be consistently estimated by a monotone transformation of the sum of the item scores. In more general polytomous-response latent trait models (with or without ordered responses), anM-estimator based upon maximum likelihood may be shown to be consistent for under essentially unidimensional violations of local independence and a variety of monotonicity/identifiability conditions. A rigorous proof of this fact is given, and the standard error of the estimator is explored. These results suggest that ability estimation methods that rely on the summation form of the log likelihood under local independence should generally be robust under essential independence, but standard errors may vary greatly from what is usually expected, depending on the degree of departure from local independence. An index of departure from local independence is also proposed.This work was supported in part by Office of Naval Research Grant N00014-87-K-0277 and National Science Foundation Grant NSF-DMS-88-02556. The author is grateful to William F. Stout for many helpful comments, and to an anonymous reviewer for raising the questions addressed in section 2. A preliminary version of section 6 appeared in the author's Ph.D. thesis.     

On the empirical indistinguishability of knowledge structures

In recent years a number of articles have focused on the identifiability of the basic local independence model. The identifiability issue usually concerns two model parameter sets predicting an identical probability distribution on the response patterns. Both parameter sets are applied to the same knowledge structure. However, nothing is known about cases where different knowledge structures predict the same probability distribution. This situation is referred to as ʻempirical indistinguishabilityʼ between two structures and is the main subject of the present paper. Empirical indistinguishability is a stronger form of unidentifiability, which involves not only the parameters, but also the structural and combinatorial properties of the model. In particular, as far as knowledge structures are concerned, a consequence of empirical indistinguishability is that the existence of certain knowledge states cannot be empirically established. Most importantly, it is shown that model identifiability cannot guarantee that a certain knowledge structure is empirically distinguishable from others. The theoretical findings are exemplified in a number of different empirical scenarios.     

Nonparametric item response theory axioms and properties under nonlinearity and their exemplification with knowledge space theory

This paper investigates the dichotomous Mokken nonparametric item response theory (IRT) axioms and properties under incomparabilities among latent trait values and items. Generalized equivalents of the unidimensional nonparametric IRT axioms and properties are formulated for nonlinear (quasi-ordered) person and indicator spaces. It is shown that monotone likelihood ratio (MLR) for the total score variable and nonlinear latent trait implies stochastic ordering (SO) of the total score variable, but may fail to imply SO of the nonlinear latent trait. The reason for this and conditions under which the implication holds are specified, based on a new, simpler proof of the fact that in the unidimensional case MLR implies SO. The approach is applied in knowledge space theory (KST), a combinatorial test theory. This leads to a (tentative) Mokken-type nonparametric axiomatization in the currently parametric theory of knowledge spaces. The nonparametric axiomatization is compared with the assumptions of the parametric basic local independence model which is fundamental in KST. It is concluded that this paper may provide a first step toward a basis for a possible fusion of the two split directions of psychological test theories IRT and KST.     

Salesperson knowledge distinctions and sales performance

Although it is conceptually and practically posited that salesperson knowledge has a direct relationship to performance, empirical research concerning this topic has been limited. In a free elicitation study of 150 multi-line insurance agents, the authors demonstrate that higher performing sales personnel have more elaborate, contingent and context-specific procedural knowledge than that of less effective agents. Moreover, higher performing sales personnel's contingent knowledge is shown to be more relevant to the sales call and more adaptive or responsive to the specific contingency than lower performers. In addition, higher performers have more intermediary and indirect goals prior to initially contacting a prospect. These results fuel a call for further research concerning the procedural knowledge roots underlying sales performance, as well as the practical exigencies involved in developing, managing and retaining the knowledge resource of the sales force as a firm-level asset.     

Approximately counting and sampling knowledge states

Approximately counting and sampling knowledge states from a knowledge space is a problem that is of interest for both applied and theoretical reasons. However, many knowledge spaces used in practice are far too large for standard statistical counting and estimation techniques to be useful. Thus, in this work we use an alternative technique for counting and sampling knowledge states from a knowledge space. This technique is based on a procedure variously known as subset simulation, the Holmes–Diaconis–Ross method, or multilevel splitting. We make extensive use of Markov chain Monte Carlo methods and, in particular, Gibbs sampling, and we analyse and test the accuracy of our results in numerical experiments.     

Perception and practical knowledge

According to G.E.M. Anscombe, an agent's knowledge of his own intentional actions differs from his knowledge of his unintended behaviors as well as the knowledge others can have of what he intentionally does, in being known “without observation”. I begin by posing a problem for any conception of this theory according to which non-observational knowledge must be independent of sense perception, and criticize several recent attempts to get around the problem. Having done this, I develop an alternative account of non-observational knowledge according to which it consists in the particular causal role of an agent's self-awareness in bringing his intentional actions about.     

Quality spaces: Mental and physical

Perceptual-role theories of mental qualities hold that we can discover the nature of a being’s mental qualities by investigating that being’s capacity to make perceptual discriminations. Many advocates of perceptual-role theories hold that the best explanation of these capacities is that mental quality spaces are homomorphic to the spaces of the physical properties that they help to discriminate. This paper disputes this thesis on largely empirical grounds, and offers an alternative. The alternative explains interesting patterns in our perception of color differences and similarities across edges, emphasizing the importance of visual discontinuities. Further, it provides a simple explanation of the intransitivity of indiscriminability. The homomorphism thesis can explain this intransitivity only with the addition of an independently unsupported empirical hypothesis. Also, in contrast to the homomorphism thesis, the alternative view is perfectly consistent with empirical data that suggest (a) that the visual system uses the dynamic allocation of a limited range of responses in order to discriminate a wider range of physical stimuli and (b) that we are much better at detecting changes in relative luminance than changes in absolute luminance. Moreover, it does all this while using a more efficient strategy for encoding sensory information.     

A latent trait theory via a stochastic learning theory for a knowledge space

To capture the cognitive organization of a set of questions or problems pertaining to a body of information, Doignon and Falmagne have proposed, and analyzed in a number of papers, the concept of aknowledge space, that is, a distinguished collection of subsets of questions, representing the possibleknowledge states. This collection of sets is assumed to satisfy a number of conditions. Since this concept is a deterministic one, the problem of empirical testing arises. A stochastic version of a knowledge space is developed in this paper, in which the knowledge states are considered as possible epochs in a subject's learning history. The knowledge space is decomposed as a union of a number of possible learning paths, calledgradations. The model specifies how a subject is channelled through and progresses along a gradation. A probabilistic axiom of the local indepencence type relates the knowledge states to the observable responses. The predictions of this model are worked out in details in the case of parametric assumptions involving gamma distributions. An application of the model to artificial data is described, based on maximum likelihood methods. The statistical analysis is shown to be capable of revealing the combinatoric core of the model.This work was supported by NSF grant IST-8418860 and ARI grant DAAG29-84-G-0083 to New York University. I am grateful to Jean-Paul Doignon, Mathieu Koppen, Geoff Iverson and Michael Landy for their reactions to previous drafts of this paper, to Michael Villano for carrying out the computer simulation and the analysis of the simulated data, and especially to one referee for his very useful comments.     

Local and global reference frames for environmental spaces

Two experiments examined how locations in environmental spaces, which cannot be overseen from one location, are represented in memory: by global reference frames, multiple local reference frames, or orientation-free representations. After learning an immersive virtual environment by repeatedly walking a closed multisegment route, participants pointed to seven previously learned targets from different locations. Contrary to many conceptions of survey knowledge, local reference frames played an important role: Participants performed better when their body or pointing targets were aligned with the local reference frame (corridor). Moreover, most participants turned their head to align it with local reference frames. However, indications for global reference frames were also found: Participants performed better when their body or current corridor was parallel/orthogonal to a global reference frame instead of oblique. Participants showing this pattern performed comparatively better. We conclude that survey tasks can be solved based on interconnected local reference frames. Participants who pointed more accurately or quickly additionally used global reference frames.     

Reviving inert knowledge: analogical abstraction supports relational retrieval of past events

We present five experiments and simulation studies to establish late analogical abstraction as a new psychological phenomenon: Schema abstraction from analogical examples can revive otherwise inert knowledge. We find that comparing two analogous examples of negotiations at recall time promotes retrieving analogical matches stored in memory—a notoriously elusive effect. Another innovation in this research is that we show parallel effects for real-life autobiographical memory (Experiments 1–3) and for a controlled memory set (Experiments 4 and 5). Simulation studies show that a unified model based on schema abstraction can capture backward (retrieval) effects as well as forward (transfer) effects.