On the empirical indistinguishability of knowledge structures

In recent years a number of articles have focused on the identifiability of the basic local independence model. The identifiability issue usually concerns two model parameter sets predicting an identical probability distribution on the response patterns. Both parameter sets are applied to the same knowledge structure. However, nothing is known about cases where different knowledge structures predict the same probability distribution. This situation is referred to as ʻempirical indistinguishabilityʼ between two structures and is the main subject of the present paper. Empirical indistinguishability is a stronger form of unidentifiability, which involves not only the parameters, but also the structural and combinatorial properties of the model. In particular, as far as knowledge structures are concerned, a consequence of empirical indistinguishability is that the existence of certain knowledge states cannot be empirically established. Most importantly, it is shown that model identifiability cannot guarantee that a certain knowledge structure is empirically distinguishable from others. The theoretical findings are exemplified in a number of different empirical scenarios.     

CLUSTER ORIENTED FACTOR SOLUTIONS: OBLIQUE POWERED VECTOR FACTOR ANALYSIS

A conceptual approach to linguistic data processing problems is sketched and empirical illustrations are presented of the major software components- indexing, storage, and retrieval-of a document processing system which offers, in principle, the advantages of complete automation, unlimited cross- indexing, effective sequential retrieval, sub-documentary indexing reflecting heterogeneity of subject matter within a document, and a procedure for automatically identifying retrieval requests which would be inadequately handled by the system. The indexing schema, designated as a "Classification Space" consists of a Euclidean model for mapping subject matter similarity within a given subject matter domain. A schema of this kind is empirically derived for certain fields of Engineering and Chemistry. A set of five related empirical studies provide convincing evidence that when appropriate experimental procedures are followed a very stable C-Space for a given content domain can be constructed on a surprisingly small data base. Other empirical studies demonstrate specific computational procedures for effective automatic indexing of documents in a C-Space, using a relatively small system vocabulary. One study demonstrates that a C-Space maps subject matter relevance as well as subject matter similarity, and thereby pro- motes effective sequential retrieval ; this result is also shown under conditions of automatic indexing. Negative results are found in an attempt to use the structural linguistic distinction of subject and object as a means of improving techniques for automatic indexing.     

Assessing Parameter Invariance in the BLIM: Bipartition Models

In knowledge space theory, the knowledge state of a student is the set of all problems he is capable of solving in a specific knowledge domain and a knowledge structure is the collection of knowledge states. The basic local independence model (BLIM) is a probabilistic model for knowledge structures. The BLIM assumes a probability distribution on the knowledge states and a lucky guess and a careless error probability for each problem. A key assumption of the BLIM is that the lucky guess and careless error probabilities do not depend on knowledge states (invariance assumption). This article proposes a method for testing the violations of this specific assumption. The proposed method was assessed in a simulation study and in an empirical application. The results show that (1) the invariance assumption might be violated by the empirical data even when the model’s fit is very good, and (2) the proposed method may prove to be a promising tool to detect invariance violations of the BLIM.     

Modellierung von Fehlkonzepten in einer algebraischen Wissensstruktur

An extension of the theory of knowledge spaces by Doignon &; Falmagne (1985) is presented that tries to account for subjects’ typical errors and wrong answers. This extension concerns two major points: The usual dichotomous item format (right/wrong) is generalized to polytomous response categories, and the theoretical structure (knowledge, skills, misconceptions) is clearly separated from the empirical structure (observable solution behavior, subject’s responses). Using examples from a set of questions about properties of simple electric circuits the general method is demonstrated. Axioms of an algebraic structure known as “information system” (Scott, 1982) are shown to provide an appropriate characterization of the theoretical domain. The structural properties of the data, on the other hand, can be derived from assumptions about the influence of knowledge and misconceptions on specific answers for a set of questions.     

An iterative procedure for extracting skill maps from data

The methodologies for the construction of a knowledge structure mainly refer to the query to experts, the skill maps, and the data-driven approaches. This last method is of growing interest in recent literature. In this paper, an iterative procedure for building a skill map from a set of data is introduced. This procedure is based on the minimization of the distance between the knowledge structure delineated by a given skill map and the data. The accuracy of the proposed method is tested through a number of simulation studies where the amount of noise in the data is manipulated as well as the kind of structure to be reconstructed. Results show that the procedure is accurate and that its performance tends to be sufficiently stable even with high error rates. The procedure is compared to two already-existing methodologies to derive knowledge structures from a set of data. The use of the corrected Akaike Information Criterion (AICc) as a stopping criterion of the iterative reconstruction procedure is tested against the app criterion introduced by Schrepp. Moreover, two empirical applications on clinical data are reported, and their results show the applicability of the procedure.     

Suppression of valid inferences and knowledge structures: the curious effect of producing alternative antecedents on reasoning with causal conditionals

These studies looked at the difficulty that reasoners have in accepting conditional ("If P then Q") major premises that are not necessarily true empirically, as a basis for deductive reasoning. Preliminary results have shown that when reasoners are asked to produce possible alternate antecedents to the major premise ("If A then Q"), they paradoxically tend to deny the modus ponens (MP) inference ("If P is true, then Q is true"). Three studies further explored these results. The first study gave university students paper-and-pencil tests in which instructions to "suppose that the major premise is true" was followed by a request to determine the next number in a sequence, to retrieve information unrelated to the premises, or to retrieve a possible case of "If A then Q." Relative to a control group, reasoners asked to produce an alternative antecedent showed a significant tendency to deny the MP inference, whereas no such tendency was observed for the two other tasks used. A second study compared performance on a condition in which reasoners were asked to produce an alternative antecedent with that when they were given an explicit alternative. Premises used in this study were such that the latter alternative antecedent was also spontaneously produced by over 70% of reasoners. Results showed that the tendency to refuse the MP premise could not be accounted for by the specific nature of the alternative produced. A third study found that the tendency to refuse the MP inference after producing an alternative antecedent was affected by the number of "disabling conditions" (i.e., conditions that allow "P to be true" and "Q to be false") available for the major premise. These results are interpreted as being consistent with a model that supposes that logical reasoning requires selective inhibition of real-world knowledge.     

Distributed skill functions and the meshing of knowledge structures

The scope of knowledge space theory was extended by bringing into the picture the underlying skills and capabilities that are relevant to solving the problems in a knowledge domain. A major challenge to this approach comes from the need to aggregate distributed information on (partially) overlapping domains and skill sets. The notion of a distributed skill function is introduced for formalizing the integration of several skill functions that represent the assignment of skills to problems. It is shown that their consistency is captured by the meshability of the delineated knowledge structures. This result draws upon a characterization of the meshing of finite or infinite collections of knowledge structures, which extends and generalizes previous results on the binary case. The discussion covers implications for knowledge assessment and for practical applications, such as integrating skill assignments coming from different experts or distributed resources in technology-enhanced learning.     

认知诊断评价中测验结构的优化设计

Q矩阵是认知诊断评价的基础和核心要素, 它反映了测验的构念和内容设计, 直接影响着测验诊断分类的效果。本文采用Monte Carlo模拟, 研究了6种属性层级关系下, 不同的Q矩阵设计对于认知诊断效果的影响。用模式判准率的均值和标准差分别从分类准确性和稳定性的角度来评价诊断效果。实验结果表明：(1) 不同属性层级关系下, 分类准确性会随着测验长度的增加而提高, 但当测验长度增加到一定程度时, 会出现“天花板效应”; (2) Q矩阵中R*的个数(NR*)会影响测验的分类准确性及稳定性：NR*越大, 测验的分类稳定性越高, 当测验长度为属性个数的整数倍, 且NR*为测验长度相对属性个数的最大奇数倍时分类准确性最高; (3) Q矩阵中除R*以外的项目考察的属性个数会随着属性层级关系的不同对测验的分类准确性和稳定性产生不同的影响。根据实验结果, 本研究提出了进行诊断评价时Q矩阵优化设计的一些建议。     

认知诊断中的Q矩阵及其作用

Q矩阵作为连接认知和测量的桥梁,在认知诊断中起重要作用。本文梳理了应用Q矩阵解决认知诊断相关问题的理论与方法。首先整理Q矩阵的相关概念、算法、性质及其在认知诊断中的作用;并根据Q矩阵可计算理论构念效度、可以构成格等,指出Q矩阵是特殊的关联矩阵;接着介绍Q矩阵理论研究方面的几个近期发展;并对Q矩阵未来的应用研究作出展望。期望本文能为测量工作者更灵活地利用Q矩阵提供参考和帮助。     

认知诊断中的Q矩阵及其作用

Q矩阵作为连接认知和测量的桥梁，在认知诊断中起重要作用。本文梳理了应用Q矩阵解决认知诊断相关问题的理论与方法。首先整理Q矩阵的相关概念、算法、性质及其在认知诊断中的作用；并根据Q矩阵可计算理论构念效度、可以构成格等，指出Q矩阵是特殊的关联矩阵；接着介绍Q矩阵理论研究方面的几个近期发展；并对Q矩阵未来的应用研究作出展望。期望本文能为测量工作者更灵活地利用Q矩阵提供参考和帮助。     

An empirical Q-matrix validation method for the sequential generalized DINA model

As a core component of most cognitive diagnosis models, the Q-matrix, or item and attribute association matrix, is typically developed by domain experts, and tends to be subjective. It is critical to validate the Q-matrix empirically because a misspecified Q-matrix could result in erroneous attribute estimation. Most existing Q-matrix validation procedures are developed for dichotomous responses. However, in this paper, we propose a method to empirically detect and correct the misspecifications in the Q-matrix for graded response data based on the sequential generalized deterministic inputs, noisy ‘and’ gate (G-DINA) model. The proposed Q-matrix validation procedure is implemented in a stepwise manner based on the Wald test and an effect size measure. The feasibility of the proposed method is examined using simulation studies. Also, a set of data from the Trends in International Mathematics and Science Study (TIMSS) 2011 mathematics assessment is analysed for illustration.     

How Experts Solve a Novel Problem in Experimental Design

Research on expert-novice differences has mainly focused on how experts solve familiar problems. We know far less about the skills and knowledge used by experts when they are confronted with novel problems within their area of expertise. This article discusses a study in which verbal protocols were taken from subjects of various expertise designing an experiment in an area with which they were unfamiliar. The results showed that even when domain knowledge is lacking, experts solve a novel problem within their area of expertise by dividing the problem into a number of subproblems that are solved in a specified order. The lack of domain knowledge is compensated for by using abstract knowledge structures and domain-specific heuristic strategies. However, the quality of their solutions is considerably lower than the quality attained by experts who were familiar with the type of problem to be solved. The results suggest that when experts are confronted with novel problems as compared with familiar problems, their form of reasoning remains intact, but the content of their reasoning suffers due to lack of domain knowledge.     

Knowledge Organization and the Acquisition of Procedural Expertise

The influence of the organization of a declarative knowledge base on the development and application of proceduralized knowledge was investigated in a complex troubleshooting domain. Domain explanations were generated in either a depth-first or breadth-first manner for different groups of subjects who were also given experience learning to troubleshoot in the domain. Although the two explanatory structures led to similar training performance, the two groups differed significantly in their overall level of performance in subsequent troubleshooting problems. Examination of objective measures of troubleshooting performance and think-aloud protocols indicated that breadth-first declarative knowledge representation fosters the use of mental models during problem-solving in training. It also facilitates proceduralization of that knowledge into fast and accurate methods for localizing faults.     

A learning algorithm for boltzmann machines

The computational power of massively parallel networks of simple processing elements resides in the communication bandwidth provided by the hardware connections between elements. These connections can allow a significant fraction of the knowledge of the system to be applied to an instance of a problem in a very short time. One kind of computation for which massively parallel networks appear to be well suited is large constraint satisfaction searches, but to use the connections efficiently two conditions must be met: First, a search technique that is suitable for parallel networks must be found. Second, there must be some way of choosing internal representations which allow the preexisting hardware connections to be used efficiently for encoding the constraints in the domain being searched. We describe a general parallel search method, based on statistical mechanics, and we show how it leads to a general learning rule for modifying the connection strengths so as to incorporate knowledge about a task domain in an efficient way. We describe some simple examples in which the learning algorithm creates internal representations that are demonstrably the most efficient way of using the preexisting connectivity structure.     

Logical and psychological partitioning of mind: Depicting the same map?

The aim of this paper is to demonstrate that empirically delimited structures of mind are also differentiable by means of systematic logical analysis. For the sake of this aim, the paper first summarizes Demetriou’s theory of cognitive organization and growth. This theory assumes that the mind is a multistructural entity that develops across three fronts: the processing system that constrains processing potentials, a set of specialized structural systems (SSSs) that guide processing within different reality and knowledge domains, and a hypercognitive system that monitors and controls the functioning of all other systems. The second part of the paper focuses on the SSSs, which are the target of our logical analysis, and it summarizes a series of empirical studies demonstrating their autonomous operation. The third part develops the logical proof showing that each SSS involves a kernel element that cannot be reduced to standard logic or to any other SSS. The implications of this analysis for the general theory of knowledge and cognitive development are discussed.     

Expertise as mental set: The effects of domain knowledge in creative problem solving

Experts generally solve problems in their fields more effectively than novices because their wellstructured, easily activated knowledge allows for efficient search of a solution space. But what happens when a problem requires a broad search for a solution? One concern is that subjects with a large amount of domain knowledge may actually be at a disadvantage, because their knowledge may confine them to an area of the search space in which the solution does not reside. In other words, domain knowledge may act as a mental set, promoting fixation in creative problem-solving attempts. A series of three experiments in which an adapted version of Mednick’s (1962) remote associates task was used demonstrates conditions under which domain knowledge may inhibit creative problem solving.     

代数应用题项目生成中的认知过程与任务特征分析

认知项目设计起源于理论驱动的测量模式,要求在详尽阐述测量建构理论的基础上,鉴别测验项目类型的任务特征,建立和标定与测量建构认知变量之间的对应关系,从而通过系统变换任务特征生成项目.本文以Greeno (1994)提出的代理者-情境交互作用(agent-situation interaction)的问题解决观为基础,对代数应用题解决的认知过程和任务蕴含性(affordance)进行了系统分析.具体而言,代数应用题被视为以文本形式呈现的,嵌套在现实情境中的某种量化结构.在代数应用题解决的转译、整合、计划、执行的认知过程中需要形成三种不同的表征,即文本基、情境模型和问题模型.其形成及解决过程受到个体问题图式知识的深刻影响.对应于不同表征方式,对代数应用题语法/语义结构、量化元素及结构、问题情境特征进行了系统分析和综合.该分析表明,为了保证代数应用题项目生成的科学性,当前研究的关键在于建立任务特征与问题解决认知变量之间的经验关系,并在问题领域整体水平上检验这种关系的可推广性.     

Judging a book by its cover: Interpretative effects of content on problem-solving transfer

We examine how cover stories of isomorphic problems affect transfer. Existing models posit that people retain content in problem representations and that similarities and differences between the “undeleted” cover stories might interfere with recognition of structural similarities.We propose that cover stories can affect transfer in another way—by inducing semantic knowledge that modifies problem structures. Two experiments examined how people represent and solve permutation problems dealing with random assignment of elements from one set to elements from another set. Although the problems were structurally isomorphic, cover stories involving different pairs of element sets led subjects to abstract different “interpreted structures.” Problems involving objects and people (e.g., prizes and students) led subjects to abstract an asymmetric structure (“get”) and problems involving similar sets of people (e.g., doctors and doctors) led subjects to abstract a symmetric structure (“pair”). Transfer was mediated by similarities and differences between the interpreted structures of the learned and the novel problems.     

The structure of choice

An unsolved fundamental problem in decision science concerns the extent to which the nature of the perceived relationships among items in a set of alternatives influences how they are chosen. More specifically, given a choice set with n items, how does human choice behavior differ as a function of the perceived relationships between the items of the set? In what follows, we study this problem empirically and theoretically from the standpoint of the dimensional structure of the choice set. In particular, we use generalized invariance structure theory (GIST; Vigo, 2013, 2014) to propose an inverse relationship between the degree of concept learning difficulty of a choice set (as determined by its degree of invariance or internal coherence) and choice response times on its members. To our knowledge, this is the first model that precisely unifies these two fundamental constructs. On average, the model, without free parameters, accounts for nearly 90% of the variance in the data from our two response-time experiments.     

A U-shaped model for the development of intuition by level of expertise

Based on a review of the literature, this paper proposes a non-linear U-shaped model of intuition development influenced by an individual's level of expertise within a given subject area. Two qualitatively different types of intuition are described: immature intuition and mature intuition, each differentiated by the level of expertise of the individual in a specific subject area. Immature intuition is most available when an individual is a novice in a given knowledge domain, where his/her analytical knowledge of the subject does not interfere with the ability to make novel insights. Mature intuition is more rare and is most available when an individual is more of an expert in the subject area with well-developed relevant knowledge structures. Issues regarding the viability of this preliminary model are discussed.