Recovering a Probabilistic Knowledge Structure by Constraining its Parameter Space

In the Basic Local Independence Model (BLIM) of Doignon and Falmagne (Knowledge Spaces, Springer, Berlin, 1999), the probabilistic relationship between the latent knowledge states and the observable response patterns is established by the introduction of a pair of parameters for each of the problems: a lucky guess probability and a careless error probability. In estimating the parameters of the BLIM with an empirical data set, it is desirable that such probabilities remain reasonably small. A special case of the BLIM is proposed where the parameter space of such probabilities is constrained. A simulation study shows that the constrained BLIM is more effective than the unconstrained one, in recovering a probabilistic knowledge structure.     

Estimation of careless error and lucky guess probabilities for dichotomous test items: A psychometric application of a biometric latent class model with random effects

Medical research has extensively dealt with the estimation of the accuracy (sensitivity and specificity) of a diagnostic test for screening individuals. In this paper we apply the biometric latent class model with random effects by Qu, Tan, and Kutner [(1996). Random effects models in latent class analysis for evaluating accuracy of diagnostic tests. Biometrics, 52, 797-810] to estimate the response error (careless error and lucky guess) probabilities for dichotomous test items in the psychometric theory of knowledge spaces. The approach is illustrated with simulated data. In particular, we extend this approach to give a generalization of the basic local independence model in knowledge space theory. This allows for local dependence among the indicators given the knowledge state of an examinee and/or for the incorporation of covariates.     

Assessing learning processes with the gain-loss model

Within the framework of knowledge space theory, a probabilistic skill multimap model for assessing learning processes is proposed. The learning process of a student is modeled as a function of the interaction between his competence state and the effect of a learning object on specific skills. Model parameters are initial probabilities of the skills, effects of learning objects on gaining and losing the skills, careless error, and lucky guess probabilities of the problems. A simulation study assessed model identifiability and goodness-of-recovery under several conditions. Practical implications of using the model are discussed, and the MATLAB code for simulating, estimating and testing it is available in the Psychonomic Society supplemental archive.     

On the unidentifiability of a certain class of skill multi map based probabilistic knowledge structures

In the process of fitting a probabilistic knowledge structure to data, standard goodness-of-fit statistics only partially describe the correctness of the fitted model. Irrespectively of how good the fit is, a too-high value of the error rates (careless error and lucky guess probabilities) might be a symptom of a misspecification of the model. In this situation, it could be critical to interpret those values as error rates. A more reasonable solution would be to hypothesize that some modifications have to be introduced in the model. In this paper, we show that in specific cases, these modifications yield basic local independence model parameterizations that are not identifiable. The applicative consequences of the theoretical results are displayed by means of an example carried out on a set of clinical data collected through the Maudsley Obsessional-Compulsive Questionnaire.     

Testing the actual equivalence of automatically generated items

If the automatic item generation is used for generating test items, the question of how the equivalence among different instances may be tested is fundamental to assure an accurate assessment. In the present research, the question was dealt by using the knowledge space theory framework. Two different ways of considering the equivalence among instances are proposed: The former is at a deterministic level and it requires that all the instances of an item template must belong to exactly the same knowledge states; the latter adds a probabilistic level to the deterministic one. The former type of equivalence can be modeled by using the BLIM with a knowledge structure assuming equally informative instances; the latter can be modeled by a constrained BLIM. This model assumes equality constraints among the error parameters of the equivalent instances. An approach is proposed for testing the equivalence among instances, which is based on a series of model comparisons. A simulation study and an empirical application show the viability of the approach.     

Empirical indistinguishability: From the knowledge structure to the skills

Recent literature has pointed out that the basic local independence model (BLIM) when applied to some specific instances of knowledge structures presents identifiability issues. Furthermore, it has been shown that for such instances the model presents a stronger form of unidentifiability named empirical indistinguishability, which leads to the fact that the existence of certain knowledge states in such structures cannot be empirically tested. In this article the notion of indistinguishability is extended to skill maps and, more generally, to the competence-based knowledge space theory. Theoretical results are provided showing that skill maps can be empirically indistinguishable from one another. The most relevant consequence of this is that for some skills there is no empirical evidence to establish their existence. This result is strictly related to the type of probabilistic model investigated, which is essentially the BLIM. Alternative models may exist or can be developed in knowledge space theory for which this indistinguishability problem disappears.     

Assessing the local identifiability of probabilistic knowledge structures

Given a collection Q of problems, in knowledge space theory Doignon & Falmagne, (International Journal of Man–Machine Studies 23:175–196, 1985) the knowledge state of a student is the collection K ? Q of all problems that this student is capable of solving. A knowledge structure is a pair (Q, ), where is a collection of knowledge states that contains at least the empty set and Q. A probabilistic knowledge structure (PKS) is a knowledge structure (Q, , π), where π is a probability distribution on the knowledge states. The PKS that has received the most attention is the basic local independence model BLIM; Falmagne & Doignon, (British Journal of Mathematical and Statistical Psychology 41:1–23, 1988a, Journal of Mathematical Psychology 32:232–258, 1988b). To the best of our knowledge, systematic investigations in the literature concerning the identifiability of the BLIM are totally missing. Based on the theoretical work of Bamber and van Santen (Journal of Mathematical Psychology 29:443–473, 1985), the present article is aimed to present a method and a corresponding computerized procedure for assessing the local identifiability of the BLIM, which is applicable to any finite knowledge structure of moderate size.     

Score-Based Tests of Differential Item Functioning via Pairwise Maximum Likelihood Estimation

Measurement invariance is a fundamental assumption in item response theory models, where the relationship between a latent construct (ability) and observed item responses is of interest. Violation of this assumption would render the scale misinterpreted or cause systematic bias against certain groups of persons. While a number of methods have been proposed to detect measurement invariance violations, they typically require advance definition of problematic item parameters and respondent grouping information. However, these pieces of information are typically unknown in practice. As an alternative, this paper focuses on a family of recently proposed tests based on stochastic processes of casewise derivatives of the likelihood function (i.e., scores). These score-based tests only require estimation of the null model (when measurement invariance is assumed to hold), and they have been previously applied in factor-analytic, continuous data contexts as well as in models of the Rasch family. In this paper, we aim to extend these tests to two-parameter item response models, with strong emphasis on pairwise maximum likelihood. The tests’ theoretical background and implementation are detailed, and the tests’ abilities to identify problematic item parameters are studied via simulation. An empirical example illustrating the tests’ use in practice is also provided.     

Testing Manifest Monotonicity Using Order-Constrained Statistical Inference

Most dichotomous item response models share the assumption of latent monotonicity, which states that the probability of a positive response to an item is a nondecreasing function of a latent variable intended to be measured. Latent monotonicity cannot be evaluated directly, but it implies manifest monotonicity across a variety of observed scores, such as the restscore, a single item score, and in some cases the total score. In this study, we show that manifest monotonicity can be tested by means of the order-constrained statistical inference framework. We propose a procedure that uses this framework to determine whether manifest monotonicity should be rejected for specific items. This approach provides a likelihood ratio test for which the p-value can be approximated through simulation. A simulation study is presented that evaluates the Type I error rate and power of the test, and the procedure is applied to empirical data.     

Hazards mean luck: Counterfactual thinking in reports of dangerous situations and careless behavior

It has been shown (Teigen, 1995) that experiences of "luck" in daily life are dependent upon the existence a worse and close hypothetical (counterfactual) outcome, rather than upon a positive evaluation of what actually happened. The present investigation focuses on the inverse relationship, namely whether a situation with a negative outcome close at hand will be perceived as lucky. To test this hypothesis, students were asked to describe dangerous situations (Experiment 1) and examples of careless behavior (Experiment 3) from their own lives, which subsequently were rated by the actors and by peer groups for good and bad luck, attractiveness, and for closeness and attractiveness of the counterfactual outcome. Dangerous situations and episodes involving careless behavior were generally regarded as more lucky than unlucky. Furthermore, degree of good luck was positively correlated with degree of dangerousness and with degree of carelessness. Luck was related to closeness, aversiveness, and (in Experiment 2) to estimated probability of the counterfactual outcome. It is concluded that luck is primarily determined by negative outcomes that did not happen, and thus a frequent by-product of risk taking and risk exposure.     

Matrices with a given number of violations of Regular Minimality

A row (or column) of an n×n matrix complies with Regular Minimality (RM) if it has a unique minimum entry which is also a unique minimum entry in its column (respectively, row). The number of violations of RM in a matrix is defined as the number of rows (equivalently, columns) that do not comply with RM. We derive a formula for the proportion of n×n matrices with a given number of violations of RM among all n×n matrices with no tied entries. The proportion of matrices with no more than a given number of violations can be treated as the p-value of a permutation test whose null hypothesis states that all permutations of the entries of a matrix without ties are equiprobable, and the alternative hypothesis states that RM violations occur with lower probability than predicted by the null hypothesis. A matrix with ties is treated as being represented by all matrices without ties that have the same set of strict inequalities among their entries.     

A simple derivation of Prelec's probability weighting function

Since Kahneman and Tversky [(1979). Prospect theory: An analysis of decision under risk. Econometrica, 47, 263-291], it has been generally recognized that decision makers overweight low probabilities. Of the several weighting functions that have been proposed, that of Prelec [(1998). The probability weighting function. Econometrica, 60, 497-528] has the attractions that it is parsimonious, consistent with much of the available empirical evidence and has an axiomatic foundation. Luce [(2001). Reduction invariance and Prelec's weighting functions. Journal of Mathematical Psychology, 45, 167-179] provided a simpler derivation based on reduction invariance, rather than compound invariance of Prelec [(1998). The probability weighting function. Econometrica, 60, 497-528]. This note introduces a behavioral assumption that we call power invariance and provides a simple derivation of Prelec's function. Thus, we have three a priori different behavioral assumptions all leading to Prelec's function.     

Bayesian tests of measurement invariance

Random item effects models provide a natural framework for the exploration of violations of measurement invariance without the need for anchor items. Within the random item effects modelling framework, Bayesian tests (Bayes factor, deviance information criterion) are proposed which enable multiple marginal invariance hypotheses to be tested simultaneously. The performance of the tests is evaluated with a simulation study which shows that the tests have high power and low Type I error rate. Data from the European Social Survey are used to test for measurement invariance of attitude towards immigrant items and to show that background information can be used to explain cross‐national variation in item functioning.     

Comment on Bernbach's prediction of the invariance of Type 2 d′ in confidence-rated recall

Bernbach Bernbach, 1967, Bernbach, 1971 has claimed that empirical findings of Type 2 d′ invariance in confidence-rated recall experiments confirm a prediction of his two-state theory of recognition and disprove strength theories of memory. In order to derive this prediction from his recognition theory, Bernbach (1967) found it necessary to add to the theory an assumption concerning recall processes. However, this recall assumption is untenable, as it leads to an empirically false prediction concerning accuracy on recognition and recall tests. Moreover, an alternative recall assumption proposed by Bernbach and Kupchak (1972) does not lead to a prediction of Type 2 d′ invariance. Thus, Bernbach's theory predicts the invariance of Type 2 d′ only with the aid of an untenable recall assumption. Consequently, empirical findings of Type 2 d′ invariance cannot be regarded as supportive of Bernbach's theory.     

A dual-process account of digit invariance learning

Performance in the McGeorge and Burton (1990) digit invariance task was originally thought to be mediated by unconscious abstraction of a “rule” that identified the invariant feature across all study items. Subsequent explanations have suggested explicit strategy use or similarity-to-exemplar matching rather than abstraction. This paper presents data that suggest that both similarity and abstraction can be used under different task demands. Delay between study and test afforded abstraction of the invariant knowledge whereas reducing the pool of study exemplars enhanced responding based on specific similarity. These results parallel effects found in the categorization literature. Rule abstraction in this sense may be due to statistical learning of feature frequency rather than abstraction of a central tendency or a complex/conceptual rule. Categorizing responses into subjective memory states (remember/know/guess) demonstrates that neither the similarity matching nor the abstraction mechanism uses information from episodic memory. Confidence measures show that participants are more confident of responses when the prototypical representation is used but not specific similarity. Taken together, these data suggest that abstracted knowledge is not held consciously but that participants have meta-awareness of when they are using the abstracted representation.     

Biased belief in the Bayesian brain: A deeper look at the evidence

A recent critique of hierarchical Bayesian models of delusion argues that, contrary to a key assumption of these models, belief formation in the healthy (i.e., neurotypical) mind is manifestly non-Bayesian. Here we provide a deeper examination of the empirical evidence underlying this critique. We argue that this evidence does not convincingly refute the assumption that belief formation in the neurotypical mind approximates Bayesian inference. Our argument rests on two key points. First, evidence that purports to reveal the most damning violation of Bayesian updating in human belief formation is counterweighted by substantial evidence that indicates such violations are the rare exception—not a common occurrence. Second, the remaining evidence does not demonstrate convincing violations of Bayesian inference in human belief updating; primarily because this evidence derives from study designs that produce results that are not obviously inconsistent with Bayesian principles.     

Knowledge space theory, formal concept analysis, and computerized psychological assessment

In the present study, the use of knowledge space theory (KST), jointly with formal concept analysis (FCA), is proposed for developing a formal representation of the relations between the items of a questionnaire and a set of psychodiagnostic criteria. This formal representation can be used to develop an efficient adaptive tool for psychological assessment. Rusch and Wille (1996) have shown some interesting connections between KST and FCA; these connections are applied in the construction of knowledge structures, starting from a formal context representing the relations between items and criteria. The proposed general methodology was applied, as an example, to the Maudsley Obsessional-Compulsive Questionnaire. We used a data set provided by a sample of patients with a diagnosis of obsessive-compulsive disorder to validate the obtained structures. The parameters of the basic local independence model (BLIM) were estimated for the obtained knowledge structures. The fit of each model was tested by parametric bootstrap because of the sparseness of the derived data matrix. The results are discussed in light of both psychological and methodological relapses. In particular, we propose a reinterpretation of the BLIM parameters that seems suitable for testing reliability and construct validity; furthermore, it is pointed out how the obtained structures could represent the starting point for the development of a computerized assessment tool.     

Updating beliefs in light of uncertain evidence: Descriptive assessment of Jeffrey's rule

Jeffrey (1983) proposed a generalisation of conditioning as a means of updating probability distributions when new evidence drives no event to certainty. His rule requires the stability of certain conditional probabilities through time. We tested this assumption (“invariance”) from the psychological point of view. In Experiment 1 participants offered probability estimates for events in Jeffrey's candlelight example. Two further scenarios were investigated in Experiment 2, one in which invariance seems justified, the other in which it does not. Results were in rough conformity to Jeffrey's (1983) principle.     

A dual-process account of digit invariance learning

Performance in the McGeorge and Burton (1990) digit invariance task was originally thought to be mediated by unconscious abstraction of a “rule” that identified the invariant feature across all study items. Subsequent explanations have suggested explicit strategy use or similarity-to-exemplar matching rather than abstraction. This paper presents data that suggest that both similarity and abstraction can be used under different task demands. Delay between study and test afforded abstraction of the invariant knowledge whereas reducing the pool of study exemplars enhanced responding based on specific similarity. These results parallel effects found in the categorization literature. Rule abstraction in this sense may be due to statistical learning of feature frequency rather than abstraction of a central tendency or a complex/conceptual rule. Categorizing responses into subjective memory states (remember/know/guess) demonstrates that neither the similarity matching nor the abstraction mechanism uses information from episodic memory. Confidence measures show that participants are more confident of responses when the prototypical representation is used but not specific similarity. Taken together, these data suggest that abstracted knowledge is not held consciously but that participants have meta-awareness of when they are using the abstracted representation.     

On Guessing: An Alternative Adjusted Positive Learning Estimator and Comparing Probability Misspecification With Monte Carlo Simulations

Practitioners in the sciences have used the “flow” of knowledge (post-test score minus pre-test score) to measure learning in the classroom for the past 50 years. Walstad and Wagner, and Smith and Wagner moved this practice forward by disaggregating the flow of knowledge and accounting for student guessing. These estimates are sensitive to misspecification of the probability of guessing correct. This work provides guidance to practitioners and researchers facing this problem. We introduce a transformed measure of true positive learning that under some knowable conditions performs better when students’ ability to guess correctly is misspecified and converges to Hake’s normalized learning gain estimator under certain conditions. We then use simulations to compare the accuracy of two estimation techniques under various violations of the assumptions of those techniques. Using recursive partitioning trees fitted to our simulation results, we provide the practitioner concrete guidance based on a set of yes/no questions.