Test theory without an answer key

A general model is presented for homogeneous, dichotomous items when the answer key is not known a priori. The model is structurally related to the two-class latent structure model with the roles of respondents and items interchanged. For very small sets of respondents, iterative maximum likelihood estimates of the parameters can be obtained by existing methods. For other situations, new estimation methods are developed and assessed with Monte Carlo data. The answer key can be accurately reconstructed with relatively small sets of respondents. The model is useful when a researcher wants to study objectively the knowledge possessed by members of a culturally coherent group that the researcher is not a member of.This research was supported by NSF Grant No. SES-8320173 to the authors. We gratefully acknowledge comments and suggestions from John Boyd, Tarow Indow, and Kathy Maher as well as the editor and several anonymous referees.     

An availability bias in professional judgment

When branches of a fault tree are pruned, subjects do not fully transfer the probability of those branches to the ‘all other’ category. This underestimation of the catch-all probability has been interpreted as an ‘out of sight, out of mind’ form of the availability bias. The present work replicates this underestimation bias with professional managers. It then demonstrates the effectiveness of a corrective tactic, extending the tree by generating additional causes, and also reveals that more easily retrieved short-term causes dominate the generation process. These results do not differ across managers' culture, education or experience. After evaluating such alternative explanations as category redefinition, we conclude that availability is a major cause, though possibly not the sole cause, of the underestimation bias.     

Examining differential item functioning due to item difficulty and alternative attractiveness

A method for analyzing test item responses is proposed to examine differential item functioning (DIF) in multiple-choice items through a combination of the usual notion of DIF, for correct/incorrect responses and information about DIF contained in each of the alternatives. The proposed method uses incomplete latent class models to examine whether DIF is caused by the attractiveness of the alternatives, difficulty of the item, or both. DIF with respect to either known or unknown subgroups can be tested by a likelihood ratio test that is asymptotically distributed as a chi-square random variable.     

Criterion-related construct validity

Established results on latent variable models are applied to the study of the validity of a psychological test. When the test predicts a criterion by measuring a unidimensional latent construct, not only must the total score predict the criterion, but the joint distribution of criterion scores and item responses must exhibit a certain pattern. The presence of this population pattern may be tested with sample data using the stratified Wilcoxon rank sum test. Often, criterion information is available only for selected examinees, for instance, those who are admitted or hired. Three cases are discussed: (i) selection at random, (ii) selection based on the current test, and (iii) selection based on other measures of the latent construct. Discriminant validity is also discussed.This work was supported in part by Grant SES-87-01890 from the Measurement Methods and Data Improvement Program of the U.S. National Science Foundation.     

A quasi-nonmetric method for multidimensional scaling VIA an extended euclidean model

An Extended Two-Way Euclidean Multidimensional Scaling (MDS) model which assumes both common and specific dimensions is described and contrasted with the standard (Two-Way) MDS model. In this Extended Two-Way Euclidean model then stimuli (or other objects) are assumed to be characterized by coordinates onR common dimensions. In addition each stimulus is assumed to have a dimension (or dimensions) specific to it alone. The overall distance between objecti and objectj then is defined as the square root of the ordinary squared Euclidean distance plus terms denoting the specificity of each object. The specificity,s _j , can be thought of as the sum of squares of coordinates on those dimensions specific to objecti, all of which have nonzero coordinatesonly for objecti. (In practice, we may think of there being just one such specific dimension for each object, as this situation is mathematically indistinguishable from the case in which there are more than one.)We further assume that _ij =F(d _ij ) +e _ij where _ij is the proximity value (e.g., similarity or dissimilarity) of objectsi andj,d _ij is the extended Euclidean distance defined above, whilee _ij is an error term assumed i.i.d.N(0, ²).F is assumed either a linear function (in the metric case) or a monotone spline of specified form (in the quasi-nonmetric case). A numerical procedure alternating a modified Newton-Raphson algorithm with an algorithm for fitting an optimal monotone spline (or linear function) is used to secure maximum likelihood estimates of the paramstatistics) can be used to test hypotheses about the number of common dimensions, and/or the existence of specific (in addition toR common) dimensions.This approach is illustrated with applications to both artificial data and real data on judged similarity of nations.     

Nonconvergence,improper solutions,and starting values in lisrel maximum likelihood estimation

In the framework of a robustness study on maximum likelihood estimation with LISREL three types of problems are dealt with: nonconvergence, improper solutions, and choice of starting values. The purpose of the paper is to illustrate why and to what extent these problems are of importance for users of LISREL. The ways in which these issues may affect the design and conclusions of robustness research is also discussed.     

Iterative least squares estimates of communality: Initial estimate need not affect stabilized value

A common criticism of iterative least squares estimates of communality is that method of initial estimation may influence stabilized values. As little systematic research on this topic has been performed, the criticism appears to be based on cumulated experience with empirical data sets. In the present paper, two studies are reported in which four types of initial estimate (unities, squared multiple correlations, highestr, and zeroes) and four levels of convergence criterion were employed using four widely available computer packages (BMDP, SAS, SPSS, and SOUPAC). The results suggest that initial estimates have no effect on stabilized communality estimates when a stringent criterion for convergence is used, whereas initial estimates appear to affect stabilized values employing rather gross convergence criteria. There were no differences among the four computer packages for matrices without Heywood cases.     

Statistical consistency and hypothesis testing for nonmetric multidimensional scaling

The properties of nonmetric multidimensional scaling (NMDS) are explored by specifying statistical models, proving statistical consistency, and developing hypothesis testing procedures. Statistical models with errors in the dependent and independent variables are described for quantitative and qualitative data. For these models, statistical consistency often depends crucially upon how error enters the model and how data are collected and summarized (e.g., by means, medians, or rank statistics). A maximum likelihood estimator for NMDS is developed, and its relationship to the standard Shepard-Kruskal estimation method is described. This maximum likelihood framework is used to develop a method for testing the overall fit of the model.     

A structural model for postpartum responses to the somatic and cognitive items on the Beck Depression Inventory

A distinction can be drawn between those items on the Beck Depression Inventory (BDI) that reflect mainly cognitive and emotional symptoms and those that reflect somatic symptoms. Responses to the BDI were obtained from women at 3, 6, 9, and 24 weeks postpartum. The covariances of cognitive and somatic symptom scores across the four time points were closely fit by a structural equations model referred to as the simplex. The modeling indicates that earlier cognitive scores directly predict later somatic scores but that early somatic scores do not directly predict later cognitive scores. Other data confirm the results.Much of this research was supported while the first author was an NIMH Fellow at the University of Iowa (MH15168).     

A nonparametric approach for assessing latent trait unidimensionality

Assuming a nonparametric family of item response theory models, a theory-based procedure for testing the hypothesis of unidimensionality of the latent space is proposed. The asymptotic distribution of the test statistic is derived assuming unidimensionality, thereby establishing an asymptotically valid statistical test of the unidimensionality of the latent trait. Based upon a new notion of dimensionality, the test is shown to have asymptotic power 1. A 6300 trial Monte Carlo study using published item parameter estimates of widely used standardized tests indicates conservative adherence to the nominal level of significance and statistical power averaging 81 out of 100 rejections for examinee sample sizes and psychological test lengths often incurred in practice.The referees' comments were remarkably detailed and greatly enhanced the writeup and sensitized the author to certain pertinent issues. Discussions with Fritz Drasgow, Lloyd Humphreys, Dennis Jennings, Brian Junker, Robert Linn, Ratna Nandakumar, and Robin Shealy were also very useful.This research was supported by the Office of Naval Research under grant N00014-84-K-0186; NR 150-533, and by the National Science Foundation under grant DMS 85-03321.