Test theory without an answer key

A general model is presented for homogeneous, dichotomous items when the answer key is not known a priori. The model is structurally related to the two-class latent structure model with the roles of respondents and items interchanged. For very small sets of respondents, iterative maximum likelihood estimates of the parameters can be obtained by existing methods. For other situations, new estimation methods are developed and assessed with Monte Carlo data. The answer key can be accurately reconstructed with relatively small sets of respondents. The model is useful when a researcher wants to study objectively the knowledge possessed by members of a culturally coherent group that the researcher is not a member of.This research was supported by NSF Grant No. SES-8320173 to the authors. We gratefully acknowledge comments and suggestions from John Boyd, Tarow Indow, and Kathy Maher as well as the editor and several anonymous referees.     

Functional measurement approach to self-estimation in multiattribute evaluation

Multiattribute analysis depends on measurement of values and weights. Unless these measures reflect the decision maker's true values and weights, the multiattribute formula may put a less-preferred alternative in first place. To avoid such disordinality requires stringent measurement conditions: First, the values and weights must be on linear (equal interval) or ratio (known zero) scales. Second, these scales must satisfy a condition of common unit across disparate attribute dimensions. Most methods of range adjustment beg both of these measurement questions. Functional measurement theory can solve both problems and so can be useful in multiattribute analysis. Past work has established the operation of a general cognitive algebra as an empirical reality. The averaging model, in particular, makes possible the definition and estimation of weights and values as distinct psychological parameters. It can also solve the problem of common unit. Cognitive algebra thus provides a grounded theoretical foundation on which to develop self-estimation methodology, in which decision makers provide direct estimates of their values and weights. The logic is straightforward. Functional measurement can analyze global judgments to obtain validated psychological scales. These scales may then be used as validational criteria for the self-estimates. Procedures to eliminate biases in the self-estimates can thus be tested and refined in well-learned multiattribute tasks, such as judgments of meals, in which global judgments are trustworthy. Once developed, such self-estimation procedures may be used with some confidence for general multiattribute analysis. A number of studies from 20-odd years of work on the theory of information integration are summarized to show good, although not unmixed promise for self-estimation.     

Effectiveness of the GOFER course in decision making for high school students

The paper describes and evaluates the GOFER course in decision making for high school students (Mann, Harmoni and Power, 1988). The course was based on principles from the conflict theory of decision making (Janis and Mann, 1977) and was designed to provide adolescents with an understanding of factors that produce good and poor decision making as well as knowledge and practice of sound decision skills. Two evaluation studies were conducted to examine effectiveness of the course. Study 1 was conducted with young adolescents (12 year olds). It found a significant difference between students trained in the course and a control group on measures of self-esteem as a decision maker, self reported decision habits and knowledge of decision strategy. Study 2 was conducted with mid-adolescents (15 year olds). It, too, found a significant difference between students who took the course and a control group on measures of self esteem as a decision maker and self-reported decision habits. It is concluded that while the evaluation study fell short of an ideal test, the GOFER course met the criteria of improving student knowledge, raising confidence in decision making and changing self reported decision habits.     

An automated test development of parallel tests from a seed test

Binary programming models are presented to generate parallel tests from an itembank. The parallel tests are created to match item for item an existing seed test and match user supplied taxonomic specifications. The taxonomic specifications may be either obtained from the seed test or from some other user requirement. An algorithm is presented along with computational results to indicate the overall efficiency of the process. Empirical findings based on an itembank for the Arithmetic Reasoning section of the Armed Services Vocational Aptitude Battery are given.The Office of Naval Research, Program in Cognitive Science, N00014-87-C-0696 partially supported the work of Douglas H. Jones. The Rutgers Research Resource Committee of the Graduate School of Management partially supported the work of Douglas H. Jones and Ing-Long Wu. A Thomas and Betts research fellowship partially supported the work of Ing-Long Wu. The Human Resources Laboratory, United States Air Force, partially supported the work of Ronald Armstrong. The authors benefited from conversations with Dr. Wayne Shore, Operational Technologies, San Antonio, Texas. The order of authors' names is alphabetical and denotes equal authorship.     

Provision of step-level public goods with continuous contribution

Twelve groups of five subjects each participated in a nonco-operative game in which each member of a group receives the same endowment and must then decide independently and anonymously how much of it to contribute to the group benefit. Regardless of the size of his or her contribution, each member receives the same reward if, and only if, the sum of contributions is equal to or larger than a prespecified provision threshold. The results show that the level of contribution depends on the provision threshold, and that it increases when contributions are not restricted to be all-or-none. We present, discuss, and competitively test two models for this class of social dilemmas, one postulating maximization of expected utility and the other yielding an equitable solution.     

Theory change in immunology part II: The clonal selection theory

This two-part article examines the competition between the clonal selection theory and the instructive theory of the immune response from 1957–1967. In Part I the concept of a temporally extended theory is introduced, which requires attention to the hitherto largely ignored issue of theory individuation. Factors which influence the acceptability of such an extended theory at different temporal points are also embedded in a Bayesian framework, which is shown to provide a rational account of belief change in science. In Part II these factors, as elaborated in the Bayesian framework, are applied to the case of the success of the clonal selection theory and the failure of the instructive theory.     

Ordinal consistency and ordinal true scores

This paper argues that test data are ordinal, that latent trait scores are only determined ordinally, and that test data are used largely for ordinal purposes. Therefore it is desirable to develop a test theory based only on ordinal assumptions. A set of ordinal assumptions is presented, including an ordinal version of local independence. From these assumptions it is first shown that the gamma-correlation between two tests is the product of their gamma-correlations with the true latent order. The theory is generalized to allow for heterogeneous tests by defining a weighted average local independence. The tau-correlations between total score and the latent order can be found in both homogeneous and heterogeneous cases, and a system of differential item weighting to maximize the tau-correlation between weighted items and the latent order is provided. Thus a purely ordinal test theory seems possible.Part of this work was done while the author was a Visiting Fellow at Macquarrie University. The paper has benefitted from discussions with Professors Thomas J. Reynolds and Roderick P. McDonald and from the comments of several anonymous reviewers.     

Criterion-related construct validity

Established results on latent variable models are applied to the study of the validity of a psychological test. When the test predicts a criterion by measuring a unidimensional latent construct, not only must the total score predict the criterion, but the joint distribution of criterion scores and item responses must exhibit a certain pattern. The presence of this population pattern may be tested with sample data using the stratified Wilcoxon rank sum test. Often, criterion information is available only for selected examinees, for instance, those who are admitted or hired. Three cases are discussed: (i) selection at random, (ii) selection based on the current test, and (iii) selection based on other measures of the latent construct. Discriminant validity is also discussed.This work was supported in part by Grant SES-87-01890 from the Measurement Methods and Data Improvement Program of the U.S. National Science Foundation.     

Influential observations in principal factor analysis

We propose a method for detecting influential observations in iterative principal factor analysis. For this purpose we derive the influence functionsI(x; LL ^T ) andI(x; ) for the common variance matrixT =LL ^T and the unique variance matrix , respectively, in the common factor decomposition =LL ^T + . A numerical example is given for illustration.The authors are grateful to Tomoyuki Tarumi and Atsuhiro Hayashi for their kind permission to use their software Seto/B for drawing Figures 1 and 2 and to anonymous reviewers for comments on the paper.