Test theory without an answer key

A general model is presented for homogeneous, dichotomous items when the answer key is not known a priori. The model is structurally related to the two-class latent structure model with the roles of respondents and items interchanged. For very small sets of respondents, iterative maximum likelihood estimates of the parameters can be obtained by existing methods. For other situations, new estimation methods are developed and assessed with Monte Carlo data. The answer key can be accurately reconstructed with relatively small sets of respondents. The model is useful when a researcher wants to study objectively the knowledge possessed by members of a culturally coherent group that the researcher is not a member of.This research was supported by NSF Grant No. SES-8320173 to the authors. We gratefully acknowledge comments and suggestions from John Boyd, Tarow Indow, and Kathy Maher as well as the editor and several anonymous referees.     

Computational behavior dynamics: an alternative description of Nevin (1969).

A computational processing behavior-dynamic model was instantiated in the form of a computer program that "behaved" on the task developed by Nevin (1969). In this classic discrete-trials experiment, the relative frequency of choosing a response alternative matched the relative frequency of reinforcement for that alternative, the local structure of responding was opposite that predicted by momentary maximizing (i.e., the probability of a changeover decreased with run length), and absolute and relative response rates varied independently. The behavior-dynamic model developed here qualitatively reproduced these three results (but not in quantitative and specific detail) and also generated some interesting, as-yet-untested predictions about performance in Nevin's task. The model was discussed as an example of a stochastic behavior-dynamic alternative to algebraic behavior theory.     

Evaluating young children''s cognitive capacities through computer versus hand drawings

Young normal and handicapped children, aged 3 to 6 years, were taught to draw a scene of a house, garden and a sky with a computer drawing program that uses icons and is operated by a mouse. The drawings were rated by a team of experts on a 7-category scale. The children's computer- and hand-produced drawings were compared with one another and with results on cognitive, visual and fine motor tests. The computer drawing program made it possible for the children to accurately draw closed shapes, to get instant feedback on the adequacy of the drawing, and to make corrections with ease. It was hypothesized that these features would compensate for the young children's limitations in such cognitive skills, as memory, concentration, planning and accomplishment, as well as their weak motor skills. In addition, it was hypothesized that traditional cognitive ratings of hand drawings may underestimate young children's intellectual ability, because drawing by hand demands motor skills and memory, concentration and planning skills that are more developed than that actually shown by young children. To test the latter hypothesis, the children completed a training program in using a computer to make drawings. The results show that cognitive processes such as planning, analysis and synthesis can be investigated by means of a computer drawing program in a way not possible using traditional pencil and paper drawings. It can be said that the method used here made it possible to measure cognitive abilities "under the floor" of what is ordinarily possible by means of traditionally hand drawings.     

An automated test development of parallel tests from a seed test

Binary programming models are presented to generate parallel tests from an itembank. The parallel tests are created to match item for item an existing seed test and match user supplied taxonomic specifications. The taxonomic specifications may be either obtained from the seed test or from some other user requirement. An algorithm is presented along with computational results to indicate the overall efficiency of the process. Empirical findings based on an itembank for the Arithmetic Reasoning section of the Armed Services Vocational Aptitude Battery are given.The Office of Naval Research, Program in Cognitive Science, N00014-87-C-0696 partially supported the work of Douglas H. Jones. The Rutgers Research Resource Committee of the Graduate School of Management partially supported the work of Douglas H. Jones and Ing-Long Wu. A Thomas and Betts research fellowship partially supported the work of Ing-Long Wu. The Human Resources Laboratory, United States Air Force, partially supported the work of Ronald Armstrong. The authors benefited from conversations with Dr. Wayne Shore, Operational Technologies, San Antonio, Texas. The order of authors' names is alphabetical and denotes equal authorship.     

The accuracy of maternal self-reports: Agreement between reports on a computer simulation compared with observed behaviour in the supermarket

The validity of maternal self-reports about their parenting behaviour is evaluated with two studies. In both studies, the mothers were observed shopping in the supermarket with their 2-year-old children. One week later they responded to multiple choice questions on a computer that simulated the experience of shopping with their children. The exact agreement between the two types of data was then assessed. The results from the first study involving 28 mothers revealed some agreement, although it was generally low. In an effort to increase the level of agreement, the computer simulation was revised to allow multiple responses and to clarify some of the simulation vigenettes. A replication study resulted in a considerably higher level of agreement for 22 mothers. Implications of this method for researchers and practitioners are discussed.     

Ordinal consistency and ordinal true scores

This paper argues that test data are ordinal, that latent trait scores are only determined ordinally, and that test data are used largely for ordinal purposes. Therefore it is desirable to develop a test theory based only on ordinal assumptions. A set of ordinal assumptions is presented, including an ordinal version of local independence. From these assumptions it is first shown that the gamma-correlation between two tests is the product of their gamma-correlations with the true latent order. The theory is generalized to allow for heterogeneous tests by defining a weighted average local independence. The tau-correlations between total score and the latent order can be found in both homogeneous and heterogeneous cases, and a system of differential item weighting to maximize the tau-correlation between weighted items and the latent order is provided. Thus a purely ordinal test theory seems possible.Part of this work was done while the author was a Visiting Fellow at Macquarrie University. The paper has benefitted from discussions with Professors Thomas J. Reynolds and Roderick P. McDonald and from the comments of several anonymous reviewers.     

An infrared system for the detection of a pigeon''s pecks at alphanumeric characters on a TV screen: the dependency of letter detection on the predictability of one letter by another.

Three pigeons pecked at letters of the alphabet and at the symbol "?" displayed on a computer-driven cathode ray screen. A 4 by 4 matrix of infrared emitting and detecting diodes and associated circuitry identified the location of a pigeon''s responses to the screen. Responses at the target letter T were probabilistically reinforced with food whenever T appeared in a string of three letters in the middle of the screen. Responses at the symbol "?" appearing below this string were probabilistically reinforced whenever T did not appear. The letter F anywhere in the three-character string either strongly predicted the occurrence of the target letter T, in two conditions, or predicted its nonoccurrence, in a third. This manipulation of the frequency with which the familiar letter F predicted T was shown to change the function relating probability of a correct peck at the symbol "?" to the number of Fs in the string. This effect may be interpreted as an instance of the phenomenon where an organism''s acquired knowledge changes what it sees.     

Multifactor measurement of bodily feelings: Conceptual development,scale construction,and empirical validation

A series of studies is described demonstrating a coherent exploratory and confirmatory factor-analytic approach to the scaling of self-reported bodily feelings. This approach attempts to optimize the orthogonality, parsimony, and meaningfulness of a multifactor inventory. Psychometric properties of the resulting 35-item Bodily Feeling Scale (BFS) are presented and its comparability with other inventories is discussed. A validity study using item-content-based scales of the Minnesota Multiphasic Personality Inventory showed both concurrent and discriminative validity of the BFS. Implications for clinical and research utility are discussed.Portions of this research were conducted while the first author was supported by a University Graduate Fellowship awarded by the Graduate School of Arts and Sciences at Vanderbilt University.Professor Nunnally passed away in August 1982. His major contributions to the several stages of this research program are recognized through coauthorship.     

A nonparametric approach for assessing latent trait unidimensionality

Assuming a nonparametric family of item response theory models, a theory-based procedure for testing the hypothesis of unidimensionality of the latent space is proposed. The asymptotic distribution of the test statistic is derived assuming unidimensionality, thereby establishing an asymptotically valid statistical test of the unidimensionality of the latent trait. Based upon a new notion of dimensionality, the test is shown to have asymptotic power 1. A 6300 trial Monte Carlo study using published item parameter estimates of widely used standardized tests indicates conservative adherence to the nominal level of significance and statistical power averaging 81 out of 100 rejections for examinee sample sizes and psychological test lengths often incurred in practice.The referees' comments were remarkably detailed and greatly enhanced the writeup and sensitized the author to certain pertinent issues. Discussions with Fritz Drasgow, Lloyd Humphreys, Dennis Jennings, Brian Junker, Robert Linn, Ratna Nandakumar, and Robin Shealy were also very useful.This research was supported by the Office of Naval Research under grant N00014-84-K-0186; NR 150-533, and by the National Science Foundation under grant DMS 85-03321.