The Specificity of Behaviors and Measurements

This essay examines the fundamental fact of specificity in behaviors both outside and inside the testing room. Two tests of the same construct typically have a moderate to low correlation between them, the level of relationship depending on whether the mode, the situation, the task, and the stimuli for one test are similar to or different from those for the other. The great majority of psychological tests have considerable specificity. Their scores tend to correlate far less than perfectly with other tests of the same variable. Even when an ability is defined within a model such as Guilford's Structure-of-Intellect, each test of that ability will usually have some specific determinants not shared by other tests of that aspect of intelligence. Moreover, such a test is likely to correlate almost as well with tests of other similar abilities as with tests of its particular one.     

Relationships Among Selected Tests of Spatial Orientation Ability

The purpose of this study was to investigate the relationships between selected tests of spatial orientation ability. 2 tests which have been used as predictors of spatial ability (Guilford-Zimmerman Aptitude Survey, Parts V and VI) plus 2 newly developed tests designed to measure that ability were given to 202 Junior high school boys aged 11-15 yr. 3 of these tests were of the paper and pencil type while one was a physical performance (tumbling) test. The paper-and-pencil tests correlated significantly with each other (.62, .61, .44) but the physical performance test did not correlate significantly with any of the other tests.     

The relationship of specific mental ability measures compared to a general mental ability measure to quality and quantity performance on a clerical job sample

Barrett (1993) presented evidence that mental ability measures designed specifically for the prediction of performance in tasks and jobs have validity at least equal to that of tests of general mental ability, while having advantages such as lower adverse impact. The current study demonstrates that the match in specificity of predictor and criterion can be attained through simple changes in the scoring of standard mental and clerical ability tests. A sample of 51 university students completed a battery of pre-employment tests and, a week or two later, worked on a clerical job sample. The results indicated that quality of performance (number of errors on the task) was best predicted using the number of errors made on the predictor tests while quantity of work was only predicted by the number correct. A test of general mental ability did not correlate with either criterion. These results contradicted the conclusions of Schmidt (1993) that specific ability tests have no incremental validity above a general ability test.Ralph Alexander died before this report was complete. Consequently, the conclusions are those of the first two authors.     

A comparison of paper-and-pencil and computerized forms of Line Orientation and Enhanced Cued Recall Tests

The purpose of this study was to assess whether a computerized battery of neuropsychological tests could produce similar results as the conventional forms. Comparisons on 77 volunteer undergraduates were carried out with two neuropsychological tests: Line Orientation Test and Enhanced Cued Recall Test. Firstly, students were assigned randomly across the test medium (paper-and-pencil versus computerized). Secondly, the groups were given the same test in the other medium after a 30-day interval between tests. Results showed that the Enhanced Cued Recall Test-Computer-based did not correlate with the Enhanced Cued Recall Test-Paper-and-pencil results. Line Orientation Test-Computer-based scores, on the other hand, did correlate significantly with the Line Orientation Test-Paper-and-pencil version. In both tests, scores were higher on paper-and-pencil tests compared to computer-based tests. Total score difference between modalities was statistically significant for both Enhanced Cued Recall Tests and for the Line Orientation Test. In both computer-based tests, it took less time for participants to complete the tests.     

The Role of Cognitive Ability Tests in Employment Selection

Cognitive ability tests correlate with measures of job performance across many jobs. However, cognitive ability tests produce racial differences that are 3 to 5 times larger than other predictors-such as biodata, personality inventories, and the structured interview-that are valid predictors of job performance. Given that (a) cognitive ability tests can be combined with other predictors such that adverse impact is reduced while overall validity is increased, and (b) alternative predictors with less adverse impact can produce validity coefficients comparable to those obtained with cognitive ability tests alone, sole reliance on cognitive ability tests when alternatives are available is unwarranted.     

A general scheme for the analytic decomposition of objective test scores: Illustrative demonstrations using the Rod-and-Frame test and the Müller-Lyer illusion

Objective tests of personality typically include a number of items or trials; the total score on the test is the sum of the subject's “correct” responses across all such trials. Normally, the trials are varied systematically across various facets of the test design, so that the total score represents a composite measure of accuracy averaged across these test facets. However, since only one score is computed for each subject, some potentially important kinds of individual differences—namely all those associated with each particular variation in the test design—are treated solely as measurement unreliability. Such a psychometric stance may serve to obscure more differentiated types of individual differences, with the result that composite scores from trials based on one type of experimental design may not be highly related to such scores from trials using a somewhat different design. The present paper presents a general procedure for scoring objective tests more analytically. To illustrate this general rationale, and to demonstrate its potential utility, data have been reanalyzed from two previous studies, one using the Rod-and-Frame test, the other the Müller-Lyer illusion. In both cases, the traditional global accuracy score did not correlate significantly with other theoretically related variables, while a number of component scores were quite highly related.     

Size of the neocerebellar vermis is associated with cognition in healthy elderly men

Cerebellar volumes show small positive correlations with cognitive ability in young adults, but no studies have examined this relationship in older adults. Furthermore, no studies have examined relationships between sizes of subareas of the cerebellum and cognitive ability. We hypothesized that size of the two areas of the neocerebellar vermis would correlate with a battery of eight cognitive tests in 50 men aged 65-70. Size of the neocerebellar areas of the vermis correlated positively with several cognitive tests (r's=.29-.37, p<.05), whilst sizes of other parts of the vermis did not correlate with any cognitive tests. Total cerebellar volumes correlated significantly with a test of nonverbal reasoning (r=.030, p=.42). These findings suggest a specific association of neocerebellar vermis size with variations in cognitive ability in older adults.     

FURTHER STUDIES OF SITUATIONAL TESTS

In the present paper, results are reported for two different situational judgment tests (SJT) used in validation studies with almost 4,000 employees in 7 different organizations. Across the 2 studies, it was shown that situational test scores were significantly related to performance (weighted average r = .19), cognitive ability (weighted average r = .45), and experience (weighted average r = .20). In one study, there was a slight tendency for experience and cognitive ability to interact in the prediction of situational judgment, such that cognitive ability became less predictive as experience increased. Situational judgment fully mediated the effects of cognitive ability in one study, but not in the other. Finally, SJT race effect sizes were consistent with past research and, while not trivial, smaller than those typically observed for cognitive ability tests. The studies yielded very similar results despite the use of different keying methods (empirical vs. subject matter expert opinion). The preponderance of the evidence indicates that situational judgment measures mediate avariety of job relevant skills. Limitations to these studies and directions for future research are reviewed.     

Stimulus control of habits: Evidence for both stimulus specificity and devaluation insensitivity in a dual-response task

Goal-directed and habitual actions are clearly defined by their associative relations. Whereas goal-directed control can be confirmed via tests of outcome devaluation and contingency-degradation sensitivity, a comparable criterion for positively detecting habits has not been established. To confirm habitual responding, a test of control by the stimulus–response association is required while also ruling out goal-directed control. Here we describe an approach to developing such a test in rats using two discriminative stimuli that set the occasion for two different responses that then earn the same outcome. Performance was insensitive to outcome devaluation and showed stimulus–response specificity, indicative of stimulus-controlled behavior. The reliance of stimulus–response associations was further supported by a lack of sensitivity during the single extinction test session used here. These results demonstrate that two concurrently trained responses can come under habitual control when they share a common outcome. By reducing the ability of one stimulus to signal its corresponding response–outcome association, we found evidence for goal-directed control that can be dissociated from habits. Overall, these experiments provide evidence that tests assessing specific stimulus–response associations can be used to investigate habits.     

Using heteromodal communication to optimize knowledge and awareness

Accurate, error-free communication is essential for success in many areas, such as eyewitness testimony, human factors design, business, education, and personal relationships. Traditional communication uses similar modalities: Participants communicate by talking or by writing, but not both at the same time with the same addressees. New communicative technologies have broadened this vista. For example, one communicator can speak and the other can type. We tested communicative effectiveness using accuracy and error detection in a trivia recall test, evaluating the roles of presentation and retrieval modalities on reporting facts stored in long-term memory. Heteromodal communication (hearing and writing or reading and saying) was more effective than homomodal communication (hearing and saying or reading and writing), with the most correct responses and the most errors caught. This has direct connections to communicative success and applied tests of skill.     

Modeling differentiation of cognitive abilities within the higher-order factor model using moderated factor analysis

Receptive foreign language proficiency is usually measured with reading and listening comprehension tasks. A novel approach to assess such proficiencies – viewing comprehension – is based on the presentation of short instructional videos followed by one or more comprehension questions concerning the preceding video stimulus. In order to evaluate a newly developed viewing comprehension test 485 German high school students completed reading, listening, and viewing comprehension tests, all measuring the receptive proficiency in English as a foreign language. Fluid and crystallized intelligence were measured as predictors of performance. Relative to traditional comprehension tasks, the viewing comprehension task has similar psychometric qualities. The three comprehension tests are very highly but not perfectly correlated with each other. Relations with fluid and crystallized intelligence show systematic differences between the three comprehension tasks. The high overlap between foreign language comprehension measures and between crystallized intelligence and language comprehension ability can be taken as support for a uni-dimensional interpretation. Implications for the assessment of language proficiency are discussed.     

Rule learning by cotton-top tamarins

Previous work suggests that human infants are capable of rapidly generalizing patterns that have been characterized as abstract algebraic rules (Science 283 (1999) 77), a process that may play a pivotal role in language acquisition. Here we explore whether this capacity is uniquely human and evolved specifically for the computational problems associated with language, or whether this mechanism is shared with other species, and therefore evolved for problems other than language. We used the same materials and methods that were originally employed in tests of human infants to assess whether cotton-top tamarin monkeys can extract abstract algebraic rules. Specifically, we habituated subjects to sequences of consonant-vowel syllables that followed one of two patterns, AAB (e.g. wi wi di) or ABB (le we we). Following habituation, we presented subjects with two novel test items, one with the same pattern as that presented during habituation and one with a different pattern. Like human infants, tamarins were more likely to dishabituate to the test item with a different pattern. We conclude that the capacity to generalize rule-like patterns, at least at the level demonstrated, did not evolve specifically for language acquisition, though it remains possible that infants might use such rules during language acquisition.     

The end of the reading age: grade and age effects in early schooling

During the school years, psychological test norms may be indexed by age or by grade. A number of studies have shown that using age-based norms appears to produce biases associated with grade assignment. Cahan and Cohen [Child Dev. 60 (1989) 1239-1249] showed that the effect of one grade was over twice the effect of 1 year of age for most verbal cognitive ability subtests in Grades 4-6. A higher ratio might be expected for more curriculum-related areas such as reading tests. Analysis of a representative sample of 4257 Grade 1 and 2 children in Tasmanian government schools (which use relatively strict age assignment to grades) for three subtests of the Woodcock Reading Mastery Test—Revised (WRMTR): Word Identification, Word Attack, and Passage Comprehension, showed that the grade effect is about twice the age effect. This data shows that using age-based norms instead of grade-based norms for reading and other verbal ability tests may produce bias in the early school years. Psychologists should thus be primarily concerned with children's educational history, as once children have entered school other developmental factors indexed by age have less influence on children's verbal performance.     

The unique correspondence of the item response function and item category response functions in polytomously scored item response models

The item response function (IRF) for a polytomously scored item is defined as a weighted sum of the item category response functions (ICRF, the probability of getting a particular score for a randomly sampled examinee of ability ). This paper establishes the correspondence between an IRF and a unique set of ICRFs for two of the most commonly used polytomous IRT models (the partial credit models and the graded response model). Specifically, a proof of the following assertion is provided for these models: If two items have the same IRF, then they must have the same number of categories; moreover, they must consist of the same ICRFs. As a corollary, for the Rasch dichotomous model, if two tests have the same test characteristic function (TCF), then they must have the same number of items. Moreover, for each item in one of the tests, an item in the other test with an identical IRF must exist. Theoretical as well as practical implications of these results are discussed.This research was supported by Educational Testing Service Allocation Projects No. 79409 and No. 79413. The authors wish to thank John Donoghue, Ming-Mei Wang, Rebecca Zwick, and Zhiliang Ying for their useful comments and discussions. The authors also wish to thank three anonymous reviewers for their comments.     

Measures of visual-auditory integration and their relations to reading achievement in early grades.

Several tests of visual-auditory integration were designed to represent the various tasks that have been found to correlate with reading achievement in the early grades. The battery of tests was administered to about 100 first, 84 second, and 110 third graders. Several questions were posed: (1) Do the various tests of integration measure the same underlying ability or are there separate abilities that are tapped by the various tests? (2) Are the various tests correlated with reading because they tap a common integrative ability that is related to reading or are there several abilities, each related to reading independently to the others? (3) To what extent are the relationships between the integrative abilities and reading separate from the relationship between reading and IQ? Results indicated that there are separate integrative abilities, and multiple regression analyses showed that several of the integrative measures were correlated with reading independently of the other measures. Further, several of the integrative measures accounted for significant reading variance beyond that accounted for by IQ tests. The results were discussed in the context of the need to differentiate adequately between various patterns of perceptual weakness in the diagnosis and remediation of perceptually based reading difficulties.     

Examining the Relation Between Visual Imagery and Spatial Ability Tests

Research evidence indicates that self-report imagery ability is psychometrically distinct from objective, spatial test measures. One hypothesis put forward in the literature to explain this finding is that the nature of the stimulus is important. The aim of this article was to examine the relation between spatial abilities and measures of visual imagery obtained using different types of stimulus material. The main finding was that imagery tasks that required the mental synthesis or transformation of visual forms, such as alphanumeric characters and simple geometric shapes, correlated strongly with tests of spatial ability. In contrast, images of familiar items retrieved from long-term memory did not correlate with spatial test performance. It is argued that tasks that better control the stimuli imagined and the standards used to rate the quality of the image provide more objective measures of imagery ability. The implications of these findings are discussed.     

Detecting group differences in sequential association using sampled permutations: Log odds,kappa, and phi compared

When determining whether a particular transition is more characteristic of one group than of another, two things are required: an index associated with the transition of interest and a statistical test that can determine whether group membership systematically affects values for that index. Here the familiar parametrict test is compared with a test based on sampled permutations. Indices considered are the odds and log odds ratio, Yule’sQ, Wampold’s (1989) transformed kappa, and phi. The odds and log odds ratio are monotonically increasing functions of Yule’sQ and so give similar results. Yule’sQ and phi are essentially rank order invariant and usually give similar results. Transformed kappa, however, rank orders subjects somewhat differently than the others; moreover, it appears somewhat biased. With respect to the tests, when subjects are 20 or more it does not matter much whether sampled permutation or parametrict tests are used; both yield essentially the same result. However, when subjects are fewer than 20, or whenever there is any other reason to think that parametric assumptions may not be met, permutation tests are recommended. A computer program that effects such tests is described.     

Predicting Training Success with General Mental Ability,Specific Ability Tests,and (Un)Structured Interviews: A meta‐analysis with unique samples

Several meta‐analyses combine an extensive amount of research concerned with predicting training success. General mental ability is regarded as the best predictor with specific abilities or tests explaining little additional variance. However, only few studies measured all predictors within one sample. Thus, intercorrelations were often estimated based on other studies. Moreover, new methods for correcting range restriction are now available. The present meta‐analyses used samples derived from a German company in which applicants for different apprenticeships were tested with an intelligence structure test, specific ability tests as well as a structured and an unstructured interview. Therefore, intercorrelations between different assessment tools did not have to be estimated from other data. Results in the final examination, taking place at least 2 years after the original assessment, served as criterion variable. The dominant role of general mental ability was confirmed. However, specific abilities were identified that can be used as valuable additions. Job complexity moderated some of the relationships. Structured interviews were found to have good incremental validity over and above general mental ability. Unstructured interviews, on the other hand, performed poorly. Practical implications are discussed.     

The correlation between cue differentiation and associative recall

Two experiments were conducted to correlate performance on a test of cue differentiation with cued recall. In one experiment, subjects were given 2, 5, 8, or 12 study repetitions on verbal items consisting of three-word cues and single-word targets. The cues had varying numbers and patterns of identical elements. Cues were tested for differentiability and for their ability to produce target recall. In the second experiment, study repetitions on the same type of lists were followed by the two tests after the second, fourth, and final study repetitions. With a few exceptions, the frequency of occurrence of items that both were successfully differentiated and produced correct recall did not deviate significantly from predictions based on the assumption of independence of the two measures.     

Memory deficits in early infantile autism: some similarities to the amnesic syndrome.

Autistic children were compared with control children on tasks in which retention was tested by different methods. In three tests of recall, using named pictures, written words and spoken words as test stimuli, autistic children were impaired in comparison with age-matched normal children and with controls matched for verbal ability. In one test of forced-choice recognition of pictures, autistic children were impaired in comparison with ability-matched controls. In three tests of cued recall, using named pictures, written words and spoken words as test stimuli, and acoustic, graphemic and semantic cues, autistic children were not impaired in comparison with normal age-matched controls. In one test of paired-associate learning using non-related word pairs as test stimuli autistic children were not impaired in comparison with normal age-matched controls. These experimental paradigms were similar to some that have been used to investigate the amnesic syndrome in man. Thus findings on paired-associate learning differ in autistic and amnesic subjects, but findings on recall, recognition and cued recall are comparable. A possible parallel between autism and amnesia is discussed.