Implications of test revisions for research

Test revisions are increasingly common in psychology and neuropsychology in particular. However, such revisions may alter in complex ways the kind of information obtained, and they may assess traits, abilities, and conditions in ways different from earlier versions. This article outlines some of the problems associated with the revision of tests facing clinicians and researchers. Three broad classes of revision are considered. Part 1 considers the aging of tests, part 2 concerns the aging of participants, and part 3 considers changes in test format. Although the article focuses largely on measures of intelligence and personality, the issues addressed in the article apply to other tests and assessment domains as well.     

Pilot selection in the Norwegian Air Force: a validation and meta-analysis of the test battery

The purpose of this study was to evaluate the test battery currently used for pilot selection to the Norwegian Air Force. Selection is currently based on a standard battery of 20 different psychological tests as well as on medical tests and on an interview by a licensed psychologist. First, two-factor analyses were conducted to examine the relation between the tests in the battery. Then, a correlation study was conducted to evaluate the predictive validity of the tests against two criteria of pilot performance collected during the basic training period. Finally, a small-scale meta-analysis of previous validation studies in Norway was conducted. me best predictors of success in training, based on the meta-analysis, were Instrument Comprehension (mean r = .29), Mechanical Principles (mean r = .23), and Aviation Information (mean r = .22)     

Implications of an initial empirical victory for the truth of the theory and additional empirical victories

Psychologists take two propositions for granted. Specifically, empirical verification of predictions derived from a theory (a) support that the theory is more likely to be true and (b) support that additional predictions derived from the theory have an increased probability of being sustained if subjected to empirical testing. In contrast, I argue that both propositions depend strongly on whether auxiliary assumptions are taken into account. When auxiliary assumptions are not taken into account, the first proposition is valid but the second is not. When auxiliary assumptions are taken into account, the first proposition is not valid, and the second proposition encounters additional problems. I use Venn diagrams and Bayesian principles to demonstrate these conclusions.     

The test validation summary

Common rates employed in classificatory testing are the true positive rate (TPR), false positive rate (FPR), positive predictive power (PPP), and negative predictive power (NPP). FPR and TPR are estimated from research samples representing populations to be distinguished by classificatory testing. PPP and NPP are used by clinicians to classify test takers into populations. PPP and NPP depend on the base rate (BR) of population members in the clinician's sample. The authors introduce the test validation summary (TVS) as a means to report within a single graph the FPR and TPR and the ranges of PPP and NPP across all potential sample BRs for any chosen cut score. The authors investigate how the TVS has other applications, including the estimation of local BR for the condition of interest and the estimation of standard errors for FPR and TPR when estimated across multiple independent validation studies of the classificatory test.     

Positive psychology progress: empirical validation of interventions

Positive psychology has flourished in the last 5 years. The authors review recent developments in the field, including books, meetings, courses, and conferences. They also discuss the newly created classification of character strengths and virtues, a positive complement to the various editions of the Diagnostic and Statistical Manual of Mental Disorders (e. g., American Psychiatric Association, 1994), and present some cross-cultural findings that suggest a surprising ubiquity of strengths and virtues. Finally, the authors focus on psychological interventions that increase individual happiness. In a 6-group, random-assignment, placebo-controlled Internet study, the authors tested 5 purported happiness interventions and 1 plausible control exercise. They found that 3 of the interventions lastingly increased happiness and decreased depressive symptoms. Positive interventions can supplement traditional interventions that relieve suffering and may someday be the practical legacy of positive psychology.     

An empirical Q-matrix validation method for the sequential generalized DINA model

As a core component of most cognitive diagnosis models, the Q-matrix, or item and attribute association matrix, is typically developed by domain experts, and tends to be subjective. It is critical to validate the Q-matrix empirically because a misspecified Q-matrix could result in erroneous attribute estimation. Most existing Q-matrix validation procedures are developed for dichotomous responses. However, in this paper, we propose a method to empirically detect and correct the misspecifications in the Q-matrix for graded response data based on the sequential generalized deterministic inputs, noisy ‘and’ gate (G-DINA) model. The proposed Q-matrix validation procedure is implemented in a stepwise manner based on the Wald test and an effect size measure. The feasibility of the proposed method is examined using simulation studies. Also, a set of data from the Trends in International Mathematics and Science Study (TIMSS) 2011 mathematics assessment is analysed for illustration.     

An application of the empirical Bayes estimation to improve the reliability of subscores

In this paper, we show a subscore augmentation procedure that improves the reliability of subscales. The approach uses empirical Bayes estimations. This a generalization of Kelley's formula. The estimates are based on the information from other related scales on the test. We describe the procedure and we apply it to a multidimensional test. The reliability of the subscores increased, and the improvement could be considered equivalent to a 58.06% increase in the length of the test.     

An empirical test of two opposing theoretical models of prefrontal function.

The purpose of this study was to compare the validity of two models which contrast with each other in the manner in which they integrate neuropsychological tests into distinct prefrontal constructs. The first prefrontal model consists of five distinct functional constructs drawn from human clinical neuropsychology. The second model, elaborated by Goldman-Rakic, is based primarily on monkey research and postulates a basic prefrontal function, "on-line representational memory," which guides behavior in the absence of, or despite discriminative environmental stimuli. In the latter model, distinct prefrontal functional constructs are primarily defined in terms of various types of representational memory involved in specific tasks. Eleven "prefrontal" measures were obtained from 259 normal adults, stratified for age, education, and sex. Confirmatory factor analyses revealed that the Goldman-Rakic model "fit" the data better than the model derived from human clinical neuropsychology, while several constructs commonly used in human neuropsychology were refuted. It was concluded that new research on brain-damaged humans with a view to understanding prefrontal function might benefit from using the Goldman-Rakic model as a starting point.     

Implications of test revisions for assessment with Asian Americans

There are serious gaps in knowledge with respect to the use of standardized assessment instruments such as the Wechsler Adult Intelligence Scale-Third Edition (WAIS-III; D. Wechsler, 1997) or the Minnesota Multiphasic Personality Inventory-2 (MMPI-2; J. N. Butcher, W. G. Dahlstrom, J. R. Graham, A. Tellegen, & B. Kaemmer, 1989) with Asian Americans. Issues surrounding the availability, reliability, and validity of assessment instruments must be addressed before extended discussions about the implication of test revisions for this population can take place. The authors review the current status of the WAIS-III and MMPI-2 with Asian Americans with respect to their availability, reliability, and validity, including reasons why Asian Americans have been severely underrepresented in validation studies. The authors argue for the need to collect data on the use of standardized assessment instruments with Asian Americans and conclude with recommendations for the inclusion of this population in future test revision projects.     

An empirical test of two psychophysical models

Two theoretical relationships between sensitivity measures (Weber fractions, Ekman fractions, and their logarithms) and the exponents of the psychophysical power function were tested empirically with the brightness attribute. One model was based on Weber and Ekman fractions, the other on the logarithms of these measures. The stimulus parameters were time interval between standard and comparison targets and position of the standard in the luminance series. Weber fractions were based on data obtained by the method of constant stimuli, whereas Ekman fractions and exponents were based on data obtained by magnitude estimation. The results were in closer agreement with the theoretical predictions generated by the logarithmic model when group data were analyzed. With individual subjects, a detailed correspondence between fact and theory was not found with either model.     

An empirical test for Fishburn's additivity axiom

Three experiments were done to test the empirical relevance of Fishburn's (1967) additivity axiom, which says that people should be indifferent between pairs of gambles which satisfy certain conditions specified in the axiom. Each of the experiments consisted of two parts. In the first part, subjects had to evaluate consequences which were used in the second part as possible outcomes in a gamble. In the second part, subjects had to make choices among pairs of gambles. The experiments differed in respect to kinds of consequences and kinds of subjects used. Additivity analysis was applied to the data of the first part of each experiment, using a conjoint measurement model. A Monte Carlo study is included, which provides some hints for the evaluation of the stress coefficients obtained after applying additivity analysis to the empirical data matrices. The data of the second part of each experiment are discussed in respect to their relevance for Fishburn's (1967) additivity axiom. It was not strongly supported by the data, unless for a very restricted situation.     

Implications of direct and indirect range restriction for meta-analysis methods and findings

Range restriction in most data sets is indirect, but the meta-analysis methods used to date have applied the correction for direct range restriction to data in which range restriction is indirect. The authors show that this results in substantial undercorrections for the effects of range restriction, and they present meta-analysis methods for making accurate corrections when range restriction is indirect. Applying these methods to a well-known large-sample empirical database, the authors estimate that previous meta-analyses have underestimated the correlation between general mental ability and job performance by about 25%, indicating that this is potentially an important methodological issue in meta-analysis in general.     

Inhibition of return: A graphical meta-analysis of its time course and an empirical test of its temporal and spatial properties

Immediately after a stimulus appears in the visual field, there is often a short period of facilitated processing of stimuli at or near this location. This period is followed by one in which processing is impaired, rather than facilitated. This impairment has been termed inhibition of return (IOR). In the present study, the time course of this phenomenon was examined in two ways. (1) A graphical metaanalysis plotted the size of the effect as a function of the stimulus onset asynchrony (SOA) of the two stimuli. This analysis showed that IOR is impressively stable for SOAs of 300-1,600 msec. It also showed that the literature does not provide any clear sense of the duration of IOR. (2) An empirical approach was, therefore, taken to fill this gap in our knowledge of IOR. In three experiments, IOR was tested using SOAs between 600 and 4,200 msec. IOR was robust for approximately 3 sec and appeared to taper off after this point; the observed duration varied somewhat as a function of the testing conditions. In addition, for the first second, the degree of inhibition was inversely related to distance of the target from the original stimulus, but for the next 2 sec this spatial distribution was not observed. Theories of the mechanisms and function of IOR must conform to these spatial and temporal properties.     

Finding a serial burglar's home using distance decay and conditional origin–destination patterns: a test of empirical Bayes journey-to-crime estimation in the Hague

Can we tell where an offender lives from where he or she commits crimes? Journey-to-crime estimation is a tool that uses crime locations to tell us where to search for a serial offender's home. In this paper, we test a new method: empirical Bayes journey-to-crime estimation. It differs from previous methods because it utilises an ‘origin–destination’ rule in addition to the ‘distance decay’ rule that prior methods have used. In the new method, the profiler not only asks ‘what distances did previous offenders travel between their home and the crime scenes?’ but also ‘where did previous offenders live who offended at the locations included in the crime series I investigate right now?’. The new method could not only improve predictive accuracy, it could also reduce the traditional distinction between marauding and commuting offenders. Utilising the CrimeStat software, we apply the new method to 62 serial burglars in The Hague, The Netherlands, and show that the new method has higher predictive accuracy than methods that only exploit a distance decay rule. The new method not only improves the accuracy of predicting the homes of commuters—offenders who live outside their offending area—it also improves the search for marauders—offenders who live inside their offending area. After presenting an example of the application of the technique for prediction of a specific burglar, we discuss the limitations of the method and offer some suggestions for its future development. Copyright © 2009 John Wiley & Sons, Ltd.