A nonparametric approach for assessing latent trait unidimensionality

Assuming a nonparametric family of item response theory models, a theory-based procedure for testing the hypothesis of unidimensionality of the latent space is proposed. The asymptotic distribution of the test statistic is derived assuming unidimensionality, thereby establishing an asymptotically valid statistical test of the unidimensionality of the latent trait. Based upon a new notion of dimensionality, the test is shown to have asymptotic power 1. A 6300 trial Monte Carlo study using published item parameter estimates of widely used standardized tests indicates conservative adherence to the nominal level of significance and statistical power averaging 81 out of 100 rejections for examinee sample sizes and psychological test lengths often incurred in practice.The referees' comments were remarkably detailed and greatly enhanced the writeup and sensitized the author to certain pertinent issues. Discussions with Fritz Drasgow, Lloyd Humphreys, Dennis Jennings, Brian Junker, Robert Linn, Ratna Nandakumar, and Robin Shealy were also very useful.This research was supported by the Office of Naval Research under grant N00014-84-K-0186; NR 150-533, and by the National Science Foundation under grant DMS 85-03321.     

Reliability of test scores in nonparametric item response theory

Three methods for estimating reliability are studied within the context of nonparametric item response theory. Two were proposed originally by Mokken (1971) and a third is developed in this paper. Using a Monte Carlo strategy, these three estimation methods are compared with four classical lower bounds to reliability. Finally, recommendations are given concerning the use of these estimation methods.The authors are grateful for constructive comments from the reviewers and from Charles Lewis.     

A taxonomy of item response models

A number of models for categorical item response data have been proposed in recent years. The models appear to be quite different. However, they may usefully be organized as members of only three distinct classes, within which the models are distinguished only by assumptions and constraints on their parameters. “Difference models” are appropriate for ordered responses, “divide-by-total” models may be used for either ordered or nominal responses, and “left-side added” models are used for multiple-choice responses with guessing. The details of the taxonomy and the models are described in this paper. The present study was supported in part by two postdoctoral fellowships awarded to Lynne Steinberg: an Educational Testing Service Postdoctoral Fellowship at ETS, Princeton, NJ and an NIMH Individual National Research Service Award at Stanford University, Stanford, CA. Helpful comments by the editor and three anonymous reviewers are gratefully acknowledged.     

Self-selection biases in correlational studies based on questionnaires

It is commonly held that even where questionnaire response is poor, correlational studies are affected only by loss of degrees of freedom or precision. We show that this supposition is not true. If the decision to respond is correlated with a substantive variable of interest, then regression or analysis of variance methods based upon the questionnaire results may be adversely affected by self-selection bias. Moreover such bias may arise even where response is 100%. The problem in both cases arises where selection information is passed to the score indirectly via the disturbance or individual effects, rather than entirely via the observable explanatory variables. We suggest tests for the ensuing self-selection bias and possible ways of handling the ensuing problems of inference.The author would like to thank members of the Psychology Department at UWA for references on the illustrative example, namely the effects of sex-stereotyping on achievements in mathematics; thanks are also due to the anonymous reviewers whose comments resulted in material improvements to the paper.     

Empirical tests of scientific realism: A quantitative framework

The scientific realism debate in philosophy of science raises some intriguing methodological issues. Scientific realism posits a link between a scientific theory's observational and referential success. This opens the possibility of testing the thesis empirically, by searching for evidence of such a link in the record of theories put forward in the history of science. Many realist philosophers working today propose case study methodology as a way of carrying out such a test. This article argues that a qualitative method such as case study methodology is not adequate for this purpose, for two reasons: to test scientific realism is to pose an effects-of-causes question, and observational and referential success are quantities that theories possess to a greater or lesser degree. The article concludes that an empirical test of scientific realism requires a quantitative method.     

The power within: Mass media,scientific entertainment,and the introduction of psychical research into China, 1900–1920

How did a new science initially promoted by only a few individuals eventually become a widespread cultural phenomenon practiced and known by thousands of people? Following a transnational approach, this article traces the introduction of psychical research into China during the first two decades of the 20th century. Known in the Republican period (1912–1949) as Spiritual Science (xinling kexue or xinling yanjiu), psychical research flourished between the 1920s and 1930s, playing a key role in the popularization of applied psychology and mind-cure across China. This article takes a step back from the heyday of Spiritual Science by looking at the period that immediately preceded and helped define it. Focused on wide-circulation newspapers, popular manuals, and stage performances, it teases out the ways in which Chinese popular culture translated European, American, and Japanese psychical research to local Chinese audiences in the midst of China's search for modernity. By naturalizing the reality of psychic powers, spiritual scientists blurred the boundaries between science and superstition in a period when these were posited as diametrically opposed.     

It's just (family) business: The impact of familial work experience on perceived qualification and hireability during the selection process

Though scholars have explored the impact of familial work experience (FWE) posthire, research has yet to determine the role of such experience during the selection process. Drawing from Attribution Theory and research on nepotism and family business dynamics, we suggest that the presence of FWE on an applicant's selection materials will lead to reduced perceptions of qualifications and hireability. Through two online experiments (N₁ = 170, N₂ = 251), we found mixed support for this proposition. In our first study focused on cover letters, we found that applicants who reported FWE were not appraised differently than those who did not. In our second study focused on letters of recommendation, however, we found that applicants who reported FWE were perceived as less qualified and less hireable. Moreover, we found that the level of job (i.e., service vs. managerial) moderates these relations, such that managerial applicants with FWE on their letter of recommendation were perceived as less hireable than those who did not. Implications for research and practice are discussed, and future research directions are offered.     

Getting interested in interests for employee selection: Key concerns and areas for future research

The paper by Wille and De Fruyt suggests that vocational interests should be used throughout the human resources cycle and provides a useful model that can facilitate the application of interests throughout this process. This commentary builds on their review to discuss the unique benefits and challenges of implementing vocational interests in the employee selection process. Using research from the employee selection literature, we also discuss potential solutions for many of the issues raised by Wille and De Fruyt. Therefore, our commentary is meant to supplement the focal article and answer specific questions about why and how vocational interests can be used in this high-stakes setting.     

Hiring on vocational interests to simultaneously improve validity and organizational diversity

We discuss how using vocational interests in the selection process can help address the diversity-validity dilemma. First, we point out how incorporating vocational interests as predictors in selection could help to reduce adverse impact. We further suggest that by using optimal predictor weights, one could simultaneously improve validity while enhancing organisational diversity. Finally, the predictive validity of vocational interests arises from their ability to capture the congruence between individuals and occupations, which is a cross-level phenomenon. Thus, when gathering validity evidence for vocational interests, multi-occupation samples should be incorporated into validation efforts.     

Intentional response distortion during the COVID-19 pandemic

COVID-19 has abruptly and unexpectedly transformed nearly every aspect of work, including but not limited to increased unemployment rates and uncertainty regarding future job prospects. Response distortion has always been a concern given that many organizations rely on information that is self-reported by applicants regarding their potential employability (e.g., responses to self-reported personality instruments, resumes, interview responses). Drawing from the Valence-Instrumentality-Expectancy (VIE) theory of motivation, we propose that the uncertainty surrounding jobs may lead to amplified distorted responses on these measures in areas where COVID-19 was most salient. In a sample of 213 working adults [~50% female, age M = 38.48], the present study shows that increases in response distortion on a measure of conscientiousness were more pronounced as a function of (a) local COVID positivity rates and (b) job type, such that frontline workers distorted their responses the most. Findings are discussed in the context of VIE theory, personality measurement, and challenges with maintaining effective selection procedures.