Examining the effects of differential item (functioning and differential) test functioning on selection decisions: when are statistically significant effects practically important?

Item response theory differential test functioning (DTP) methods are often used to address issues in personnel selection, but the results are frequently difficult to interpret because statistically significant findings may have little practical importance. In this article, the authors proposed 2 effect size measures for DTP. One related DTP to mean raw score differences across groups: the other related DTP to the 4/5th rule for adverse impact at successive cut scores. The effects of DTP were examined in the context of personality assessment, professional licensure, and college admissions. Overall, the result indicated that although many items exhibited bias in analyses of the large samples, the net magnitudes of effect on potential selection decisions were nugatory.     

Interactions among chronic cold, corticosterone and puberty on energy intake and deposition

We have shown that chronic cold stress strongly interacts with corticosterone (B) to determine subsequent regulation of the hypothalamo-pituitary-adrenal (HPA) axis responses to novel stress. These studies, using the same 2 sets of rats, show that chronic cold also interacts with B and testosterone on signals of energy balance. The two groups of rats differed in weight by 20% and in age by 2 weeks (44-59 days of age). Adrenalectomized rats, replaced with varying doses of B, were exposed to cold or served as controls. Food intake and body weight during the experiments and hormones, metabolites and fat depots were measured on day 5. B, but not cold, affected food intake in the younger rats; by contrast, cold, but not B, affected food intake in the older rats. Testosterone was higher in older control rats and was markedly depressed by cold; younger rats had lower testosterone that was minimally affected by cold. Weight gain decreased in all rats at room temperature with increasing B, whereas they all lost weight in cold independently of B. Cold stimulated and B inhibited interscapular brown adipose tissue DNA content (reflecting sympathetic stimulation of thermogenesis). B stimulated insulin, whereas cold inhibited leptin and insulin; B also increased white adipose tissue weight gain in controls and inhibited its loss in cold. Leptin was unrelated to white adipose tissue depots in older control rats but was strongly related to these stores in younger rats and in all rats in cold. We conclude that: 1. By decreasing signals that act centrally to inhibit food intake (insulin, leptin and testosterone) cold allows B to stimulate food intake; 2. B inhibits weight gain although it causes accrual of fat; 3. Cold, probably through sympathetic stimulation of white adipose tissue, causes fat loss which is modulated by the inhibitory effect of B on sympathetic outflow; and, 4. The slope of the relationship between fat depot size and leptin becomes flatter in cold, possibly because of increased sympathetic outflow to these depots.     

Introduction: Some Remarks on History,Philosophy, and Education

Studies in Philosophy and Education -     

Some NEO-PI five-factor personality profiles

The aim of this study was to identify any clusters of score profiles to be found in a college sample of 236 subjects administered the five-factor NEO [Neuroticism, Extroversion, Openness] Personality Inventory (Costa & McCrae, 1985). Application of Ward's agglomerative hierarchical procedure to the score profiles disclosed six clusters that were replicated in a K-means partitioning process. The six clusters were then compared by a one-way analysis of variance (ANOVA) with respect to their mean five higher order scores on the Interpersonal Style Inventory (Lorr, 1986). The highly significant F tests provided confirmation of the characteristics of the cluster profiles isolated.     

Factor analysis of MCMI-II and PACL basic personality scales in a college sample

This study examined factor dimensions common to the eight basic personality scales of the Millon Clinical Multiaxial Inventory-II (MCMI-II; Millon, 1987) and Personality Adjective Check List (PACL; Strack, 1987, 1990). Subjects were 140 college students (65 men and 75 women). MCMI-II weighted raw scores (WRS) and WRS corrected for number of items endorsed true by regression were employed for analysis along with PACL T-scores. Principal components analyses with varimax and direct oblimin rotations were carried out separately on the two sets of MCMI-II and PACL scores. MCMI-II and PACL scales measuring the same personalities were usually correlated most highly with each other, although some divergences were noted. WRS yielded three bipolar dimensions and a fourth unipolar method factor that loaded only the five MCMI-II scales that were strongly correlated with number of items endorsed true. Residual scores yielded a more meaningful set of three bipolar dimensions labeled Social Introversion-Extraversion, Emotionality-Restraint, and Social Dominance-Submissiveness, without the method factor, that were very similar to personality dimensions found separately in the two tests. More research is needed to clarify the response bias issue in the MCMI-II and to further explicate similarities and differences between the MCMI-II and the PACL.     

Commentaries on “Beyond statistical significance: Five principles for the new era of data analysis and reporting”

Three commentaries below provide different perspectives on data analysis and reporting. They generally focus on how the quality of the measures and manipulations determines the value of the analysis. Norbert Schwarz and Fritz Strack's comment is less on the right statistic and more on “sloppy reasoning, gaps between theoretical concepts and their operationalizations, and blissful ignorance of the situated nature of human thinking, feeling, and doing contribute more to the limited reproducibility of empirical findings than the choice of a particular test statistic.” They propose that particular effects are contextual and inappropriately labeled as true or false. Instead, our job is to focus on general constructs that make sense of the diversity of human experience and psychological reactions. Too often studies replicating psychological effects in the noisy and confounded conditions of the marketplace result in statistical uncertainty of garbage in, garbage out. Researchers instead need to look toward tests of specific interactions, which can clarify the influencing factors based on theoretical considerations. The second comment is by Andrew Gelman, an outstanding psychological statistician. He proposes that “once the data have been collected, the most important decisions have already been done.” He then provides four recommendations that enable the statistics to work appropriately. The first requirement of an effective study is to be sure that the measures address the construct of interest. Similar to the position of Schwarz and Strack, it is important to articulate the relevance of a statistically significant finding. The second recommendation seeks to curb large number of studies with inflated effect sizes built from narrow studies and unwarranted optimism. The third recommendation is to simulate data from a model and consider the distribution of possible results. That is often done to test a new analysis method, but it can be even more important in marketplace studies where novel characteristics of the sample and experimental conditions are included in the analysis. Finally, he recommends that one consider likely analyses needed before getting the data. Such foresight would encourage, for example, thinking about the kind of data needed to defend the equality of the control demographics against the treatment. The final commentary is by Stijn van Osselaer. He agrees that p-values reflect the detailed methods from a given study but do not focus on the problem of generalizability. Like Gelman, he sees designs focused on effect sizes may have generated too many studies that do not replicate. He contrasts broad explorations with narrowly defined existence tests that provide evidence that an effect exists somewhere but are mute on other contexts where they may apply. For theoretical problems relevant to applications, it is important to identify moderators through broad sampling across population characteristics, stimuli, and situations. He proposes that consumer psychologists should not try to do everything in one paper, but to build practically relevant, applicable knowledge across multiple articles. Different articles, authors, and research methods play various roles, with each article focusing on important stages in the process from generating hypotheses, providing existence proofs, and exploring their broad applicability. That pragmatic approach can integrate theoretical silos that seek to resolve complex human problems and has promise as a criterion for relevant publications.     

At the boundaries of automaticity: negation as reflective operation

The present research investigated whether automatic social-cognitive skills are based on the same representations and processes as their controlled counterparts. Using the cognitive task of negating valence, the authors demonstrate that enhanced practice in negating the valence of a stimulus can lead to changes in the underlying associative representation. However, procedural, rule-based components of negations were generally unaffected by practice (Experiments 1-3). Moreover, negations of evaluative stimuli did not influence automatic evaluative responses to these stimuli, unless the negation was included in the associative representation of a stimulus (Experiments 4-6). These results suggest that some practice-related skill improvements are limited to conditions in which a general procedure can be substituted by the retrieval of results of previous applications from associative memory. Implications for research on automaticity and social cognition are discussed.     

A field examination of two measures of work motivation as predictors of leaders' influence tactics

The authors tested 2 motivation measures, the Motivation Sources Inventory (MSI; J. E. Barbuto & R. W. Scholl, 1998) and the Job Choice Decision-Making Exercise (A. M. Harrell & M. J. Stahl, 1981) as predictors of leaders' influence tactics. The authors sampled 219 leader-member dyads from a variety of organizations and communities throughout the central United States. Results strongly favored the MSI as a predictor of influence tactics. Limitations of the study include low power of relationships, sample size as limited by the research design, and education levels of participants. Future researchers should use larger and more diverse samples and test other relevant antecedents of leaders' behaviors.     

Simulating the impact during human jumping by means of a 4-degrees-of-freedom model with time-dependent properties

The authors simulated the vertical movements of a jumper and the force time courses by means of a 4-degrees-of-freedom model consisting of 4 masses, springs, and dampers. Of the motions simulated, only that of the mass imitating the trunk corresponded to the measured data. The best fit to the measured force curves were obtained in the simulation in which time-dependent model parameters were used. From the results, the authors concluded that at the beginning of the landing, a jumper behaves like a 2-mass model in which the leg segments (thighs, shanks, and feet) effectively combine into 1 mass. After approximately 60 ms, the connections between the leg segments become more compliant and the jumper behaves like a 4-mass model with a soft coupling between the leg segments. The process is equivalent to an increase of the degrees of freedom of the movements. At the end of the ground contact phase during hopping, the jumper has to contract the muscles in order to reach the envisaged jump height. In the model, that contraction could not be satisfactorily simulated.