Examining the effects of differential item (functioning and differential) test functioning on selection decisions: when are statistically significant effects practically important?

Item response theory differential test functioning (DTP) methods are often used to address issues in personnel selection, but the results are frequently difficult to interpret because statistically significant findings may have little practical importance. In this article, the authors proposed 2 effect size measures for DTP. One related DTP to mean raw score differences across groups: the other related DTP to the 4/5th rule for adverse impact at successive cut scores. The effects of DTP were examined in the context of personality assessment, professional licensure, and college admissions. Overall, the result indicated that although many items exhibited bias in analyses of the large samples, the net magnitudes of effect on potential selection decisions were nugatory.     

An anatomically constrained, stochastic model of eye movement control in reading

This article presents SERIF, a new model of eye movement control in reading that integrates an established stochastic model of saccade latencies (LATER; R. H. S. Carpenter, 1981) with a fundamental anatomical constraint on reading: the vertically split fovea and the initial projection of information in either visual field to the contralateral hemisphere. The novel features of the model are its simulation of saccade latencies as a race between two stochastic rise-to-threshold LATER units and its probabilistic selection of the target for the next saccade. The model generates simulated eye movement behavior that exhibits important characteristics of actual eye movements made during reading; specifically, simulations produce realistic saccade target distributions and replicate a number of critical reading phenomena, including the effects of word frequency on fixation durations, the inverted optimal viewing position effect, the trade-off between first and second fixation durations of refixated words, and the dependence of parafoveal preview benefit on eccentricity.     

Introduction: Some Remarks on History,Philosophy, and Education

Studies in Philosophy and Education -     

An event-related brain potential study of inhibition of return

Event-related brain potentials (ERPs) were recorded during two spatial-cuing experiments using nonpredictive cues. Our primary goal was to determine the electrophysiological consequences of inhibition of return (IOR). At long (> 500 msec) cue-target intervals, subjects responded more slowly to targets that appeared at or near the cued location, relative to targets that appeared on the opposite side of fixation from the cue. This behavioral IOR effect was associated with cue-validity effects on several components of the target-elicited ERP waveforms. The earliest such effect was a smaller occipital P1 on valid-cue trials, which we interpret as a P1 reduction. The P2 component was also smaller on valid-cue trials, indicating that nonpredictive spatial cues influence multiple stages of information processing at long cue-target intervals. Both of these effects were observed when sensory interactions between cue and target were likely to be negligible, indicating that they were not caused by sensory refractoriness. A different effect of cue validity, the posterior negative difference, was found when sensory interactions were likely to be greatest, indicating that it could arise from sensory refractoriness.     

Best univocal estimates of orthogonal common factors

A general formulation is presented for obtaining conditionally unbiased, univocal common-factor score estimates that have maximum validity for the true orthogonal factor scores. We note that although this expression is formally different from both Bartlett's formulation and Heermann's approximate expression, all three, while developed from very different rationales, yield identical results given that the common-factor model holds for the data. Although the true factor score validities can be raised by a different non-orthogonal transformation of orthogonalized regression estimates—as described by Mulaik—the resulting estimates lose their univocality.     

The effect of dimension content on observation and ratings of job performance

Recent theory and research in the performance appraisal area (e.g., Denisi, Cafferty, & Meglino, 1984; Feldman, 1986; Ilgen & Feldman, 1983; Williams, Denisi, & Blencoe, 1985) have suggested that providing information regarding the performance dimension to be rated will cause raters to select appropriate observational schemata and, as a result, produce higher quality ratings. A study was conducted to determine if giving raters dimension-relevant information prior to performance observation would affect their attention processes and rating quality. Prior to watching a videotape of an instructor giving a lecture, 156 subjects were given either: 1) correct information, 2) incorrect information, or 3) no information regarding dimensions of performance they would subsequently be asked to rate. The results indicated that giving prior information regarding dimension content affected subjects' attention processes. Further, raters receiving no information and those receiving misinformation prior to performance observation produced less accurate ratings compared to expert raters. Ratings produced by subjects receiving correct information did not differ significantly from experts' ratings. These results are discussed in terms of both their practical and theoretical implications.     

Commentaries on “Beyond statistical significance: Five principles for the new era of data analysis and reporting”

Three commentaries below provide different perspectives on data analysis and reporting. They generally focus on how the quality of the measures and manipulations determines the value of the analysis. Norbert Schwarz and Fritz Strack's comment is less on the right statistic and more on “sloppy reasoning, gaps between theoretical concepts and their operationalizations, and blissful ignorance of the situated nature of human thinking, feeling, and doing contribute more to the limited reproducibility of empirical findings than the choice of a particular test statistic.” They propose that particular effects are contextual and inappropriately labeled as true or false. Instead, our job is to focus on general constructs that make sense of the diversity of human experience and psychological reactions. Too often studies replicating psychological effects in the noisy and confounded conditions of the marketplace result in statistical uncertainty of garbage in, garbage out. Researchers instead need to look toward tests of specific interactions, which can clarify the influencing factors based on theoretical considerations. The second comment is by Andrew Gelman, an outstanding psychological statistician. He proposes that “once the data have been collected, the most important decisions have already been done.” He then provides four recommendations that enable the statistics to work appropriately. The first requirement of an effective study is to be sure that the measures address the construct of interest. Similar to the position of Schwarz and Strack, it is important to articulate the relevance of a statistically significant finding. The second recommendation seeks to curb large number of studies with inflated effect sizes built from narrow studies and unwarranted optimism. The third recommendation is to simulate data from a model and consider the distribution of possible results. That is often done to test a new analysis method, but it can be even more important in marketplace studies where novel characteristics of the sample and experimental conditions are included in the analysis. Finally, he recommends that one consider likely analyses needed before getting the data. Such foresight would encourage, for example, thinking about the kind of data needed to defend the equality of the control demographics against the treatment. The final commentary is by Stijn van Osselaer. He agrees that p-values reflect the detailed methods from a given study but do not focus on the problem of generalizability. Like Gelman, he sees designs focused on effect sizes may have generated too many studies that do not replicate. He contrasts broad explorations with narrowly defined existence tests that provide evidence that an effect exists somewhere but are mute on other contexts where they may apply. For theoretical problems relevant to applications, it is important to identify moderators through broad sampling across population characteristics, stimuli, and situations. He proposes that consumer psychologists should not try to do everything in one paper, but to build practically relevant, applicable knowledge across multiple articles. Different articles, authors, and research methods play various roles, with each article focusing on important stages in the process from generating hypotheses, providing existence proofs, and exploring their broad applicability. That pragmatic approach can integrate theoretical silos that seek to resolve complex human problems and has promise as a criterion for relevant publications.