Examining the effects of differential item (functioning and differential) test functioning on selection decisions: when are statistically significant effects practically important?

Item response theory differential test functioning (DTP) methods are often used to address issues in personnel selection, but the results are frequently difficult to interpret because statistically significant findings may have little practical importance. In this article, the authors proposed 2 effect size measures for DTP. One related DTP to mean raw score differences across groups: the other related DTP to the 4/5th rule for adverse impact at successive cut scores. The effects of DTP were examined in the context of personality assessment, professional licensure, and college admissions. Overall, the result indicated that although many items exhibited bias in analyses of the large samples, the net magnitudes of effect on potential selection decisions were nugatory.     

Evaluation of fixed momentary dro schedules under signaled and unsignaled arrangements

Fixed momentary schedules of differential reinforcement of other behavior (FM DRO) generally have been ineffective as treatment for problem behavior. Because most early research on FM DRO included presentation of a signal at the end of the DRO interval, it is unclear whether the limited effects of FM DRO were due to (a) the momentary response requirement of the schedule per se or (b) discrimination of the contingency made more salient by the signal. To separate these two potential influences, we compared the effects of signaled versus unsignaled FM DRO with 4 individuals with developmental disabilities whose problem behavior was maintained by social-positive reinforcement. During signaled FM DRO, the experimenter presented a visual stimulus 3 s prior to the end of the DRO interval and delivered reinforcement contingent on the absence of problem behavior at the second the interval elapsed. Unsignaled DRO was identical except that interval termination was not signaled. Results indicated that signaled FM DRO was effective in decreasing 2 subjects' problem behavior, whereas an unsignaled schedule was required for the remaining 2 subjects. These results suggest that the response requirement per se of FM DRO may not be problematic if it is not easily discriminated.     

What triggers causal attributions? The impact of valence and subjective probability

Various field studies and experimental simulations demonstrated that causal reasoning increases after unexpected as well as after unpleasant events. However, unpleasant events are seen as less likely than pleasant ones in everyday life. Accordingly, the subjective probability of the event and its hedonic quality were naturally confounded in these studies. To isolate the contribution of both determinants, the subjective probability and the valence of an event were independently manipulated in a laboratory experiment. Subjects completed an ostensible ‘professional skills test’ and received either success or failure feedback in relation to a criterion set by the experimenter. The subjective probability of success was varied by informing subjects about the distribution of success and failure in a comparable population (either 23 per cent or 77 per cent were said to meet the criterion). The results indicate a pronounced valence effect: The intensity of causal reasoning and the number of possible reasons reported for the outcome was greater after negative than after positive feedback, independent of the a priori probability of the outcome. No evidence for an increase in causal explanations after unexpected, as compared to expected, events was obtained. Several mediating processes are discussed.     

Erroneous knowledge of results affects decision and memory processes on timing tasks

On mental timing tasks, erroneous knowledge of results (KR) leads to incorrect performance accompanied by the subjective judgment of accurate performance. Using the start-stop technique (an analogue of the peak interval procedure) with both reproduction and production timing tasks, the authors analyze what processes erroneous KR alters. KR provides guidance (performance error information) that lowers decision thresholds. Erroneous KR also provides targeting information that alters response durations proportionately to the magnitude of the feedback error. On the production task, this shift results from changes in the reference memory, whereas on the reproduction task this shift results from changes in the decision threshold for responding. The idea that erroneous KR can alter different cognitive processes on related tasks is supported by the authors' demonstration that the learned strategies can transfer from the reproduction task to the production task but not visa versa. Thus effects of KR are both task and context dependent.     

Commentaries on “Beyond statistical significance: Five principles for the new era of data analysis and reporting”

Three commentaries below provide different perspectives on data analysis and reporting. They generally focus on how the quality of the measures and manipulations determines the value of the analysis. Norbert Schwarz and Fritz Strack's comment is less on the right statistic and more on “sloppy reasoning, gaps between theoretical concepts and their operationalizations, and blissful ignorance of the situated nature of human thinking, feeling, and doing contribute more to the limited reproducibility of empirical findings than the choice of a particular test statistic.” They propose that particular effects are contextual and inappropriately labeled as true or false. Instead, our job is to focus on general constructs that make sense of the diversity of human experience and psychological reactions. Too often studies replicating psychological effects in the noisy and confounded conditions of the marketplace result in statistical uncertainty of garbage in, garbage out. Researchers instead need to look toward tests of specific interactions, which can clarify the influencing factors based on theoretical considerations. The second comment is by Andrew Gelman, an outstanding psychological statistician. He proposes that “once the data have been collected, the most important decisions have already been done.” He then provides four recommendations that enable the statistics to work appropriately. The first requirement of an effective study is to be sure that the measures address the construct of interest. Similar to the position of Schwarz and Strack, it is important to articulate the relevance of a statistically significant finding. The second recommendation seeks to curb large number of studies with inflated effect sizes built from narrow studies and unwarranted optimism. The third recommendation is to simulate data from a model and consider the distribution of possible results. That is often done to test a new analysis method, but it can be even more important in marketplace studies where novel characteristics of the sample and experimental conditions are included in the analysis. Finally, he recommends that one consider likely analyses needed before getting the data. Such foresight would encourage, for example, thinking about the kind of data needed to defend the equality of the control demographics against the treatment. The final commentary is by Stijn van Osselaer. He agrees that p-values reflect the detailed methods from a given study but do not focus on the problem of generalizability. Like Gelman, he sees designs focused on effect sizes may have generated too many studies that do not replicate. He contrasts broad explorations with narrowly defined existence tests that provide evidence that an effect exists somewhere but are mute on other contexts where they may apply. For theoretical problems relevant to applications, it is important to identify moderators through broad sampling across population characteristics, stimuli, and situations. He proposes that consumer psychologists should not try to do everything in one paper, but to build practically relevant, applicable knowledge across multiple articles. Different articles, authors, and research methods play various roles, with each article focusing on important stages in the process from generating hypotheses, providing existence proofs, and exploring their broad applicability. That pragmatic approach can integrate theoretical silos that seek to resolve complex human problems and has promise as a criterion for relevant publications.     

At the boundaries of automaticity: negation as reflective operation

The present research investigated whether automatic social-cognitive skills are based on the same representations and processes as their controlled counterparts. Using the cognitive task of negating valence, the authors demonstrate that enhanced practice in negating the valence of a stimulus can lead to changes in the underlying associative representation. However, procedural, rule-based components of negations were generally unaffected by practice (Experiments 1-3). Moreover, negations of evaluative stimuli did not influence automatic evaluative responses to these stimuli, unless the negation was included in the associative representation of a stimulus (Experiments 4-6). These results suggest that some practice-related skill improvements are limited to conditions in which a general procedure can be substituted by the retrieval of results of previous applications from associative memory. Implications for research on automaticity and social cognition are discussed.     

A field examination of two measures of work motivation as predictors of leaders' influence tactics

The authors tested 2 motivation measures, the Motivation Sources Inventory (MSI; J. E. Barbuto & R. W. Scholl, 1998) and the Job Choice Decision-Making Exercise (A. M. Harrell & M. J. Stahl, 1981) as predictors of leaders' influence tactics. The authors sampled 219 leader-member dyads from a variety of organizations and communities throughout the central United States. Results strongly favored the MSI as a predictor of influence tactics. Limitations of the study include low power of relationships, sample size as limited by the research design, and education levels of participants. Future researchers should use larger and more diverse samples and test other relevant antecedents of leaders' behaviors.