Commentaries on “Beyond statistical significance: Five principles for the new era of data analysis and reporting”

Three commentaries below provide different perspectives on data analysis and reporting. They generally focus on how the quality of the measures and manipulations determines the value of the analysis. Norbert Schwarz and Fritz Strack's comment is less on the right statistic and more on “sloppy reasoning, gaps between theoretical concepts and their operationalizations, and blissful ignorance of the situated nature of human thinking, feeling, and doing contribute more to the limited reproducibility of empirical findings than the choice of a particular test statistic.” They propose that particular effects are contextual and inappropriately labeled as true or false. Instead, our job is to focus on general constructs that make sense of the diversity of human experience and psychological reactions. Too often studies replicating psychological effects in the noisy and confounded conditions of the marketplace result in statistical uncertainty of garbage in, garbage out. Researchers instead need to look toward tests of specific interactions, which can clarify the influencing factors based on theoretical considerations. The second comment is by Andrew Gelman, an outstanding psychological statistician. He proposes that “once the data have been collected, the most important decisions have already been done.” He then provides four recommendations that enable the statistics to work appropriately. The first requirement of an effective study is to be sure that the measures address the construct of interest. Similar to the position of Schwarz and Strack, it is important to articulate the relevance of a statistically significant finding. The second recommendation seeks to curb large number of studies with inflated effect sizes built from narrow studies and unwarranted optimism. The third recommendation is to simulate data from a model and consider the distribution of possible results. That is often done to test a new analysis method, but it can be even more important in marketplace studies where novel characteristics of the sample and experimental conditions are included in the analysis. Finally, he recommends that one consider likely analyses needed before getting the data. Such foresight would encourage, for example, thinking about the kind of data needed to defend the equality of the control demographics against the treatment. The final commentary is by Stijn van Osselaer. He agrees that p-values reflect the detailed methods from a given study but do not focus on the problem of generalizability. Like Gelman, he sees designs focused on effect sizes may have generated too many studies that do not replicate. He contrasts broad explorations with narrowly defined existence tests that provide evidence that an effect exists somewhere but are mute on other contexts where they may apply. For theoretical problems relevant to applications, it is important to identify moderators through broad sampling across population characteristics, stimuli, and situations. He proposes that consumer psychologists should not try to do everything in one paper, but to build practically relevant, applicable knowledge across multiple articles. Different articles, authors, and research methods play various roles, with each article focusing on important stages in the process from generating hypotheses, providing existence proofs, and exploring their broad applicability. That pragmatic approach can integrate theoretical silos that seek to resolve complex human problems and has promise as a criterion for relevant publications.     

Decomposition can harm the accuracy of behavioural frequency reports

In survey research, the use of decomposition can lead to pronounced reporting errors as seen by overreporting and overall reporting error. A total of 87 subjects answered either decomposed or undecomposed questions concerning telephone calls made by them while at work. The questionnaire conditions varied the length of the reference period (1 week or 6 months), and the type of call (local or long‐distance). Decomposition conditions introduced either spatial or temporal cues. In all comparisons, decomposed questions increased overreporting bias relative to undecomposed questions. In addition, undecomposed questions with a 1‐week reference period led to increased overreporting bias in comparison to undecomposed/6‐month questions. Results are consistent with a category split estimation model in which smaller categories are predicted to lead to overreporting, and larger categories to underreporting. Decomposition is not recommended for gaining retrospective reports of non‐distinctive, frequent events. Copyright © 2000 John Wiley & Sons, Ltd.     

The Janus face of power in intergroup contexts: a further exploration of the noblesse oblige effect

The authors examined the impact of power on in-group bias by manipulating group members' power over the in-group and power over the out-group as orthogonal factors. Each factor had 3 levels: 0%, 50%, and 100%. Participants were 216 male pupils (12-13 years old). Participants showed no in-group bias when they had 0% control over the in-group, strong in-group bias with 50% control, but less in-group bias with 100% control. Participants showed more in-group bias when they had 0% control over the out-group than when they had 50% or 100% control. The combination of these 2 main effects resulted in the noblesse oblige effect: Group members with complete control over both in-group and out-group expressed less in-group bias than did group members who shared control with an out-group.     

Accessibility experiences and the hindsight bias: I knew it all along versus it could never have happened

In two experiments, we tested accessibility experiences versus accessible content in influencing the hindsight bias when participants generated either thoughts about alternative outcomes or thoughts about known outcomes. Participants who had listed many thoughts (Experiment 1) and those who had contracted their brow muscles (Experiment 2) when considering alternate outcomes rated the known outcome as more likely than did than those who had listed two thoughts or who had not contracted their brows--a "backfire" effect. In contrast, but no less ironically, participants who had listed many thoughts and those who contracted their brows when considering known outcomes rated those outcomes as less likely--an "it could never have happened" effect. Both effects are due to subjective accessibility experiences, and their role in influencing and debiasing the hindsight bias is discussed.     

“How Many Partners Is Too Many?” Shaping Perceptions of Personal Vulnerability1

The effectiveness of interventions that directly inform people of the risks posed by their own behavior is frequently undermined by people's ability to defend themselves against unwanted information. In response to this difficulty, an alternative approach to shaping people's perceptions of personal vulnerability in which people remained unaware of the intervention was tested. As part of a survey on sexual issues, college undergraduates reported the total number of sexual partners they had had on a response scale that systematically framed their sexual behavior as being either above average or below average. Participants in the 2 conditions reported the same absolute number of sexual partners. However, those who received a response scale that implied they had had more partners than the average student reported greater concern about their sexual behavior than did those who received a scale that implied they had had fewer partners than average. The implications of this approach for influencing perceptions of personal risk are discussed.     

Getting groups to develop good strategies: Effects of reflexivity interventions on team process,team performance,and shared mental models

This study examines the effect of guided reflection on team processes and performance, based on West’s (1996, 2000) concept of reflexivity. Communicating via e-mail, 49 hierarchically structured teams (one commander and two specialists) performed seven 15 min shifts of a simulated team-based military air-surveillance task (TAST) in two meetings, a week apart. At the beginning of the second meeting, teams were assigned either to a reflexivity (individual or group) or to a control condition. Results show that reflexivity enhanced performance, the link between reflexivity and team performance being mediated by communication and implementation of strategies as well as by similarity of mental models. Contrary to expectations, individual reflexivity was superior to group reflexivity. Additional analyses suggested that group reflexivity decreased the commanders’ active behavior and increased discussion of strategies that were too general to be helpful. Results point to the usefulness of reflexivity as a generic intervention but underscore the importance of focusing on strategies that are task-specific.