The Impact of P-hacking on “Redefine Statistical Significance”

In their proposal to “redefine statistical significance,” Benjamin et al. claim that lowering the default cutoff for statistical significance from .05 to .005 would “immediately improve the reproducibility of scientific research in many fields.” Benjamin et al. assert specifically that false positive rates would fall below 10% and replication rates would double under the lower cutoff. I analyze these claims here, showing how the failure to account for P-hacking and other widespread reporting issues leads to exaggerated and misleading conclusions about the potential impact of the .005 proposal.     

The Effective Use of Secondary Data

In primary data analysis the individuals who collect the data also analyze it; for meta-analysis an investigator quantitatively combines the statistical results from multiple studies of a phenomenon to reach a conclusion; in secondary data analysis individuals who were not involved in the collection of the data analyze the data. Secondary data analysis may be based on the published data or it may be based on the original data. Most studies of animal cognition involve primary data analysis; it was difficult to identify any that were based on meta-analysis; secondary data analysis based on published data has been used effectively, and examples are given from the research of John Gibbon on scalar timing theory. Secondary data analysis can also be based on the original data if the original data are available in an archive. Such an archive in the field of animal cognition is feasible and desirable.     

An evaluation of estimation data collection to trial‐by trial data collection during discrete trial teaching

There are many data collection procedures used during discrete trial teaching including first‐trial data collection, probe data, trial‐by‐trial data collection, and estimation data. Continuous, or trial‐by‐trial data collection, consists of the interventionist collecting data on learner behavior on each trial. Estimation data consists of the interventionist estimating learner performance after a teaching session using a rating scale. The purpose of the present study was to compare trial‐by‐trial data collection to estimation data collection during discrete trial teaching to teach children expressive labels. The data collection procedures were examined in terms of accuracy of data collection, efficiency of teaching (i.e., number of trials delivered per session), and rate of child acquisition of targets. Results of the adapted alternating treatment design replicated across three participants and multiple targets found estimation data collection to be as accurate as trial‐by‐trial data collection in determining mastery of targets. Estimation data collected by the interventionist was also found to be accurate when compared to the actual trial‐by‐trial data collected after the study concluded.     

Simple imputation methods versus direct likelihood analysis for missing item scores in multilevel educational data

Missing data, such as item responses in multilevel data, are ubiquitous in educational research settings. Researchers in the item response theory (IRT) context have shown that ignoring such missing data can create problems in the estimation of the IRT model parameters. Consequently, several imputation methods for dealing with missing item data have been proposed and shown to be effective when applied with traditional IRT models. Additionally, a nonimputation direct likelihood analysis has been shown to be an effective tool for handling missing observations in clustered data settings. This study investigates the performance of six simple imputation methods, which have been found to be useful in other IRT contexts, versus a direct likelihood analysis, in multilevel data from educational settings. Multilevel item response data were simulated on the basis of two empirical data sets, and some of the item scores were deleted, such that they were missing either completely at random or simply at random. An explanatory IRT model was used for modeling the complete, incomplete, and imputed data sets. We showed that direct likelihood analysis of the incomplete data sets produced unbiased parameter estimates that were comparable to those from a complete data analysis. Multiple-imputation approaches of the two-way mean and corrected item mean substitution methods displayed varying degrees of effectiveness in imputing data that in turn could produce unbiased parameter estimates. The simple random imputation, adjusted random imputation, item means substitution, and regression imputation methods seemed to be less effective in imputing missing item scores in multilevel data settings.     

Which Source Possesses the Best Data on the Empirical Aspects of Criminal Events? A Theory of Opportunity and Necessary Conditions

ABSTRACT

Offenders and nonoffenders possess valuable information about crime. But which possesses the best data? This is a complex issue, so I narrow my focus to data on empirical aspects of criminal events. Drawing on the necessary conditions perspective, I theorize that a source’s possession 1) of data varies directly with its involvement in cases; 2) of representative data varies inversely with nonrandom involvement in cases and nonrandom siphoning off from the larger group to which it belongs; and, 3) of accurate data varies inversely with time since involvement in cases. Those general principles suggest that offenders, especially active ones, possess the most data, representative data, and accurate data on empirical aspects of criminal events. I conclude by discussing the implications of those general principles for observation research, sources’ possession of subjective data, and their possession of empirical data on other criminological events, specifically victimization and policing.     

Missing data: prevalence and reporting practices

Results are described for a survey assessing prevalence of missing data and reporting practices in studies with missing data in a random sample of empirical research journal articles from the PsychINFO database for the year 1999, two years prior to the publication of a special section on missing data in Psychological Methods. Analysis indicates missing data problems were found in about one-third of the studies. Further, analytical methods and reporting practices varied widely for studies with missing data. One may consider these results as baseline data to assess progress as reporting standards evolve for studies with missing data. Some potential reporting standards are discussed.     

Simultaneous analysis of coupled data matrices subject to different amounts of noise

In many areas of science, research questions imply the analysis of a set of coupled data blocks, with, for instance, each block being an experimental unit by variable matrix, and the variables being the same in all matrices. To obtain an overall picture of the mechanisms that play a role in the different data matrices, the information in these matrices needs to be integrated. This may be achieved by applying a data‐analytic strategy in which a global model is fitted to all data matrices simultaneously, as in some forms of simultaneous component analysis (SCA). Since such a strategy implies that all data entries, regardless the matrix they belong to, contribute equally to the analysis, it may obfuscate the overall picture of the mechanisms underlying the data when the different data matrices are subject to different amounts of noise. One way out is to downweight entries from noisy data matrices in favour of entries from less noisy matrices. Information regarding the amount of noise that is present in each matrix, however, is, in most cases, not available. To deal with these problems, in this paper a novel maximum‐likelihood‐based simultaneous component analysis method, referred to as MxLSCA, is proposed. Being a stochastic extension of SCA, in MxLSCA the amount of noise in each data matrix is estimated and entries from noisy data matrices are downweighted. Both in an extensive simulation study and in an application to data stemming from cross‐cultural emotion psychology, it is shown that the novel MxLSCA strategy outperforms the SCA strategy with respect to disclosing the mechanisms underlying the coupled data.     

Parallel Factor Analysis of gait waveform data: A multimode extension of Principal Component Analysis

Gait data are typically collected in multivariate form, so some multivariate analysis is often used to understand interrelationships between observed data. Principal Component Analysis (PCA), a data reduction technique for correlated multivariate data, has been widely applied by gait analysts to investigate patterns of association in gait waveform data (e.g., interrelationships between joint angle waveforms from different subjects and/or joints). Despite its widespread use in gait analysis, PCA is for two-mode data, whereas gait data are often collected in higher-mode form. In this paper, we present the benefits of analyzing gait data via Parallel Factor Analysis (Parafac), which is a component analysis model designed for three- or higher-mode data. Using three-mode joint angle waveform data (subjects×time×joints), we demonstrate Parafac's ability to (a) determine interpretable components revealing the primary interrelationships between lower-limb joints in healthy gait and (b) identify interpretable components revealing the fundamental differences between normal and perturbed subjects' gait patterns across multiple joints. Our results offer evidence of the complex interconnections that exist between lower-limb joints and limb segments in both normal and abnormal gaits, confirming the need for the simultaneous analysis of multi-joint gait waveform data (especially when studying perturbed gait patterns).     

Maintaining a minimassive database of microdrinking measures

For longitudinal studies of rats’ consumption of ethanol, water, and food, we require manual entry of directly observed measurements, consolidation with computer-collected measurements, quick computation of derived statistics, and the export of selected data for further analysis and graphing. As in a spreadsheet, our solution propagates changes in data values through all computations dependent on them but, as in a relational database, maintains data in discrete disk records indexed by day and cage, in which the format of these records is also described by records stored in the same database. During manual data updates, each day’s data are presented in a spreadsheet-like format but with more control over data entry validity than a spreadsheet typically affords. Experimental integrity is further enhanced by producing printed worksheets for data entry, assignment of rats to treatments, and printed injection instructions directly from such customized data files.     

MISSING DATA: A CONCEPTUAL REVIEW FOR APPLIED PSYCHOLOGISTS

There has been conspicuously little research concerning missing data problems in the applied psychology literature. Fortunately, other fields have begun to investigate this issue. These include survey research, marketing, statistics, economics, and biometrics. A review of this literature suggests several trends for applied psychologists. For example, listwise deletion of data is often the least accurate technique to deal with missing data. Other methods for estimating missing data scores may be more accurate and preserve more data for investigators to analyze. Further, the literature reveals that the amount of missing data and the reasons for deletion of data impact how investigators should handle the problem. Finally, there is a great need for more investigation of strategies for dealing with missing data, especially when data are missing in nonrandom or systematic patterns.     

计划缺失设计——通过有意缺失让研究更高效

缺失值是社会科学研究中非常普遍的现象。全息极大似然估计和多重插补是目前处理缺失值最有效的方法。计划缺失设计利用特殊的实验设计有意产生缺失值, 再用现代的缺失值处理方法来完成统计分析, 获得无偏的统计结果。计划缺失设计可用于横断面调查减少(或增加)问卷长度和纵向调查减少测量次数, 也可用于提高测量有效性。常用的计划缺失设计有三式设计和两种方法测量。     

A microprocessor-based psychopathology laboratory: IV. Software design

Software designed for the microprocessor-based psychopathology laboratory provides a powerful data collection and maintenance package. A library of prepackaged well-documented program modules has been developed to aid in developing and maintaining programs to administer experiments, to store, retrieve, and interpret data. A simple, but powerful, data base system allows new experiments to be incorporated easily, and obsolete experiments to be deleted. The data collection programs are independent of the general data base, permitting them to be moved to dedicated remote systems as necessary. The data are stored in raw form to permit the researcher to try novel approaches in interpreting existing data.     

Becoming Racially Literate About Data and Data-Literate About Race: Data Visualizations in the Classroom as a Site of Racial-Ideological Micro-Contestations

Data visualizations are now commonplace in the public media. The ability to interpret and create such visualizations, as a form of data literacy, is increasingly important for democratic participation. Yet, the cross-disciplinary knowledge and skills needed to produce and use data visualizations and to develop data literacy are not fluidly integrated into traditional K–12 subject areas. In this article, we nuance and complicate the push for data literacy in STEM reform efforts targeting youth of color. We explore a curricular reform project that integrated explicit attention to issues pertaining to the collection, analysis, interpretation, representation, visualization, and communication of data in an introductory computer science class. While the study of data in this unit emphasized viewing and approaching data in context, neither the teacher nor the students were supported in negotiating the racialized context of data that emerged in classroom discussions. To better understand these dynamics, we detail the construct of racial literacy and develop an interpretative framework of racial-ideological micro-contestations. Through an in-depth analysis of a classroom interaction using this framework, we explore how contestations about race can emerge when data visualizations from the public media are incorporated into STEM learning precisely because the contexts of data are often racialized. We argue that access to learning about data visualization, without a deep interrogation of race and power, can be counterproductive and that efforts to develop authentic data literacy require the concomitant development of racial literacy.     

Best practices for missing data management in counseling psychology

This article urges counseling psychology researchers to recognize and report how missing data are handled, because consumers of research cannot accurately interpret findings without knowing the amount and pattern of missing data or the strategies that were used to handle those data. Patterns of missing data are reviewed, and some of the common strategies for dealing with them are described. The authors provide an illustration in which data were simulated and evaluate 3 methods of handling missing data: mean substitution, multiple imputation, and full information maximum likelihood. Results suggest that mean substitution is a poor method for handling missing data, whereas both multiple imputation and full information maximum likelihood are recommended alternatives to this approach. The authors suggest that researchers fully consider and report the amount and pattern of missing data and the strategy for handling those data in counseling psychology research and that editors advise researchers of this expectation.     

Official and unofficial data

Unofficial data are empirical findings that guide our research but are generally not reported. This article delineates four forms of unofficial data: casual observation of ourselves and others, unsystematic naturalistic observation, uncodable forms of clinical and phenomenological data, and accidental and nonquantifiable incidents and findings arising during pilot testing and data analysis. The article argues for a broadened conception of empiricism that recognizes unofficial data as data, explores the different contexts of the scientific process in which official and unofficial data are useful, and suggests the implications of the existence and utility of unofficial data for research and publication practices.     

Measuring data sharing intention and its association with the acceptance of connected vehicles

Transportation agencies and researchers are optimistic about the potential use of data collected from connected vehicles (CVs) for a variety of traffic and transportation applications. However, the literature lacks the evaluation of data sharing intention of the public for CV applications and its relationship with CV acceptance. This study investigated this gap by conducting a questionnaire survey of 2400 US adults. The results showed that the intention to share CV data depends upon the use of data but not the type of data. The possible uses of CV data were found to be grouped under four categories: driver information, congestion assessment and reduction, and pavement and infrastructure assessment and improvement (ICP); enforcement of traffic rules and fees based on usage (EF); roadside assistance and crash investigation (RC); and research purposes (RP). The data sharing intention for these four data uses vary, though with some commonality, which reflects the overall data sharing intention in CV technology (CVT). In addition, it was found that data privacy and security issues of CVT lower the data sharing intention and CV acceptance. Thus, a number of ways to improve CV acceptance by minimizing the data issues of CVT are discussed. Significant differences in perception of data privacy and security, data sharing intention, and CV acceptance were observed for individuals of different socio-economic and driving-related characteristics.     

Ecological Validity as a Key Feature of External Validity in Research on Human Development

This issue addresses the topic of ecological validity in research in human development across the life span. Although recent developments in study design, data collection methods, and data analysis techniques have greatly advanced researchers’ ability to collect large amounts of data on large groups of individuals in natural settings, it is important to approach these data with a reflected understanding of their ecological validity. Just because data were collected using everyday and familiar stimuli or ecological momentary assessment methods does not mean automatically that the ecological validity of these data is guaranteed and can go unquestioned.     

概化理论方差分量估计的跨分布分析

方差分量估计是进行概化理论分析的关键。采用MonteCarlo模拟技术,探讨心理与教育测量数据分布对概化理论各种方法估计方差分量的影响。数据分布包括正态、二项和多项分布,估计方法包括Traditional、Jackknife、Bootstrap和MCMC方法。结果表明:(1)Traditional方法估计正态分布和多项分布数据的方差分量相对较好,估计二项分布数据需要校正,Jackknife方法准确地估计了三种分布数据的方差分量,校正的Bootstrap方法和有先验信息的MCMC方法(MCMCinf)估计三种分布数据的方差分量结果较好;(2)心理与教育测量数据分布对四种方法估计概化理论方差分量有影响,数据分布制约着各种方差分量估计方法性能的发挥,需要加以区分地使用。     

Judgments about the duration of brief stimuli

Visual duration discrimination data for durations between 70 and 1,020 msec are presented. A model for duration discrimination proposed by Allan, Kristofferson, and Wiens (1972) is elaborated, and the data are discussed in terms of the model. The data axe in agreement with the basic assumptions of the model. Differences between our data and duration discrimination data presented by others are discussed.