Recent reliability reporting practices in Psychological Assessment: recognizing the people behind the data

Helms, Henze, Sass, and Mifsud (2006) defined good practices for internal consistency reporting, interpretation, and analysis consistent with an alpha-as-data perspective. Their viewpoint (a) expands on previous arguments that reliability coefficients are group-level summary statistics of samples' responses rather than stable properties of scales or measures and (b) encourages researchers to investigate characteristics of reliability data for their own samples and subgroups within their samples. In Study 1, we reviewed past and current reliability reporting practices in a sample of Psychological Assessment articles published across 3 decades (i.e., from the years 1989, 1996, and 2006). Results suggested that contemporary and past researchers' reliability reporting practices have not improved over time and generally were not consistent with good practices. In Study 2, we analyzed an archival data set to illustrate the real-life repercussions of researchers' ongoing misconstrual and misuse of reliability data. Our analyses suggested that researchers should conduct preliminary analyses of their data to determine whether their data fit the assumptions of their reliability analyses. Also, the results indicated that reliability coefficients varied across racial or ethnic and gender subgroups, and these variations had implications for whether the same depression measure should be used across groups. We concluded that the alpha-as-data perspective has implications for one's choice of psychological measures and interpretation of results, which subsequently affect conclusions and recommendations. We encourage researchers to recognize the people behind their data by adopting better practices in internal consistency reporting, analysis, and interpretation.     

Missing data techniques for structural equation modeling

As with other statistical methods, missing data often create major problems for the estimation of structural equation models (SEMs). Conventional methods such as listwise or pairwise deletion generally do a poor job of using all the available information. However, structural equation modelers are fortunate that many programs for estimating SEMs now have maximum likelihood methods for handling missing data in an optimal fashion. In addition to maximum likelihood, this article also discusses multiple imputation. This method has statistical properties that are almost as good as those for maximum likelihood and can be applied to a much wider array of models and estimation methods.     

Missing data: our view of the state of the art

Statistical procedures for missing data have vastly improved, yet misconception and unsound practice still abound. The authors frame the missing-data problem, review methods, offer advice, and raise issues that remain unresolved. They clear up common misunderstandings regarding the missing at random (MAR) concept. They summarize the evidence against older procedures and, with few exceptions, discourage their use. They present, in both technical and practical language, 2 general approaches that come highly recommended: maximum likelihood (ML) and Bayesian multiple imputation (MI). Newer developments are discussed, including some for dealing with missing data that are not MAR. Although not yet in the mainstream, these procedures may eventually extend the ML and MI methods that currently represent the state of the art.     

Missing data imputation and corrected statistics for large-scale behavioral databases

This article presents a new methodology for solving problems resulting from missing data in large-scale item performance behavioral databases. Useful statistics corrected for missing data are described, and a new method of imputation for missing data is proposed. This methodology is applied to the Dutch Lexicon Project database recently published by Keuleers, Diependaele, and Brysbaert (Frontiers in Psychology, 1, 174, 2010), which allows us to conclude that this database fulfills the conditions of use of the method recently proposed by Courrieu, Brand-D’Abrescia, Peereman, Spieler, and Rey (2011) for testing item performance models. Two application programs in MATLAB code are provided for the imputation of missing data in databases and for the computation of corrected statistics to test models.     

Evaluating and reporting data quality in eye movement research

In order to judge the degree of confidence one should have in the results of an experiment using eye movement records as data, it is necessary to have information about the quality of the eye movement data themselves. Suggestions are made for ways of assessing and reporting this information. The paper deals with three areas: characteristics of the eye movement signal, algorithms used in reducing the data, and accuracy of the eye position data. It is suggested that all studies involving eye movement data should report such information. Appendices include linear interpolation algorithms for mapping from the eye movement signal to stimulus space and a way of obtaining an index of accuracy for each data point.     

2PLM下缺失数据处理方法及其比较

项目反应理论(IRT)是用于客观测量的现代教育与心理测量理论之一,广泛用于缺失数据十分常见的大尺度测验分析。IRT中两参数逻辑斯蒂克模型(2PLM)下仅有完全随机缺失机制下缺失反应和缺失能力处理的EM算法。本研究推导2PLM下缺失反应忽略的EM 算法,并提出随机缺失机制下缺失反应和缺失能力处理的EM算法和考虑能力估计和作答反应不确定性的多重借补法。研究显示：在各种缺失机制、缺失比例和测验设计下,缺失反应忽略的EM算法和多重借补法表现理想。     

Medication status of participants in psychopathology research: selective review of current reporting practices.

The majority of Ss included in studies of psychopathology are typically receiving psychoactive medication. In view of concerns regarding the confounding effects of psychoactive medication, it appears that researchers should provide information regarding at least the drug status of the Ss studied. A review of the articles on adult psychopathology that appeared in the Journal of Abnormal Psychology during a 5-year period, however, revealed that nearly 50% of reports on Ss presumably receiving psychoactive medications failed to specify the medication status of those Ss. Few studies (15%) provided adequate information regarding the types and dosages of psychoactive drugs received by the Ss. It is argued that researchers should provide a full report of the psychoactive drugs received by research participants and, when participants are unmedicated, to indicate the reasons for drug-free status.     

Beyond statistical significance: Five principles for the new era of data analysis and reporting

A crisis of confidence in research findings in consumer psychology and other academic disciplines has led to various proposals to abandon, replace, strengthen, or supplement the null hypothesis significance testing paradigm. The proliferation of such proposals, and their often-conflicting recommendations, can increase confusion among researchers. We aim to bring some clarity by proposing five simple principles for the new era of data analysis and reporting of research in consumer psychology. We avoid adding to researchers' confusion and proposing more onerous or rigid standards. Our goal is to offer straightforward practical principles that are easy for researchers to keep in mind while analyzing their data and reporting their findings. These principles involve (1) interpreting p-values as continuous measures of the strength of evidence, (2) being aware of assumptions that determine whether one can rely on p-values, (3) using theory to establish the applicability of findings to new settings, (4) employing multiple measures of evidence and various processes to obtain them, but assigning special privilege to none, and (5) reporting procedures and findings transparently and completely. We hope that these principles provide researchers with some guidance and help to strengthen the reliability of the conclusions derived from their data, analyses, and findings.     

Measuring the prevalence of questionable research practices with incentives for truth telling

Cases of clear scientific misconduct have received significant media attention recently, but less flagrantly questionable research practices may be more prevalent and, ultimately, more damaging to the academic enterprise. Using an anonymous elicitation format supplemented by incentives for honest reporting, we surveyed over 2,000 psychologists about their involvement in questionable research practices. The impact of truth-telling incentives on self-admissions of questionable research practices was positive, and this impact was greater for practices that respondents judged to be less defensible. Combining three different estimation methods, we found that the percentage of respondents who have engaged in questionable practices was surprisingly high. This finding suggests that some questionable practices may constitute the prevailing research norm.     

计划缺失设计——通过有意缺失让研究更高效

缺失值是社会科学研究中非常普遍的现象。全息极大似然估计和多重插补是目前处理缺失值最有效的方法。计划缺失设计利用特殊的实验设计有意产生缺失值, 再用现代的缺失值处理方法来完成统计分析, 获得无偏的统计结果。计划缺失设计可用于横断面调查减少(或增加)问卷长度和纵向调查减少测量次数, 也可用于提高测量有效性。常用的计划缺失设计有三式设计和两种方法测量。     

Cyber terror: Missing in action

This article discusses the limited utility of the concepts of “cyber terrorism” or “electronic Pearl Harbor.” It contrasts the absence of cyber terror incidents with the large number of terrorist attacks and computer security incidents in recent years. It suggests that this discrepancy is explained both by the low appeal of cyber weapons to terrorists and because critical infrastructures and nations are not vulnerable to computer attacks. The article calls for a more rigorous approach to data collection for cyber security incidents, and suggests cyber security should not be seen primarily as a national security issue. Before joining CSIS, he served in the U.S. Foreign Service and worked on security, intelligence, and technology issues. Lewis’s current research covers cyber security, spectrum management, authentication, and technology’s role in national security. He received his Ph.D. from the University of Chicago in 1984.     

Best practices for missing data management in counseling psychology

This article urges counseling psychology researchers to recognize and report how missing data are handled, because consumers of research cannot accurately interpret findings without knowing the amount and pattern of missing data or the strategies that were used to handle those data. Patterns of missing data are reviewed, and some of the common strategies for dealing with them are described. The authors provide an illustration in which data were simulated and evaluate 3 methods of handling missing data: mean substitution, multiple imputation, and full information maximum likelihood. Results suggest that mean substitution is a poor method for handling missing data, whereas both multiple imputation and full information maximum likelihood are recommended alternatives to this approach. The authors suggest that researchers fully consider and report the amount and pattern of missing data and the strategy for handling those data in counseling psychology research and that editors advise researchers of this expectation.     

Uncovering the problem-solving process: cued retrospective reporting versus concurrent and retrospective reporting

This study investigated the amounts of problem-solving process information ("action," "why," "how," and "metacognitive") elicited by means of concurrent, retrospective, and cued retrospective reporting. In a within-participants design, 26 participants completed electrical circuit troubleshooting tasks under different reporting conditions. The method of cued retrospective reporting used the original computer-based task and a superimposed record of the participant's eye fixations and mouse-keyboard operations as a cue for retrospection. Cued retrospective reporting (with the exception of why information) and concurrent reporting (with the exception of metacognitive information) resulted in a higher number of codes on the different types of information than did retrospective reporting.     

Treatment of Missing Data: Beyond Ends and Means

The ethical decision making process behind the treatment of missing data has yet to be examined in the research literature in any discipline. The purpose of the current paper is to begin to discuss this decision-making process in view of a Foucauldian framework. The paper suggests how the ethical treatment of missing data should be considered from the adoption of this theoretical framework.     

Psychotherapy process and research: Missing persons and missing values

The task of ‘understanding’ in psychotherapy is compared with the mundance basis of interpersonal understanding. This implicates a discussion of three ‘principles’ which are put forward as distinguishing features of psychotherapy: ‘empathy’ (as a process, not a technique); ‘interpretation’ (particularly with regard to its relation to values); and the ‘faded person of the psychotherapist’ (related to empathy and interpretation, but particularly the power and remoteness of therapist). This allows a critical evaluation of psychotherapy research. Process research, particularly the innovative work of Robert Elliott, which seems best able to meet criticisms related to the sensitivity of research methodologies, is given particular attention. However, epistemological limitations are identified which are likely to unnecessarily distort the meaningfulness of the ‘events’ of psychotherapy.     

Attachment to Technology: The Missing Link

Technology has transformed the way in which we conduct our interactions and communications in our relationships. As smartphones and the Internet have become embedded within managing our romantic relationships, they adopt a role much like another member of the couple. The purpose of this report is to present a perspective that, much like our attachment to our partners in romantic relationships, we are developing attachments to the technologies themselves. We will briefly summarize the state of the research on technology and anxiety and the work to date on technology use based on attachment style and present characteristics and examples elucidating attachment styles to technology.