An external review of the conclusions regarding the peak direct training module

Promoting the Emergence of Advanced Knowledge—Direct Training Module (PEAK‐DTM) is a commercially available assessment and curriculum for individuals whose language skills are not developmentally adequate. In their review of PEAK‐DTM, Reed and Luiselli (2016) analyzed the extant literature on PEAK‐DTM and concluded that it has a sound and growing body of empirical support on its efficacy, usability, and psychometric properties. Similar conclusions are mirrored in the PEAK‐DTM literature and promotional material. I review these conclusions and contend that many overrate the research that backs them. Suggestions for a more rigorous research agenda on PEAK‐DTM and its related modules are provided.     

Panning for the gold in health research: Incorporating studies’ methodological quality in meta-analysis

Background: Systematic reviews now routinely assess methodological quality to gauge the validity of the included studies and of the synthesis as a whole. Although trends from higher quality studies should be clearer, it is uncertain how often meta-analyses incorporate methodological quality in models of study results either as predictors, or, more interestingly, in interactions with theoretical moderators.

Objective and methods: We survey 200 meta-analyses in three health promotion domains to examine when and how meta-analyses incorporate methodological quality.

Results: Although methodological quality assessments commonly appear in contemporary meta-analyses (usually as scales), they are rarely incorporated in analyses, and still more rarely analysed in interaction with theoretical determinants of the success of health promotions. The few meta-analyses (2.5%) that did include such an interaction analysis showed that moderator results remained significant in higher quality studies or were present only among higher quality studies. We describe how to model quality interactively with theoretically derived moderators and discuss strengths and weaknesses of this approach and in relation to current meta-analytic practice.

Conclusions: In large literatures exhibiting heterogeneous effects, meta-analyses can incorporate methodological quality and generate conclusions that enable greater confidence not only about the substantive phenomenon but also about the role that methodological quality itself plays.     

Developing mixed methods research in sport and exercise psychology: Critical reflections on five points of controversy

ObjectivesTo stimulate debate in sport and exercise psychology about the nature of mixed methods research as currently practiced and how this approach might develop in the future.DesignAn exploration of five points of controversy relating to mixed methods research.MethodA presentation of critical reflections on the following. (1) Mixing methods as a non-debate, (2) Purists, pragmatists and mixing paradigms, (3) Integrating findings and representational forms, (4) Judgment criteria and mixed methods research, and (5) Power, politics and what counts in mixed methods research.ResultsThe examples provided of mixed methods research in action indicate that a number of problematic issues regarding both process and product have been neglected.ConclusionsMixed methods research offers a number of conceptual, practical and pedagogical challenges that need to be addressed if this form of inquiry is to develop its full potential in sport and exercise psychology.     

Distribution-free tests of stochastic dominance for small samples

One variable is said to “stochastically dominate” another if the probability of observations smaller than x is greater for one variable than the other, for all x. Inferring stochastic dominance from data samples is important for many applications of econometrics and experimental psychology, but little is known about the performance of existing inferential methods. Through simulation, we show that three of the most widely used inferential methods are inadequate for use in small samples of the size commonly encountered in many applications (up to 400 observations from each distribution). We develop two new inferential methods that perform very well in a limited, but practically important, case where the two variables are guaranteed not to be equal in distribution. We also show that extensions of these new methods, and an improved version of an existing method, perform quite well in the original, unlimited case.     

A Test of Order-Constrained Hypotheses for Circular Data With Applications to Human Movement Science

Researchers studying the movements of the human body often encounter data measured in angles (e.g., angular displacements of joints). The evaluation of these circular data requires special statistical methods. The authors introduce a new test for the analysis of order-constrained hypotheses for circular data. Through this test, researchers can evaluate their expectations regarding the outcome of an experiment directly by representing their ideas in the form of a hypothesis containing inequality constraints. The resulting data analysis is generally more powerful than one using standard null hypothesis testing. Two examples of circular data from human movement science are presented to illustrate the use of the test. Results from a simulation study show that the test performs well.     

Strategy instruction for writing composition at school and at home

Abstract

The objectives of this study are 1) to analyze the effectiveness of a writing composition intervention program aimed to increase children's writing performance and other personal variables; 2) to analyze whether implementation in the environment (school and home) or responsible figures (parents and teachers) produced different effects in program effectiveness; and 3) to check whether increasing parental involvement in doing homework was sufficient to improve children's outcomes or whether prior family training is required to enable them to be efficient. The sample consisted of 112 students (5th-6th grade) and their families. 26 of them made up the EFP group (mothers implementing a writing program), 25 of them were assigned to the PAD group (mothers increasing help with writing homework), 35 to the PRO group (teachers applying a writing program) and 26 to the OC group (ordinary curriculum). We analyze the results and present and discuss our conclusions.     

Administering Quantitative Instruments With Qualitative Interviews: A Mixed Research Approach

The authors demonstrate how collecting quantitative data via psychometrically sound quantitative instruments during the qualitative interview process enhances interpretations by helping researchers better contextualize qualitative findings, specifically through qualitative dominant crossover mixed analyses. They provide an example of this strategy, whereby a baseline was established using a quantitative scale and normative data to help interpret qualitative interviews, resulting in what they call a mixed methods interview. Philosophical and practical implications are discussed.     

Data Collection in a Flat World: The Strengths and Weaknesses of Mechanical Turk Samples

Mechanical Turk (MTurk), an online labor system run by Amazon.com, provides quick, easy, and inexpensive access to online research participants. As use of MTurk has grown, so have questions from behavioral researchers about its participants, reliability, and low compensation. In this article, we review recent research about MTurk and compare MTurk participants with community and student samples on a set of personality dimensions and classic decision‐making biases. Across two studies, we find many similarities between MTurk participants and traditional samples, but we also find important differences. For instance, MTurk participants are less likely to pay attention to experimental materials, reducing statistical power. They are more likely to use the Internet to find answers, even with no incentive for correct responses. MTurk participants have attitudes about money that are different from a community sample's attitudes but similar to students' attitudes. Finally, MTurk participants are less extraverted and have lower self‐esteem than other participants, presenting challenges for some research domains. Despite these differences, MTurk participants produce reliable results consistent with standard decision‐making biases: they are present biased, risk‐averse for gains, risk‐seeking for losses, show delay/expedite asymmetries, and show the certainty effect—with almost no significant differences in effect sizes from other samples. We conclude that MTurk offers a highly valuable opportunity for data collection and recommend that researchers using MTurk (1) include screening questions that gauge attention and language comprehension; (2) avoid questions with factual answers; and (3) consider how individual differences in financial and social domains may influence results. Copyright © 2012 John Wiley & Sons, Ltd.