STATISTICAL POWER AND COST IN TRAINING EVALUATION: SOME NEW CONSIDERATIONS

Two ways to reduce the costs of training evaluation are examined. First, we examine the potential for reducing the costs of training evaluation by assigning different numbers of subjects into training and control groups. Given a total N of subjects, statistical power to detect the effectiveness of a training program can be maximized by assigning the subjects equally to training and control groups. If we take into account the costs of training evaluation, however, an unequal-group-size design with a larger total N may achieve the same level of statistical power at lower cost. We derive formulas for the optimal ratios of the control group size to the training group size for both ANOVA and ANCOVA designs, incorporating the differential costs of training and control group participation. Second, we examine the possibility that using a less expensive proxy criterion measure in place of the target criterion measure of interest when evaluating the training effectiveness can pay off. We show that using a proxy criterion increases the sample size needed to achieve a given level of statistical power, and then we describe procedures for examining the tradeoff between the costs saved by using the less expensive proxy criterion and the costs incurred by the larger sample size.     

Strategy Use in Reasoning Training With Older Adults

The relationship between strategy use and cognitive training gains on reasoning ability is examined in a sample of 393 older participants in the Seattle Longitudinal Training Study. Pre- and posttest gains on the use of strategies specific to reasoning ability were compared for the elderly trained on reasoning versus spatial orientation ability. The present study involves an objective behavioral method of measuring strategy use on tests of inductive reasoning. Results showed that participants trained on reasoning significantly increased strategy use from pre- to posttest on two reasoning outcome measures compared to participants trained on spatial orientation. Higher strategy use by inductive reasoning trainees was also associated with greater training gain on reasoning outcome measures, suggesting strategy use as a possible mechanism of training gain. In addition, young–old participants and those with higher education, irrespective of training condition, exhibited greater pre- to posttest gain in strategy use.     

Effect of concurrent visual feedback on acquisition of a weightlifting skill

Practice in front of a mirror is a common procedure for activities such as dance, gymnastics, and other sports. The purpose of this study was to examine the effect that performing with concurrent visual feedback from a mirror had on the acquisition of the power clean movement. 18 college-age males who had no prior experience with the power clean movement served as subjects who were assigned to one of two groups. One group had use of a mirror during the practice trials and the other practiced without the mirror. All subjects viewed an instructional videotape and had practice trials. All subjects were evaluated for proper technique on a pretest, a posttest without the mirror, and a posttest with the mirror. Analysis showed a significant difference between pre- and posttest performances for both groups and a significant difference between groups on the posttest performances with the mirror. Evidently the videotaped instruction was sufficient to allow both groups to improve in performance of the power clean. Differences in posttest performances with the mirror reflected the type of feedback (with or without the mirror) available during training.     

Reliability and validity of gain scores considered graphically

The ordinary gain score, g, is defined as g = x2-x1, where x1 is the pretest score and x2 is the posttest score. The present study extends and refines previous research on the reliability and validity of gain scores. Using particular values as stated in the tables and graphs, the pre- and posttest reliabilities, pre- and posttest validities, ratios of pretest to posttest standard deviations, and correlations between the pretest and posttest were varied systematically to examine the effects of these parameter configurations on gain scores' reliability and validity. Results plotted graphically provide insight via visual interpretation not easily inferred using only values from a table. One interesting finding was that the reliability of a gain score can be at a maximum when the validity is at a minimum. Another is that a high correlation between pre- and posttest was beneficial to the validity of the gain score but detrimental to its reliability. By identifying the situations in which gain scores can be reliable and valid, findings inform researchers when gain scores should or should not be used.     

Experimental Power for Indirect Effects in Group-randomized Studies with Group-level Mediators

Mediation analyses have provided a critical platform to assess the validity of theories of action across a wide range of disciplines. Despite widespread interest and development in these analyses, literature guiding the design of mediation studies has been largely unavailable. Like studies focused on the detection of a total or main effect, an important design consideration is the statistical power to detect indirect effects if they exist. Understanding the sensitivity to detect indirect effects is exceptionally important because it directly influences the scale of data collection and ultimately governs the types of evidence group-randomized studies can bring to bear on theories of action. However, unlike studies concerned with the detection of total effects, literature has not established power formulas for detecting multilevel indirect effects in group-randomized designs. In this study, we develop closed-form expressions to estimate the variance of and the power to detect indirect effects in group-randomized studies with a group-level mediator using two-level linear models (i.e., 2-2-1 mediation). The results suggest that when carefully planned, group-randomized designs may frequently be well positioned to detect mediation effects with typical sample sizes. The resulting power formulas are implemented in the R package PowerUpR and the PowerUp!-Mediator software (causalevaluation.org).     

Some cautions regarding statistical power in split-plot designs

We show that if overall sample size and effect size are held constant, the power of theF test for a one-way analysis of variance decreases dramatically as the number of groups increases. This reduction in power is even greater when the groups added to the design do not produce treatment effects. If a second independent variable is added to the design, either a split-plot or a completely randomized design may be employed. For the split-plot design, we show that the power of theF test on the betweengroups factor decreases as the correlation across the levels of the within-groups factor increases. The attenuation in between-groups power becomes more pronounced as the number of levels of the withingroups factor increases. Sample size and total cost calculations are required to determine whether the split-plot or completely randomized design is more efficient in a particular application. The outcome hinges on the cost of obtaining (or recruiting) a single subject relative to the cost of obtaining a single observation: We call this thesubject-to-observation cost (SOC) ratio. Split-plot designs are less costly than completely randomized designs only when the SOC ratio is high, the correlation across the levels of the within-groups factor is low, and the number of such levels is small.     

Can Visual Illusions Be Used to Facilitate Sport Skill Learning?

Recently it has been reported that practicing putting with visual illusions that make the hole appear larger than it actually is leads to longer-lasting performance improvements. Interestingly, from a motor control and learning perspective, it may be possible to actually predict the opposite to occur, as facing a smaller appearing target should enforce performers to be more precise. To test this idea the authors invited participants to practice an aiming task (i.e., a marble-shooting task) with either a visual illusion that made the target appear larger or a visual illusion that made the target appear smaller. They applied a pre–post test design, included a control group training without any illusory effects and increased the amount of practice to 450 trials. In contrast to earlier reports, the results revealed that the group that trained with the visual illusion that made the target look smaller improved performance from pre- to posttest, whereas the group practicing with visual illusions that made the target appear larger did not show any improvements. Notably, also the control group improved from pre- to posttest. The authors conclude that more research is needed to improve our understanding of whether and how visual illusions may be useful training tools for sport skill learning.     

Facilitating children's understanding of misinterpretation: explanatory efforts and improvements in perspective taking

The authors investigated children's understanding of how mistaken beliefs can arise through misinterpretation of ambiguous information. Children (N = 91), aged 4 to 5 years, were given pre- and posttests on their ability to infer a puppet's interpretation of a restricted-view drawing after the puppet had been led to an erroneous expectation about the drawing's identity. Before the posttest, the children received either self-explanation training or other-explanation training in which they explained the source of their own or a puppet's misinterpretations of drawings; a control group received no training. The children who received training improved from pre- to posttest, and those who had practiced explaining misinterpretations by referring to previously viewed pictures or to features of a target picture showed the greatest improvement. These results indicate that learning to explain misinterpretations can help children recognize situations in which misinterpretations are likely to occur.     

The Impact of PREP Training on Marital Conflicts Reduction: A Randomized Controlled Trial With Iranian Distressed Couples

The purpose of this study was to examine the efficacy of the Prevention and Relationship Education Program (PREP) training on marital conflict and marital satisfaction among a sample of distressed couples in Iran. The research procedure was experimental with a pretest, posttest, and follow-up design, including a control group. The sample included 76 volunteer couples among a sample of distressed couples who were randomly selected and assigned to the experimental or control group. They completed demographic questions, the Marital Conflicts questionnaire, and a revised Marital Satisfaction Inventory in pretest, posttest and at the 1-year follow-up. Results showed that PREP training effectively led to decreased marital conflict and improvement of marital satisfaction of couples at posttest and at the 1-year follow-up. The result of covariance analysis showed significant differences between the experimental and the control groups' marital conflict and marital satisfaction at posttest and at the 1-year follow-up.     

On the power of multivariate latent growth curve models to detect correlated change

We evaluated the statistical power of single-indicator latent growth curve models (LGCMs) to detect correlated change between two variables (covariance of slopes) as a function of sample size, number of longitudinal measurement occasions, and reliability (measurement error variance). Power approximations following the method of Satorra and Saris (1985) were used to evaluate the power to detect slope covariances. Even with large samples (N = 500) and several longitudinal occasions (4 or 5), statistical power to detect covariance of slopes was moderate to low unless growth curve reliability at study onset was above .90. Studies using LGCMs may fail to detect slope correlations because of low power rather than a lack of relationship of change between variables. The present findings allow researchers to make more informed design decisions when planning a longitudinal study and aid in interpreting LGCM results regarding correlated interindividual differences in rates of development.     

Determining the sample size for a replication attempt: A short and simple microcomputer program

Replication studies frequently fail to detect genuine effects because too few subjects are employed to yield an acceptable level of power. To remedy this situation, a method of sample size determination in replication attempts is described that uses information supplied by the original experiment to establish a distribution of probable effect sizes. The sample size to be employed is that which supplies an expected power of the desired amount over the distribution of probable effect sizes. The method may be used in replication attempts involving the comparison of means, the comparison of correlation coefficients, and the comparison of proportions. The widely available equation-solving program EUREKA provides a rapid means of executing the method on a microcomputer. Only ten lines are required to represent the method as a set of equations in EUREKA’s language. Such an equation file is readily modified, so that even inexperienced users find it a straightforward means of obtaining the sample size for a variety of designs.     

Web-based training improves on-field offside decision-making performance

ObjectiveThe present study examined to what extent off-field offside decision-making training transfers to real-life offside situations.Design/methodsEighteen Belgian assistant referees were included in the experiment. Ten assistant referees (i.e., training group) were exposed to a pre- and posttest and, in between, four off-field offside training sessions via a web-based training protocol. The remaining eight assistant referees participated in the control group and only completed the pre- and posttest. During both test sessions, which were conducted separately for each group, both an on- and off-field offside decision-making test was completed.ResultsFirst, an increase in response accuracy and a decrease in flag errors were observed for the training group from pre- to posttest in both the on- and off-field offside test. Second, only the training group improved in the recall and recognition accuracy of the position of the receiving attacker at the moment of the pass.ConclusionsThis study demonstrates that perceptual-cognitive skill training results in a positive and direct transfer to on-field offside decisions. Therefore, the structure and the content of the current training intervention mimics the perceptual difficulties of real-match situations and can help the assistant referees to mediate and enhance their offside decision-making skills, both on- and off-field.     

Reproducible research in sport and exercise psychology: The role of sample sizes

ObjectivesWe aim to introduce the discussion on the crisis of confidence to sport and exercise psychology. We focus on an important aspect of this debate, the impact of sample sizes, by assessing sample sizes within sport and exercise psychology. Researchers have argued that publications in psychological research contain numerous false-positive findings and inflated effect sizes due to small sample sizes.MethodWe analyse the four leading journals in sport and exercise psychology regarding sample sizes of all quantitative studies published in these journals between 2009 and 2013. Subsequently, we conduct power analyses.ResultsA substantial proportion of published studies does not have sufficient power to detect effect sizes typical for psychological research. Sample sizes and power vary between research designs. Although many correlational studies have adequate sample sizes, experimental studies are often underpowered to detect small-to-medium effects.ConclusionsAs sample sizes are small, research in sport and exercise psychology may suffer from false-positive results and inflated effect sizes, while at the same time failing to detect meaningful small effects. Larger sample sizes are warranted, particularly in experimental studies.     

Multilevel factorial experiments for developing behavioral interventions: power, sample size, and resource considerations

Factorial experimental designs have many potential advantages for behavioral scientists. For example, such designs may be useful in building more potent interventions by helping investigators to screen several candidate intervention components simultaneously and to decide which are likely to offer greater benefit before evaluating the intervention as a whole. However, sample size and power considerations may challenge investigators attempting to apply such designs, especially when the population of interest is multilevel (e.g., when students are nested within schools, or when employees are nested within organizations). In this article, we examine the feasibility of factorial experimental designs with multiple factors in a multilevel, clustered setting (i.e., of multilevel, multifactor experiments). We conduct Monte Carlo simulations to demonstrate how design elements-such as the number of clusters, the number of lower-level units, and the intraclass correlation-affect power. Our results suggest that multilevel, multifactor experiments are feasible for factor-screening purposes because of the economical properties of complete and fractional factorial experimental designs. We also discuss resources for sample size planning and power estimation for multilevel factorial experiments. These results are discussed from a resource management perspective, in which the goal is to choose a design that maximizes the scientific benefit using the resources available for an investigation.     

Point-biserial correlation: Interval estimation,hypothesis testing,meta-analysis,and sample size determination

The point-biserial correlation is a commonly used measure of effect size in two-group designs. New estimators of point-biserial correlation are derived from different forms of a standardized mean difference. Point-biserial correlations are defined for designs with either fixed or random group sample sizes and can accommodate unequal variances. Confidence intervals and standard errors for the point-biserial correlation estimators are derived from the sampling distributions for pooled-variance and separate-variance versions of a standardized mean difference. The proposed point-biserial confidence intervals can be used to conduct directional two-sided tests, equivalence tests, directional non-equivalence tests, and non-inferiority tests. A confidence interval for an average point-biserial correlation in meta-analysis applications performs substantially better than the currently used methods. Sample size formulas for estimating a point-biserial correlation with desired precision and testing a point-biserial correlation with desired power are proposed. R functions are provided that can be used to compute the proposed confidence intervals and sample size formulas.     

Developing Anticipation Skills in Tennis Using On-Court Instruction: Perception versus Perception and Action

On-court instruction involving either Perception–action training or Perception-only training was used to improve anticipation skill in novice tennis players. A technical instruction group acted as a control. Participants' ability to anticipate an opponent's serve was assessed pre- and posttest using established on-court measures involving frame-by-frame video analysis. The perception–action and perception-only groups significantly improved their anticipatory performance from pretest to posttest. No pretest-to-posttest differences in anticipation skill were reported for the technical instruction group. The ability to anticipate an opponent's serve can be improved through on-court instruction where the relationship between key postural cues and subsequent performance is highlighted, and both practice and feedback are provided. No significant differences were observed between the perception–action and perception-only training groups, implying that either mode of training may be effective in enhancing perceptual skill in sport.     

Program of training in solution of practical problems applied to people with intellectual disability

The lack of programs to train people in practical problem-solving is one of the challenges that professionals who attend people with mental deficiency must cope with. The main goal of this study is to assess the effects of a program designed to improve these skills, aimed at people with intellectual discapacity. The sample was made up of 66 subjects, aged between 17 and 36 years old. The program was based on the use of examples of fictitious characters who undergo situations that are similar to those of the subjects to whom the program is administered, and techniques to improve skills were generated from these situations. In order to achieve the goal, a classic pre- posttest design was used, with an experimental and a control group. The results show positive effects in most of the variables considered as a consequence of the administration of the program.     

Principles for Designing Randomized Preventive Trials in Mental Health: An Emerging Developmental Epidemiology Paradigm

An emerging population-based paradigm is now being used to guide the design of preventive trials used to test developmental models. We discuss elements of the designs of several ongoing randomized preventive trials involving reduction of risk for children of divorce, for children who exhibit behavioral or learning problems, and for children whose parents are being treated for depression. To test developmental models using this paradigm, we introduce three classes of design issues: design for prerandomization, design for intervention, and design for postintervention. For each of these areas, we present quantitative results from power calculations. Both scientific and cost implications of these power calculations are discussed in terms of variation among subjects on preintervention measures, unit of intervention, assignment, balancing, number of pretest and posttest measures, and the examination of moderation effects.     

The effect of multiple indicators on the power to detect inter‐individual differences in change

Hertzog et al. evaluated the statistical power of linear latent growth curve models (LGCMs) to detect individual differences in change, i.e., variances of latent slopes, as a function of sample size, number of longitudinal measurement occasions, and growth curve reliability. We extend this work by investigating the effect of the number of indicators per measurement occasion on power. We analytically demonstrate that the positive effect of multiple indicators on statistical power is inversely related to the relative magnitude of occasion‐specific latent residual variance and is independent of the specific model that constitutes the observed variables, in particular of other parameters in the LGCM. When designing a study, researchers have to consider trade‐offs of costs and benefits of different design features. We demonstrate how knowledge about power equivalent transformations between indicator measurement designs allows researchers to identify the most cost‐efficient research design for detecting parameters of interest. Finally, we integrate different formal results to exhibit the trade‐off between the number of measurement occasions and number of indicators per occasion for constant power in LGCMs.