Maximizing data quality and shortening survey time: Three-form planned missing data survey design

Simulation studies have shown the three-form planned missing data design efficiently collects high quality data while reducing participant burden. This methodology is rarely used in sport and exercise psychology. Therefore, we conducted a re-sampling study with existing sport and exercise psychology survey data to test how three-form planned missing data survey design implemented with different item distribution approaches effect constructs’ internal measurement structure and validity. Results supported the efficacy of the three-form planned missing data survey design for cross-sectional data collection. Sample sizes of at least 300 (i.e., 100 per form) are recommended for having unbiased parameter estimates. It is also recommended items be distributed across survey forms to have representation of each facet of a construct on every form, and that a select few of these items be included across all survey forms. Further guidelines for three-form surveys based upon the results of this resampling study are provided.     

计划缺失设计——通过有意缺失让研究更高效

缺失值是社会科学研究中非常普遍的现象。全息极大似然估计和多重插补是目前处理缺失值最有效的方法。计划缺失设计利用特殊的实验设计有意产生缺失值, 再用现代的缺失值处理方法来完成统计分析, 获得无偏的统计结果。计划缺失设计可用于横断面调查减少(或增加)问卷长度和纵向调查减少测量次数, 也可用于提高测量有效性。常用的计划缺失设计有三式设计和两种方法测量。     

Planned missing data designs in psychological research

The authors describe 2 efficiency (planned missing data) designs for measurement: the 3-form design and the 2-method measurement design. The 3-form design, a kind of matrix sampling, allows researchers to leverage limited resources to collect data for 33% more survey questions than can be answered by any 1 respondent. Power tables for estimating correlation effects illustrate the benefit of this design. The 2-method measurement design involves a relatively cheap, less valid measure of a construct and an expensive, more valid measure of the same construct. The cost effectiveness of this design stems from the fact that few cases have both measures, and many cases have just the cheap measure. With 3 brief simulations involving structural equation models, the authors show that compared with the same-cost complete cases design, a 2-method measurement design yields lower standard errors and a higher effective sample size for testing important study parameters. With a large cost differential between cheap and expensive measures and small effect sizes, the benefits of the design can be enormous. Strategies for using these 2 designs are suggested.     

Individual differences methods for randomized experiments

Experiments allow researchers to randomly vary the key manipulation, the instruments of measurement, and the sequences of the measurements and manipulations across participants. To date, however, the advantages of randomized experiments to manipulate both the aspects of interest and the aspects that threaten internal validity have been primarily used to make inferences about the average causal effect of the experimental manipulation. This article introduces a general framework for analyzing experimental data to make inferences about individual differences in causal effects. Approaches to analyzing the data produced by a number of classical designs and 2 more novel designs are discussed. Simulations highlight the strengths and weaknesses of the data produced by each design with respect to internal validity. Results indicate that, although the data produced by standard designs can be used to produce accurate estimates of average causal effects of experimental manipulations, more elaborate designs are often necessary for accurate inferences with respect to individual differences in causal effects. The methods described here can be diversely applied by researchers interested in determining the extent to which individuals respond differentially to an experimental manipulation or treatment and how differential responsiveness relates to individual participant characteristics.     

Measuring Creativity with Planned Missing Data

This paper introduces a method for the assessment of creativity that relies on creativity tasks, a subjective evaluation procedure, and a planned missing data design that offers a drastic reduction in the overall implementation costs (administration time and scoring procedure). This method was tested on a sample of 149 people, using three creativity tasks as a basis. Participants were instructed to produce several ideas in each task and then to select what they considered to be their best two ideas (i.e., “Top 2” procedure; Silvia, Winterstein, Willse, Barona, et al., Psychology of Aesthetics, Creativity, and the Arts, 2 , 2008 and 68). These ideas were then evaluated by a panel of peers and experts. Creativity ratings were analyzed with structural equations; measurement models were estimated for each task and correlations between factor-scores across the three tasks were investigated. Further insights regarding validity are provided through systematic investigation of the relationship between fluency scores, creativity ratings, intelligence tasks, self-reported idea generation abilities, and creative activities and achievements. Overall, the results support the viability of this new approach, providing evidence of convergent and discriminant validity. They are discussed in relation to past research and avenues for further extension are proposed.     

The measurement of participant-reported memory across diverse populations and settings: a systematic review and meta-analysis of the Multifactorial Memory Questionnaire

ABSTRACT

The Multifactorial Memory Questionnaire (MMQ) is a participant-reported measure of memory satisfaction, ability, and strategy use. Initially validated with healthy older adults, it has since been used in many different populations and settings for a variety of purposes. We conducted a systematic review and meta-analysis of the measurement properties of the MMQ across multiple, diverse studies. Methods: The study was designed using the Consensus-Based Standards for the Selection of Health Measurement Instruments and the Preferred Reporting Items for Systematic Reviews and Meta-Analyses. We calculated aggregate statistics and evaluated the methodological quality of 29 studies retrieved from PsycINFO, MEDLINE, EMBASE, and Web of Science. Results: Analyses revealed high-quality evidence for internal consistency, stability, measurement error, convergent validity, and known-groups validity of the three MMQ scales. There was moderate-quality evidence for responsiveness and structural validity, with some studies identifying separate factors for internal and external memory strategy use. Measurement properties were similar across languages, participant samples, and study designs. Conclusions: The MMQ is a valid, reliable, and responsive measure across diverse settings and populations. Future research is needed to determine whether more detailed information can be obtained from the scales, specifically, internal versus external strategy use.     

Experimental Power for Indirect Effects in Group-randomized Studies with Group-level Mediators

Mediation analyses have provided a critical platform to assess the validity of theories of action across a wide range of disciplines. Despite widespread interest and development in these analyses, literature guiding the design of mediation studies has been largely unavailable. Like studies focused on the detection of a total or main effect, an important design consideration is the statistical power to detect indirect effects if they exist. Understanding the sensitivity to detect indirect effects is exceptionally important because it directly influences the scale of data collection and ultimately governs the types of evidence group-randomized studies can bring to bear on theories of action. However, unlike studies concerned with the detection of total effects, literature has not established power formulas for detecting multilevel indirect effects in group-randomized designs. In this study, we develop closed-form expressions to estimate the variance of and the power to detect indirect effects in group-randomized studies with a group-level mediator using two-level linear models (i.e., 2-2-1 mediation). The results suggest that when carefully planned, group-randomized designs may frequently be well positioned to detect mediation effects with typical sample sizes. The resulting power formulas are implemented in the R package PowerUpR and the PowerUp!-Mediator software (causalevaluation.org).     

Neuropsychological changes in children with cancer: The treatment of missing data in longitudinal studies

Researchers conducting longitudinal studies with children or adults are inevitably confronted with problems of attrition and missing data. Missing data in longitudinal studies is frequently handled by excluding from analyses those cases for whom data are incomplete. This approach to missing data is not optimal. On the one hand, if data are missing at random, then dropping incomplete cases ignores information collected on those cases that could be used to improve estimates of population parameters (e.g., means, variances, covariances, and growth rates) and improve the power of significance tests of statistical hypotheses. On the other hand, if data are not missing at random, then dropping incomplete cases leads to biased parameter estimates and hypothesis tests that may be internally and externally invalid. This study uses three years of follow-up data from a longitudinal investigation of neuropsychological outcomes of cancer in children to demonstrate the problems presented by missing data in repeated measures designs and some solutions. In evaluating potential biasing effects of attrition, the study extends previous research on neuropsychological outcomes in pediatric cancer by inclusion of patients whose disease had relapsed, and by comparison of surviving and nonsurviving patients. Although the data presented have specific relevance to the study of neuropsychological outcome in pediatric cancer, the problems of missing data and the solutions presented are relevant to a wide variety of diseases and conditions of interest to researchers in child and adult neuropsychology.     

The influences of intrinsic motivation on execution of social behaviour within the theory of planned behaviour

The present study examined the utility of two forms of measurement of intrinsic motivation in increasing the predictive validity of the theory of planned behaviour. Self‐report questionnaires were administered to school pupils (n = 174), University students (n = 129) and adults (n = 157). The data were analysed using confirmatory factor analysis and regression analysis. Confirmatory analysis supported discriminant validity between Forms A and B measures of intrinsic motivation. In addition, hierarchical regression analysis demonstrated that Form B measure of intrinsic motivation increased effectiveness of the theory of planned behaviour in predicting intentions and social behaviour. Further, the regression analysis showed that age and past behaviour did not reduce the effects observed for intrinsic motivation. It is recommended that intrinsic motivation could increase the predictive utility of the theory of planned behaviour. Copyright © 2006 John Wiley & Sons, Ltd.     

A systematic review of physical guidance procedures as an open‐mouth prompt to increase acceptance for children with pediatric feeding disorders

This article reviews behavioral treatments of pediatric feeding disorders using physical guidance procedures as an open‐mouth prompt (i.e., jaw prompt, finger prompt, Nuk prompt, side deposit) to increase food acceptance. We identified 9 articles containing 35 systematic evaluations. We coded participant and study characteristics and assessed the experimental rigor, quality, and outcomes of each evaluation. Of the high‐quality research present, the finger prompt variation and side deposit reliably increased acceptance. We found mixed results on the efficacy of the jaw prompt, although it was the most widely researched procedure. Further, authors reported interobserver agreement for 100% of the participants, procedural integrity for 60%, social validity for 80%, fading for 5%, and follow‐up for 55%. Based on the invasive nature of physical guidance, we provide recommendations for researchers and clinicians to increase the quality of their treatment evaluations. We discuss limitations, implications for practice, and future research.     

Stop With the Questions Already! Does Data Quality Suffer for Scales Positioned Near the End of a Lengthy Questionnaire?

Research questionnaires frequently include dozens—if not hundreds—of self-report items. Lengthy questionnaires, however, are often a necessity. In some cases, they are needed to assess the many variables found in a complex model; in other cases, they are the result of the inclusion of a single lengthy measure. This raises an important question: Do participants provide accurate responses to measures positioned at the end of a lengthy questionnaire? One possibility is that participants experience fatigue during questionnaire completion, leading them to engage in careless responding, and thus compromising the accuracy of their responses. Another possibility is that even the longest research questionnaires are generally too short to evoke participant fatigue. This latter possibility suggests that participants are largely able to maintain their attention while completing most questionnaires. Given the lack of clarity on this issue, we conducted two experiments (Study 1 N = 244; Study 2 N = 461) in which we randomly assigned each participant to complete a block of target scales at either the beginning or the end of a lengthy (> 300-item) questionnaire. Each participant also recruited an informant who provided reports of the participant’s personality, attitudes, and behaviors. These informant data allowed us to examine the effects of the experimental manipulation on the target scales’ convergent and criterion-related validity. The findings of both studies indicated that the target scales performed similarly across the two conditions. Given the ubiquity of lengthy questionnaires, these findings have far-reaching practical implications.

     

Fisher transformations for correlations corrected for selection and missing data

The validity of a test is often estimated in a nonrandom sample of selected individuals. To accurately estimate the relation between the predictor and the criterion we correct this correlation for range restriction. Unfortunately, this corrected correlation cannot be transformed using Fisher'sZ transformation, and asymptotic tests of hypotheses based on small or moderate samples are not accurate. We developed a Fisherr toZ transformation for the corrected correlation for each of two conditions: (a) the criterion data were missing due to selection on the predictor (the missing data were MAR); and (b) the criterion was missing at random, not due to selection (the missing data were MCAR). The twoZ transformations were evaluated in a computer simulation. The transformations were accurate, and tests of hypotheses and confidence intervals based on the transformations were superior to those that were not based on the transformations.     

Quality Vs. Quantity: Assessing Behavior Change over Time

Beyond the typical design factors that impact a study’s power (e.g., participant sample size), planning longitudinal research involves additional considerations such as assessment frequency and participant retention. Because this type of research relies so strongly on individual commitment, investigators must be judicious in determining how much information is necessary to study the phenomena in question; collecting too little information will render the data less useful, but requiring excessive participant investment will likely lower participation rates. We conducted a simulation study to empirically examine statistical power and the trade-off between assessment quality (as a function of instrument length) and assessment frequency across a number of sample sizes with intermittently missing data or attrition. Results indicated that reductions in power resulting from shorter, less reliable measurements can be at least somewhat offset by increasing assessment frequency. Because study planning involves a number of factors competing for finite resources, equations were derived to find the balance points between pairs of design characteristics affecting statistical power. These equations allow researchers to calculate the amount that a particular design factor (e.g., assessment frequency) would need to increase to result in the same improvement in power as increasing an alternative factor (e.g., measurement reliability. Applications for the equations are discussed.     

Bayes factor design analysis: Planning for compelling evidence

A sizeable literature exists on the use of frequentist power analysis in the null-hypothesis significance testing (NHST) paradigm to facilitate the design of informative experiments. In contrast, there is almost no literature that discusses the design of experiments when Bayes factors (BFs) are used as a measure of evidence. Here we explore Bayes Factor Design Analysis (BFDA) as a useful tool to design studies for maximum efficiency and informativeness. We elaborate on three possible BF designs, (a) a fixed-n design, (b) an open-ended Sequential Bayes Factor (SBF) design, where researchers can test after each participant and can stop data collection whenever there is strong evidence for either \(\mathcal {H}_{1}\) or \(\mathcal {H}_{0}\), and (c) a modified SBF design that defines a maximal sample size where data collection is stopped regardless of the current state of evidence. We demonstrate how the properties of each design (i.e., expected strength of evidence, expected sample size, expected probability of misleading evidence, expected probability of weak evidence) can be evaluated using Monte Carlo simulations and equip researchers with the necessary information to compute their own Bayesian design analyses.     

The Theory of Planned Behavior: Some Measurement Issues Concerning Belief-Based Variables

The theory of planned behavior presents clear operational definitions of attitudes, subjective norm, perceived behavioral control, and each of their corresponding belief-based measures. Theoretically, the direct and indirect measures of a given construct must be closely correlated. Empirical results, however, indicate that this is not always the case. In the present study, 2 of the factors that could be responsible for this situation-namely, the scaling of the variables defining each belief-based construct and the adequacy of using an expectancy-value model within the belief-based measures-were verified among a data set of 16 studies concerned with the application of the theory of planned behavior to the field of health. The results indicate that the scaling method used affected the correlation coefficients between indirect and direct measures. However, the face validity of these scaling methods must be demonstrated. The results also support the idea that, in most cases, using the expectancy-value model is no better than using only one arm of the belief-based measure.     

Data Collection via Online Platforms: Challenges and Recommendations for Future Research

Online platforms such as Amazon's Mechanical Turk (MTurk) are increasingly used by researchers to collect survey and experimental data. Yet, such platforms often represent a tumultuous terrain for both researchers and reviewers. Researchers have to navigate the complexities of obtaining representative samples from online participant cohorts, ensuring data quality, ethically incentivizing participant engagement, and maintaining transparency. Reviewers, on the other hand, have to navigate the complexities of evaluating the efficacy of such data collection and execution efforts in answering important research questions. In order to provide clarity to these issues, this article provides researchers and reviewers with a series of recommendations for effectively executing and evaluating data collection via online platforms, respectively.     

Small is beautiful: In defense of the small-<Emphasis Type="Italic">N</Emphasis> design

The dominant paradigm for inference in psychology is a null-hypothesis significance testing one. Recently, the foundations of this paradigm have been shaken by several notable replication failures. One recommendation to remedy the replication crisis is to collect larger samples of participants. We argue that this recommendation misses a critical point, which is that increasing sample size will not remedy psychology’s lack of strong measurement, lack of strong theories and models, and lack of effective experimental control over error variance. In contrast, there is a long history of research in psychology employing small-N designs that treats the individual participant as the replication unit, which addresses each of these failings, and which produces results that are robust and readily replicated. We illustrate the properties of small-N and large-N designs using a simulated paradigm investigating the stage structure of response times. Our simulations highlight the high power and inferential validity of the small-N design, in contrast to the lower power and inferential indeterminacy of the large-N design. We argue that, if psychology is to be a mature quantitative science, then its primary theoretical aim should be to investigate systematic, functional relationships as they are manifested at the individual participant level and that, wherever possible, it should use methods that are optimized to identify relationships of this kind.     

A maximum likelihood approach to test validation with missing and censored dependent variables

A maximum likelihood approach is described for estimating the validity of a test (x) as a predictor of a criterion variable (y) when there are both missing and censoredy scores present in the data set. The missing data are due to selection on a latent variable (y _s ) which may be conditionally related toy givenx. Thus, the missing data may not be missing random. The censoring process in due to the presence of a floor or ceiling effect. The maximum likelihood estimates are constructed using the EM algorithm. The entire analysis is demonstrated in terms of hypothetical data sets.     

Modeling Concordance Correlation Coefficient for Longitudinal Study Data

Measures of agreement are used in a wide range of behavioral, biomedical, psychosocial, and health-care related research to assess reliability of diagnostic test, psychometric properties of instrument, fidelity of psychosocial intervention, and accuracy of proxy outcome. The concordance correlation coefficient (CCC) is a popular measure of agreement for continuous outcomes. In modern-day applications, data are often clustered, making inference difficult to perform using existing methods. In addition, as longitudinal study designs become increasingly popular, missing data have become a serious issue, and the lack of methods to systematically address this problem has hampered the progress of research in the aforementioned fields. In this paper, we develop a novel approach to tackle the complexities involved in addressing missing data and other related issues for performing CCC analysis within a longitudinal data setting. The approach is illustrated with both real and simulated data.