The Random Stimulus Design

In the 1960s, a methodological advancement of significant proportion occured with the development of single case research designs in the field of appleid behavior analysis. Among the methods popularized were the multiple baseline, reversal, and, more recently, the alternating treatment designs. Despite the flexibility and wide applicability of each of these single subject designs, there are conditions under which none of them are appropriate. For example, when limited subjects are available and when repeated presentation of the same stimuli result in improvement due to practice effects of decremental effects due to boredom or habituation, problems arise. To offset this difficulty, the investigators describe a new single case design, the random stimulus design. This design is so named becuase an item pool of equal stimuli is established and then items are randomly selected and assigned to one of several cohorts. Cohorts are then assessed in the typical withdrawal or reversal manner.     

A Gradual Effects Model for Single-Case Designs

Single-case designs are a class of repeated measures experiments used to evaluate the effects of interventions for small or specialized populations, such as individuals with low-incidence disabilities. There has been growing interest in systematic reviews and syntheses of evidence from single-case designs, but there remains a need to further develop appropriate statistical models and effect sizes for data from the designs. We propose a novel model for single-case data that exhibit nonlinear time trends created by an intervention that produces gradual effects, which build up and dissipate over time. The model expresses a structural relationship between a pattern of treatment assignment and an outcome variable, making it appropriate for both treatment reversal and multiple baseline designs. It is formulated as a generalized linear model so that it can be applied to outcomes measured as frequency counts or proportions, both of which are commonly used in single-case research, while providing readily interpretable effect size estimates such as log response ratios or log odds ratios. We demonstrate the gradual effects model by applying it to data from a single-case study and examine the performance of proposed estimation methods in a Monte Carlo simulation of frequency count data.     

Addressing two blind spots of commonly used experimental designs: The Highly‐Repeated Within‐Person approach

Two well documented but still neglected blind spots of often‐used study designs limit a researcher's ability to make inferences about psychological phenomenon. First, typical designs focus on effects of conditions at the group level and are not able to assess the extent to which effects characterize each participant in the study. This blind spot can lead to erroneous (or incomplete) conclusions about the effects of manipulations both for a given participant and at the group level. Second, commonly used research designs often use a limited sample of stimuli, constraining conclusions to the particular stimuli. This blind spot can lead to non‐replication when different stimuli are used. We propose that the Highly‐Repeated Within‐Person (HRWP) approach helps mitigate these limitations. Using a study on the effects of anti‐smoking messages, we illustrate how the HRWP approach helps alert researchers when the conclusions at the group level may not apply to all (or any) participant, quantifies the heterogeneity of effects of manipulations across people, and increases confidence regarding the generalizability of the effects. We discuss how the HRWP approach may help conceptualize issues of replicability in a new light.     

Is advance reconfiguration in voluntary task switching affected by the design employed?

In task switching, when the amount of preparation time is increased, a reduction in switch cost or RISC effect is observed. This RISC effect is frequently attributed to advance reconfiguration processes. In the explicit task-cueing procedure, RISC effects are observed when varying the preparation time within participants but not when varying the preparation time across participants—a finding suggesting that RISC effects in the explicit task-cueing procedure are restricted to specific designs. The present study investigated RISC effects in voluntary task switching and compared RISC effects in a within-subjects design with RISC effects in a between-subjects design. Our results indicate that RISC effects are present in both designs. We conclude that advance reconfiguration in voluntary task switching is robust.     

Fractional designs: A simulation study of usefulness in the social sciences

Fractional designs can be extremely useful in social science research, especially when a large number of factors is involved. Reluctance for the use of fractional designs seems to be warranted for two reasons: (1) In the social sciences, the amount of measurement error is often large, which may decrease the power, and (2) higher order interactions are assumed to be nonsignificant, which is difficult to guarantee without sufficient research. This simulation study shows the effects of measurement error and assumption violations under various conditions. It is concluded that fractional designs handle measurement error gracefully and that they are as powerful as a full design when equal degrees of freedom are available. Significant interaction effects can cause serious problems, especially in situations with low or intermediate measurement error, and can lead to erroneous conclusions. Only when estimated confounded effects are clearly not significant, the chance of a wrong decision is reasonably small. Therefore, fractional designs are especially warranted for the exclusion of irrelevant factors. However, we note pitfalls in the use of Version 1.0 of the program Trail Run from SPSS, Inc., to implement the procedures.     

INTERACTION EFFECTS IN MULTIELEMENT DESIGNS: INEVITABLE,DESIRABLE, AND IGNORABLE

A single-subject design often used to compare the effectiveness of two or more independent variables (like treatment programs) is the multielement (alternating treatments or simultaneous treatments) design. Variants of this design approximate the concurrent comparison of the effects of two or more variables (or levels of variables) by programming the variables (or levels) in rapid alternation, typically across or within daily sessions. Properly combined with conventional reversal designs, these designs can also display a variety of interaction effects, some of them worrisome, others highly desirable for the future development of the field. A worrisome model is the possibility that when Treatment B alternates rapidly with Treatment C, the effects of each will not be the same as when each is the only treatment used. A desirable model is the use of the multielement design as a fast-paced component of an otherwise conventional reversal design examining contextual control of some relationship: the possibility that some behavior responds differently to Controlling Variables A and B in Context X than in Context Y. This second possibility opens single-subject designs to the more efficient examination of all interactive effects and is highly desirable, considering the prevalence and importance of interactions in determining the limits and the generality of currently understood behavioral phenomena.     

Testing Mean Differences among Groups: Multivariate and Repeated Measures Analysis with Minimal Assumptions

To date, there is a lack of satisfactory inferential techniques for the analysis of multivariate data in factorial designs, when only minimal assumptions on the data can be made. Presently available methods are limited to very particular study designs or assume either multivariate normality or equal covariance matrices across groups, or they do not allow for an assessment of the interaction effects across within-subjects and between-subjects variables. We propose and methodologically validate a parametric bootstrap approach that does not suffer from any of the above limitations, and thus provides a rather general and comprehensive methodological route to inference for multivariate and repeated measures data. As an example application, we consider data from two different Alzheimer’s disease (AD) examination modalities that may be used for precise and early diagnosis, namely, single-photon emission computed tomography (SPECT) and electroencephalogram (EEG). These data violate the assumptions of classical multivariate methods, and indeed classical methods would not have yielded the same conclusions with regards to some of the factors involved.     

Regression analyses of repeated measures data in cognitive research

Repeated measures designs involving nonorthogonal variables are being used with increasing frequency in cognitive psychology. Researchers usually analyze the data from such designs inappropriately, probably because the designs are not discussed in standard textbooks on regression. Two commonly used approaches to analyzing repeated measures designs are considered in this article. It is argued that both approaches use inappropriate error terms for testing the effects of independent variables. A more appropriate analysis is presented, and two alternative computational procedures for the analysis are illustrated.     

Modeling external events in the three-level analysis of multiple-baseline across-participants designs: A simulation study

In this study, we focus on a three-level meta-analysis for combining data from studies using multiple-baseline across-participants designs. A complicating factor in such designs is that results might be biased if the dependent variable is affected by not explicitly modeled external events, such as the illness of a teacher, an exciting class activity, or the presence of a foreign observer. In multiple-baseline designs, external effects can become apparent if they simultaneously have an effect on the outcome score(s) of the participants within a study. This study presents a method for adjusting the three-level model to external events and evaluates the appropriateness of the modified model. Therefore, we use a simulation study, and we illustrate the new approach with real data sets. The results indicate that ignoring an external event effect results in biased estimates of the treatment effects, especially when there is only a small number of studies and measurement occasions involved. The mean squared error, as well as the standard error and coverage proportion of the effect estimates, is improved with the modified model. Moreover, the adjusted model results in less biased variance estimates. If there is no external event effect, we find no differences in results between the modified and unmodified models.     

Examining Spirituality Over Time: Latent Growth Curve and Individual Growth Curve Analyses

SUMMARY

Research on spirituality and religiousness has gained growing attention in recent years; however, most studies have used cross-sectional designs. As research on this topic evolves, there has been increasing recognition of the need to examine these constructs and their effects through the use of longitudinal designs. Beyond repeated-measures ANOVA and OLS regression models, what tools are available to examine these constructs over time? The purpose of this paper is to provide an overview of two cutting-edge statistical techniques that will facilitate longitudinal investigations of spirituality and religiousness: latent growth curve analysis using structural equation modeling (SEM) and individual growth curve models. The SEM growth curve approach examines change at the group level, with change over time expressed as a single latent growth factor. In contrast, individual growth curve models consider longitudinal change at the level of the person. While similar results may be obtained using either method, researchers may opt for one over the other due to the strengths and weaknesses associated with these methods. Examples of applications of both approaches to longitudinal studies of spirituality and religiousness are presented and discussed, along with design and data considerations when employing these modeling techniques.     

Precision of Curriculum-Based Measurement Reading Data: Considerations for Multiple-Baseline Designs

Single-case designs provide an established technology for evaluating the effects of academic interventions. Researchers interested in studying the long-term effects of reading interventions often use curriculum-based measures of reading (CBM-R) as they possess many of the desirable characteristics for use in a time-series design. The reliability of CBM-R scores is often supported by research from group designs, but making idiographic interpretations regarding the change in a student’s oral reading rate requires attention to the precision of static scores and growth estimates. The purpose of this paper is twofold. First, we discuss how recent empirical work on the technical adequacy of CBM-R scores has revealed multiple threats to the data-evaluation validity when CBM-R passages are used to measure oral reading rate. Second, we identify pertinent considerations for conducting a visual analysis of intervention effects based on CBM-R data. We conclude with a brief discussion of implications for researchers considering the use of CBM-R within multiple-baseline designs.     

A Meta‐Analysis of the ELM's Argument Quality × Processing Type Predictions

One of the key dual‐process model predictions is that audiences will be more persuaded by strong persuasive arguments than weak and that this difference in persuasiveness will be larger when they are processing centrally rather than peripherally. A series of meta‐analyses were conducted (k = 134) to assess this claim and explore moderators. The data were generally consistent with the hypothesized interaction. The effects tended to be smaller when pre‐post designs were used rather than post‐test only. Assessments of the strength of the inductions did not tend to be associated with the size of the effects associated with those inductions.     

Alternatives to Randomized Experiments

ABSTRACT— Randomized experiments are preferred for making inferences about causality when they can be implemented and their assumptions are met. Yet assumptions can fail (e.g., attrition, treatment noncompliance) or randomization may be unethical or infeasible. I describe alternative design and statistical approaches that permit testing causal hypotheses and present current empirical evidence related to alternative designs. Alternative designs permit a wider range of research questions to be answered and permit more direct generalization of causal effects; however, when using such designs, estimates of the magnitude of the causal effect may be more uncertain.     

The Application of Single-Case Research Designs to Study Elite Athletes’ Conditioning: An Update

Sport scientists have recommended to use of single-subject research designs to understand how well an intervention works and to predict performance for an elite athlete. In the present paper, the latest studies on elite athletes’ conditioning using single-subject research designs are reviewed. Although some case studies in elite athletes are reported in the literature, there are still very few studies using single-subject research designs. Rigorous and practical approaches using magnitude-based inferences such as effect size and likelihoods of substantial effects are suggested. Recommendations are also provided to facilitate further studies using single-subject research designs.     

HOW MUCH DO HIGH‐PERFORMANCE WORK PRACTICES MATTER? A META‐ANALYSIS OF THEIR EFFECTS ON ORGANIZATIONAL PERFORMANCE

Although there is growing evidence that high performance work practices (HPWPs) affect organizational performance, varying sample characteristics, research designs, practices examined, and organizational performance measures used has led extant findings to vary dramatically, making the size of the overall effect difficult to estimate. We use meta-analysis to estimate the effect size and test whether effects are larger for (a) HPWP systems versus individual practices, (b) operational versus financial performance measures, and (c) manufacturing versus service organizations. Statistical aggregation of 92 studies reveals an overall correlation that we estimate at .20. Also, the relationship is stronger when researchers examine systems of HPWPs and among manufacturers, but it appears invariant across performance measures. We use our findings as a basis to offer 4 suggestions intended to shape research practices such that future meta-analyses might answer today's emerging questions.     

Individual differences methods for randomized experiments

Experiments allow researchers to randomly vary the key manipulation, the instruments of measurement, and the sequences of the measurements and manipulations across participants. To date, however, the advantages of randomized experiments to manipulate both the aspects of interest and the aspects that threaten internal validity have been primarily used to make inferences about the average causal effect of the experimental manipulation. This article introduces a general framework for analyzing experimental data to make inferences about individual differences in causal effects. Approaches to analyzing the data produced by a number of classical designs and 2 more novel designs are discussed. Simulations highlight the strengths and weaknesses of the data produced by each design with respect to internal validity. Results indicate that, although the data produced by standard designs can be used to produce accurate estimates of average causal effects of experimental manipulations, more elaborate designs are often necessary for accurate inferences with respect to individual differences in causal effects. The methods described here can be diversely applied by researchers interested in determining the extent to which individuals respond differentially to an experimental manipulation or treatment and how differential responsiveness relates to individual participant characteristics.     

A Comparison of Using the Fixed Common-Precalibrated Parameter Method and the Matched Characteristic Curve Method for Linking Multiple-Test Items

A linking design typically consists of a data collection procedure together with an item linking procedure that places item parameters calibrated from multiple test forms onto a common scale. This study considered 2 potentially useful item response theory linking designs. The first one is characterized by selecting a single set of common items across all multiple test forms, the precalibrated item parameters of which are kept fixed while the unknown parameters of the other items are being estimated. This linking design will be referred to as the fixed common-precalibrated item parameter design. However, data collected under this design could also be analyzed by the characteristic curve method, which constituted an alternative linking procedure. In this study, the relative merits of the 2 linking designs were examined with respect to their robustness against 3 manipulated conditions-namely, when the common items have imprecise estimates, when there is a noticeable difference in the average item difficulty between the common and the noncommon items, and when the examinees are heterogeneous in terms of their abilities. A parameter recovery study was conducted to achieve this purpose. The results indicated that both linking designs were capable of producing accurate linking of items and equivalent estimation of ability parameters under the 3 conditions. When the 2 designs were actually utilized in the development of an item bank, it was found that both linking designs produced quite consistent solutions despite minor differences on some item and ability estimates. Condition under which a linking design is preferred over the other is also provided in the Discussion section of this article.     

Setting things straight: A comparison of measures of saccade trajectory deviation

In eye movements, saccade trajectory deviation has often been used as a physiological operationalization of visual attention, distraction, or the visual system’s prioritization of different sources of information. However, there are many ways to measure saccade trajectories and to quantify their deviation. This may lead to noncomparable results and poses the problem of choosing a method that will maximize statistical power. Using data from existing studies and from our own experiments, we used principal components analysis to carry out a systematic quantification of the relationships among eight different measures of saccade trajectory deviation and their power to detect the effects of experimental manipulations, as measured by standardized effect size. We concluded that (1) the saccade deviation measure is a good default measure of saccade trajectory deviation, because it is somewhat correlated with all other measures and shows relatively high effect sizes for two well-known experimental effects; (2) more generally, measures made relative to the position of the saccade target are more powerful; and (3) measures of deviation based on the early part of the saccade are made more stable when they are based on data from an eyetracker with a high sampling rate. Our recommendations may be of use to future eye movement researchers seeking to optimize the designs of their studies.     

Cross-sectional,longitudinal and composite longitudinal data on the Career Maturity Inventory Attitude Scale

Longitudinal data collection designs are frequently used in the study of developmental variables but typically yield patterns of results that are different from those obtained using cross-sectional designs. Recent research has focused on ways to clarify and reconcile the methodological distinctions between these two designs by utilizing alternative designs that incorporate features of both. This investigation compared the results obtained by the two traditional designs on the Career Maturity Inventory-Attitude Scale with the effects of an alternative design: constructing a composite longitudinal gradient. The two traditional methodologies yielded different results from each other for males, but not for females. However, the alternative design failed to eliminate the obtained differences for males, and introduced differences for females, casting doubt on the efficacy of the “solution.” The implications of these results for developmental research and career maturity were discussed.