Sample size planning for composite reliability coefficients: accuracy in parameter estimation via narrow confidence intervals

Composite measures play an important role in psychology and related disciplines. Composite measures almost always have error. Correspondingly, it is important to understand the reliability of the scores from any particular composite measure. However, the point estimates of the reliability of composite measures are fallible and thus all such point estimates should be accompanied by a confidence interval. When confidence intervals are wide, there is much uncertainty in the population value of the reliability coefficient. Given the importance of reporting confidence intervals for estimates of reliability, coupled with the undesirability of wide confidence intervals, we develop methods that allow researchers to plan sample size in order to obtain narrow confidence intervals for population reliability coefficients. We first discuss composite reliability coefficients and then provide a discussion on confidence interval formation for the corresponding population value. Using the accuracy in parameter estimation approach, we develop two methods to obtain accurate estimates of reliability by planning sample size. The first method provides a way to plan sample size so that the expected confidence interval width for the population reliability coefficient is sufficiently narrow. The second method ensures that the confidence interval width will be sufficiently narrow with some desired degree of assurance (e.g., 99% assurance that the 95% confidence interval for the population reliability coefficient will be less than W units wide). The effectiveness of our methods was verified with Monte Carlo simulation studies. We demonstrate how to easily implement the methods with easy-to-use and freely available software.     

Chaotic homes and children's disruptive behavior: a longitudinal cross-lagged twin study

Chaotic home lives are correlated with behavior problems in children. In the study reported here, we tested whether there was a cross-lagged relation between children's experience of chaos and their disruptive behaviors (conduct problems and hyperactivity-inattention). Using genetically informative models, we then tested for the first time whether the influence of household chaos on disruptive behavior was environmentally mediated and whether genetic influences on children's disruptive behaviors accounted for the heritability of household chaos. We measured children's perceptions of household chaos and parents' ratings of children's disruptive behavior at ages 9 and 12 in a sample of 6,286 twin pairs from the Twins Early Development Study (TEDS). There was a phenotypic cross-lagged relation between children's experiences of household chaos and their disruptive behavior. In genetically informative models, we found that the effect of household chaos on subsequent disruptive behavior was environmentally mediated. However, genetic influences on disruptive behavior did not explain why household chaos was heritable.     

Is the Web as good as the lab? Comparable performance from Web and lab in cognitive/perceptual experiments

With the increasing sophistication and ubiquity of the Internet, behavioral research is on the cusp of a revolution that will do for population sampling what the computer did for stimulus control and measurement. It remains a common assumption, however, that data from self-selected Web samples must involve a trade-off between participant numbers and data quality. Concerns about data quality are heightened for performance-based cognitive and perceptual measures, particularly those that are timed or that involve complex stimuli. In experiments run with uncompensated, anonymous participants whose motivation for participation is unknown, reduced conscientiousness or lack of focus could produce results that would be difficult to interpret due to decreased overall performance, increased variability of performance, or increased measurement noise. Here, we addressed the question of data quality across a range of cognitive and perceptual tests. For three key performance metrics??mean performance, performance variance, and internal reliability??the results from self-selected Web samples did not differ systematically from those obtained from traditionally recruited and/or lab-tested samples. These findings demonstrate that collecting data from uncompensated, anonymous, unsupervised, self-selected participants need not reduce data quality, even for demanding cognitive and perceptual experiments.     

On effect size

The call for researchers to report and interpret effect sizes and their corresponding confidence intervals has never been stronger. However, there is confusion in the literature on the definition of effect size, and consequently the term is used inconsistently. We propose a definition for effect size, discuss 3 facets of effect size (dimension, measure/index, and value), outline 10 corollaries that follow from our definition, and review ideal qualities of effect sizes. Our definition of effect size is general and subsumes many existing definitions of effect size. We define effect size as a quantitative reflection of the magnitude of some phenomenon that is used for the purpose of addressing a question of interest. Our definition of effect size is purposely more inclusive than the way many have defined and conceptualized effect size, and it is unique with regard to linking effect size to a question of interest. Additionally, we review some important developments in the effect size literature and discuss the importance of accompanying an effect size with an interval estimate that acknowledges the uncertainty with which the population value of the effect size has been estimated. We hope that this article will facilitate discussion and improve the practice of reporting and interpreting effect sizes.     

Social Antecedents of Children's Trustworthiness

A total of 1329 children were tested twice across 1 year (M = 7 years 5 months of age at Time 1 (T1)) in the Zurich Project on Social Development. The measures at T1 were corporal punishment, neighbourhood trustworthiness and children's trustworthiness (not lying/cheating and not stealing). At Time 2 (T2), children reported the promise keeping of their classmates, which, via social relations analyses, yielded evidence for individual differences in reliability trustworthiness. Structural equation modelling analyses confirmed that there was stability in children's trustworthiness as a latent variable. The structural equation modelling further yielded evidence that (1) corporal punishment at T1 was negatively associated with children's trustworthiness at T1 and negatively predicted changes in children's trustworthiness and (2) neighbourhood trustworthiness at T1 was positively associated with children's trustworthiness at T1 and positively predicted changes in children's trustworthiness. The findings yielded support for the hypotheses that corporal punishment negatively, and neighbourhood trustworthiness positively, contributes to the development of trustworthiness in children. Copyright © 2011 John Wiley & Sons, Ltd.     

Postural Sway and the Amplitude of Horizontal Eye Movements

In 2 experiments, we evaluated relations between postural activity and the amplitude of visually guided eye movements. Participants shifted their gaze to follow horizontal oscillation of a visible target. The target moved with amplitude 9° or 24°. In different experiments, the frequency of target oscillation was 0.5 Hz or 1.1 Hz. In both experiments, the variability of head and torso motion was reduced (in the anterior-posterior axis) when participants viewed moving targets, relative to sway during viewing of stationary targets. Sway variability was not influenced by the amplitude of target motion. The results are compatible with the hypothesis that postural activity was modulated relative to the demands of suprapostural visual tasks.