Multiple trials may yield exaggerated effect size estimates

Published psychological research attempting to support the existence of small and medium effect sizes may not have enough participants to do so accurately, and thus, repeated trials or the use of multiple items may be used in an attempt to obtain significance. Through a series of Monte-Carlo simulations, this article describes the results of multiple trials or items on effect size estimates when the averages and aggregates of a dependent measure are analyzed. The simulations revealed a large increase in observed effect size estimates when the numbers of trials or items in an experiment were increased. Overestimation effects are mitigated by correlations between trials or items, but remain substantial in some cases. Some concepts, such as a P300 wave or a test score, are best defined as a composite of measures. Troubles may arise in more exploratory research where the interrelations among trials or items may not be well described.     

Human nonverbal counting estimated by response production and verbal report

In three experiments, people were shown sequential displays and were prevented from verbal counting by being required to perform other cognitive tasks. In Experiment 1, the subjects were shown three 1.target (target5 = 8, 16, or 32) sequences of colored geometric shapes. On occasional question trials, the subjects were asked to estimate the target number after the final item in the sequence. On other test trials, items continued to appear beyond the target, and the subjects estimated the target manually by tapping a space bar. In Experiments 2 and 3, a matching-to-sample procedure required the subjects to estimate the same sequence of items (target = 8, 11, 14, 17, or 20) both verbally and manually. The results indicated that (1) manual and verbal estimates closely approximated target size in Experiments 1 and 2, (2) coefficients of variation were constant across target size, and (3) correlations between manual and verbal estimates were positive in Experiments 2 and 3. Requiring the subjects to perform a counting task during presentation of items led to underestimation of number in Experiment 3. nt]mis|This research was funded by a research grant from the Natural Sciences and Engineering Research Council of Canada to W.A.R. and by the Ontario Graduate Scholarship in Science and Technology and the Ontario Graduate Scholarship to M.J.B.     

Sex differences in memory estimates for pictures and words with multiple recall trials

Undergraduate students (23 men and 23 women) provided memory performance estimates before and after each of three recall trials involving 80 stimuli (40 pictures and 40 words). No sex differences were found across trials for the total recall of items or for the recall of pictures and words separately. A significant increase in recall for pictures (not words) was found for both sexes across trials. The previous results of Ionescu were replicated on the first and second recall trials: men underestimated their performance on the pictures and women underestimated their performance on the word items. These differences in postrecall estimates were not found after the third recall trial: men and women alike underestimated their performance on both the picture and word items. The disappearance of item-specific sex differences in postrecall estimates for the third recall trial does not imply that men and women become more accurate at estimating their actual performance with multiple recall trials.     

How Many Subjects Does It Take To Do A Regression Analysis

Numerous rules-of-thumb have been suggested for determining the minimum number of subjects required to conduct multiple regression analyses. These rules-of-thumb are evaluated by comparing their results against those based on power analyses for tests of hypotheses of multiple and partial correlations. The results did not support the use of rules-of-thumb that simply specify some constant (e.g., 100 subjects) as the minimum number of subjects or a minimum ratio of number of subjects (N) to number of predictors (m). Some support was obtained for a rule-of-thumb that N ≥ 50 + 8 m for the multiple correlation and N ≥104 + m for the partial correlation. However, the rule-of-thumb for the multiple correlation yields values too large for N when m ≥ 7, and both rules-of-thumb assume all studies have a medium-size relationship between criterion and predictors. Accordingly, a slightly more complex rule-of thumb is introduced that estimates minimum sample size as function of effect size as well as the number of predictors. It is argued that researchers should use methods to determine sample size that incorporate effect size.     

The effect of semantic relatedness on immediate serial recall and serial recognition

The present study examined the effects of semantic relatedness on immediate serial recall and serial recognition. Each participant received either blocked or randomly intermixed serial recall or serial recognition trials. Replicating the findings of previous studies (e.g., Saint-Aubin, Ouellette, & Poirier, 2005), semantic relatedness boosted percentage serial recall but also increased order errors, after taking into account the proportion of correctly recalled items, regardless of their orders, in serial recall trials. In serial recognition trials, participants' responses were slower and less accurate for related lists than for unrelated lists. There were intraindividual correlations among order memory measures in serial recall versus serial recognition trials. The implications of these findings for item redintegration theories are discussed.     

Mental chronometry and individual differences: Modeling reliabilities and correlations of reaction time means and effect sizes

We used a general stage-based model of reaction time (RT) to investigate the psychometric properties of mean RTs and experimental effect sizes (i.e., differences in mean RTs). Using the model, formulas were derived for the reliabilities of mean RTs and RT difference scores, and these formulas provide guidance about the number of trials per participant needed to obtain reliable estimates of these measures. In addition, formulas were derived for various different types of correlations computed in RT research (e.g., correlations between a mean RT and an external non-RT measure, between two mean RTs, between a mean RT and an RT effect size). The analysis revealed that observed RT-based correlations depend on many parameters of the underlying processes contributing to RT. We conclude that these correlations often fail to support the inferences drawn from them and that their proper interpretation is far more complex than is generally acknowledged.     

Perceived size and perceived distance in stereoscopic vision and an analysis of their causal relations

Effects of visual angle and convergence upon the perceived sizes and perceived distances of a familiar object (playing card) and a nonrepresentational object (blank white card) were investigated by means of a projector stereoscope with polarizing filters. The results obtained with six Ss indicated that size estimates increased nearly proportionally as the visual angle increased and decreased nearly linearly as the convergence increased. Distance estimates decreased nearly linearly as either the visual angle or the convergence increased. The ratio of the size estimate to the distance estimate for a given visual angle was almost constant irrespective of convergence. In this sense, the size-distance invariance hypothesis held. No clear effect of familiarity was found. Partial correlations were used to discriminate direct and indirect causal relationships between the stimulus variables and perceptual estimates. Both perceived size and perceived distance were found to be determined directly by the two stimulus variables, but to be mutually related only indirectly.     

Complex partial epileptic-like experiences in university students and practitioners of Dharmakaya in Thailand: comparison with Canadian university students.

We tested the hypothesis that individuals who frequently practice meditation within another culture whose assumptions explicitly endorse this practice should exhibit more frequent and varied experience associated with complex partial epilepsy (without the seizures) as inferred by the Personal Philosophy Inventory and Roberts' Questionnaire for the Epileptic Spectrum Disorder. 80 practitioners of Dharma Meditation and 24 university students in Thailand were compared with 76 students from first-year courses in psychology in a Canadian university. Although there were large significant differences for some items and clusters of items expected as a result of cultural differences, there were no statistically significant differences between the two populations for the proportions of complex partial epileptic-like experiences or their frequency of occurrence. There were no strong or consistent correlations between the history of meditation within the sample who practiced Dharma meditation and these experiences. These results suggest complex partial epileptic-like experiences may be a normal feature of the human species.     

A Bayesian Power Analysis Procedure Considering Uncertainty in Effect Size Estimates from a Meta-analysis

In conventional frequentist power analysis, one often uses an effect size estimate, treats it as if it were the true value, and ignores uncertainty in the effect size estimate for the analysis. The resulting sample sizes can vary dramatically depending on the chosen effect size value. To resolve the problem, we propose a hybrid Bayesian power analysis procedure that models uncertainty in the effect size estimates from a meta-analysis. We use observed effect sizes and prior distributions to obtain the posterior distribution of the effect size and model parameters. Then, we simulate effect sizes from the obtained posterior distribution. For each simulated effect size, we obtain a power value. With an estimated power distribution for a given sample size, we can estimate the probability of reaching a power level or higher and the expected power. With a range of planned sample sizes, we can generate a power assurance curve. Both the conventional frequentist and our Bayesian procedures were applied to conduct prospective power analyses for two meta-analysis examples (testing standardized mean differences in example 1 and Pearson's correlations in example 2). The advantages of our proposed procedure are demonstrated and discussed.     

Order and serial position effects in response time with multiple-item probe recognition

We examined the order effect in item-recognition response time, that is, differences in response time for multiple-item probes containing items in the same or in the reverse order as those in the memory set. Experiment 1 used the response condition in which only one item must be positive for a positive response, Experiment 2 used homogeneous probes in which all the items are either positive or negative, and Experiment 3 used the condition in which all the items must be positive. Of particular interest were the serial position variations in order effects for probes containing items that were adjacent in the memory set. We previously found that such effects are an indication of subjective grouping of the memory set and the matching of the probe with these subgroups. The order effect in the one-positive condition was only weak in most cases, but it was strong with homogeneous probes when the memory set was objectively grouped or was ungrouped but with a constant set size. There were also strong order effects in the all-positive condition for probes with items that were nonadjacent in the memory set. Our results are interpreted in terms of a parallel match process based on a distribution over position of items in subjective or objective groups. We account for the origin of the distribution-over-position process in terms of multiple representations of the grouped memory sets. The model assumes that each subgroup is represented in memory several, and perhaps very many, times and that considerable error in item positioning can occur over the multiple representations of any group.     

Cumulative intertrial inhibition in repeated visual search

In the present study the author examined visual search when the items remain visible across trials but the location of the target varies. Reaction times for inefficient search cumulatively increased with increasing numbers of repeated search trials, suggesting that inhibition for distractors carried over successive trials. This intertrial inhibition held across at least 16 items and when the search items moved randomly; however, it disappeared when the search items were removed from the display in an intertrial interval. In contrast, improvements to search when a target appeared at the same location on successive trials were weakened in a dynamic display, and this effect was resistant to the removal of search items. This dissociation implies that intertrial inhibition is based on a different mechanism than intertrial facilitation. The potential mechanisms for these effects are discussed.     

A nonparametric item analysis of a selected item subset of the Learning Process Questionnaire

Background: Although there have been numerous studies conducted on the psychometric properties of Biggs' Learning Process Questionnaire (LPQ), these have involved the use of traditional omnibus measures of scale quality such as corrected item total correlations, internal consistency estimates of reliability, and factor analysis. However, these omnibus measures of scale quality are sample dependent and fail to model item responses as a function of trait level. And since the item trait relationship is typically nonlinear, traditional factor analytic methods are inappropriate. Aims: The purpose of this study was to identify a unidimensional subset of LPQ items and examine the effectiveness of these items and their options in discriminating between changes in the underlying trait level. In addition to assessing item quality, we were interested in assessing overall scale quality with non‐sample dependent measures. Method: The sample was split into two nearly equal halves, and a undimensional subset of items was identified in one of these samples and cross‐validated in the other. The nonlinear relationship between the probability of endorsing an item option and the underlying trait level was modelled using a nonparametric latent trait technique known as kernel smoothing and implemented with the program TestGraf. After item and scale quality were established, maximum likelihood estimates of participants' trait level were obtained and used to examine grade and gender differences. Results: A undimensional subset of 16 deep and achieving items was identified. Slightly more than half of these items needed some of their options combined so that the probability of endorsing an item option as a function of increasing trait level corresponded to the ideal rank ordering of the item options. With this adjustment, scale quality as measured by the information function and standard error function was found to be good. However, no statistically significant gender differences were observed and, although statistically significant grade differences were observed, they were not substantively meaningful. Conclusions: The use of nonparametric kernel‐smoothing techniques is advocated over parametric latent trait methods for the analysis of attitudinal and psychological measures involving polychotomous ordered‐response categories. It is also suggested that latent trait methods are more appropriate than traditional test‐based measures for studying differential item functioning both within and between cultures. Nonparametric kernel‐smoothing techniques hold particular promise in identifying and understanding cross‐cultural differences in student approaches to learning at both the item and scale level.     

Effects of food-pellet size on rate,latency, and topography of autoshaped key pecks and gapes in pigeons.

Four pigeons responded under autoshaping contingencies in which different conditional stimuli (red or green keylights) were associated with unconditional stimuli of different magnitudes (large or small food pellets) over successive trials within a session. Both topography (beak opening or gape) and strength (rates and latencies of key pecks and gapes) of responding during the conditional stimuli depended on the magnitude of the correlated unconditional stimulus. Key-peck and gape rates were higher and latencies were shorter in large-pellet trials than in small-pellet trials. Gape amplitudes varied directly with pellet size, although conditional and unconditional gapes were larger than either pellet. These findings were replicated when the key colors were presented either on one or two keys and after reversals of the color-size correlations. Because the unconditional stimulus was varied through pellet size, magnitude was not confounded with food-access duration or quality. These results demonstrate the effects of the magnitude of the unconditional stimulus, in that rates and latencies of both key pecks (which are directed movements toward the key) and gapes (which are independent of the bird''s position and key properties) varied with pellet size. Gape measures were unique in that two dimensions (response strength and topography) of a single response class varied simultaneously with magnitude.     

Ordinary Least Squares Estimation of Parameters in Exploratory Factor Analysis With Ordinal Data

Exploratory factor analysis (EFA) is often conducted with ordinal data (e.g., items with 5-point responses) in the social and behavioral sciences. These ordinal variables are often treated as if they were continuous in practice. An alternative strategy is to assume that a normally distributed continuous variable underlies each ordinal variable. The EFA model is specified for these underlying continuous variables rather than the observed ordinal variables. Although these underlying continuous variables are not observed directly, their correlations can be estimated from the ordinal variables. These correlations are referred to as polychoric correlations. This article is concerned with ordinary least squares (OLS) estimation of parameters in EFA with polychoric correlations. Standard errors and confidence intervals for rotated factor loadings and factor correlations are presented. OLS estimates and the associated standard error estimates and confidence intervals are illustrated using personality trait ratings from 228 college students. Statistical properties of the proposed procedure are explored using a Monte Carlo study. The empirical illustration and the Monte Carlo study showed that (a) OLS estimation of EFA is feasible with large models, (b) point estimates of rotated factor loadings are unbiased, (c) point estimates of factor correlations are slightly negatively biased with small samples, and (d) standard error estimates and confidence intervals perform satisfactorily at moderately large samples.     

Research on individual differences requires correlational analysis,not ANOVA

The data of Welsh and Baucom (1977) have been reanalyzed using correlational analysis for the complete distributions in place of the analysis of variance of extreme groups. The analysis reveals a number of statistically significant correlations between composites of M-F scales and several measures of intelligence, but these are trival in size. Also the sign of the correlations depends directly on which M-F scales are selected to form composites. One can obtain a correlation between either masculinity and intelligence or femininity and intelligence depending on whether the supposed trait is measured from scales formed from the Adjective Check List or from a combination of the MMPI and the SVIB.     

The Mental Changes Due to Frontal Lobe Damage

The literature suggests association between arousal, general activation, and anxiety on the one hand, and time judgments on the other hand, implying that reported differences in time judgment between nosological groups may be confounded by group differences in arousal-anxiety.

Self-report measures of anxiety, as well as magnitude estimates and magnitude productions of standards ranging from 500 to 2000 msec in 250 msec steps and presented in 10 randomized blocks, were obtained from 16 male normals and from 16 male hospitalized patients with a tentative diagnosis of chronic undifferentiated schizophrenia. Only 10 of the 16 patients were later found to have the same confirmed diagnosis. Data from nine normals and from seven chronic undifferentiated schizophrenics met a criterion of linearity of response functions for both time judgment methods and were further analyzed.

Magnitude estimates and magnitude productions showed underestimation of elapsed time, both types of judgment exhibited satisfactory reliability, estimates showed “shortening,” and productions showed “lengthening” over blocks of trials.

Intercepts of the response × standards functions were not generally equal to zero, were more negative in estimation than in production, had marginal reliability or were unreliable, did not correlate significantly between methods, and did not show significant trends over blocks of trials.

Following a model by Carlson and Feinberg, slopes of response × standard functions were used as estimates of the rate of the “internal clock.” (In estimation, the rate is equal to the slope; in production, it is equal to the reciprocal of the slope.) Average rates of the internal clock did not differ between methods for normals, but were higher in production than in estimation for the seven patients. Clock rates did not differ significantly between groups, were reliable, and exhibited positive correlation between methods. Clock rates exhibited trends over blocks of trials: arctan equivalents of clock rates were linearly related to ordinal numbers of blocks of trials and showed decreases, or “slowing” of the internal clock, in both methods.

Differences in mean anxiety between groups were not significant. In each group, anxiety scores showed positive average correlations with magnitude estimates and negative average correlations with magnitude productions, failed to correlate significantly with intercepts, but showed positive correlations with clock rates.

The data also suggest that anxiety and intrasubject variability may be interrelated.

To conclude: Reported differences in time judgments between nosological groups may not solely be due to nosological differences per se, but instead may be due to group differences in anxiety.     

Some rao-guttman relationships

An examination of the determinantal equation associated with Rao's canonical factors suggests that Guttman's best lower bound for the number of common factors corresponds to the number of positive canonical correlations when squared multiple correlations are used as the initial estimates of communality. When these initial communality estimates are used, solving Rao's determinantal equation (at the first stage) permits expressing several matrices as functions of factors that differ only in the scale of their columns; these matrices include the correlation matrix with units in the diagonal, the correlation matrix with squared multiple correlations as communality estimates, Guttman's image covariance matrix, and Guttman's anti-image covariance matrix. Further, the factor scores associated with these factors can be shown to be either identical or simply related by a scale change. Implications for practice are discussed, and a computing scheme which would lead to an exhaustive analysis of the data with several optional outputs is outlined.     

权力相关词语的判断的大小效应及其可控性

采用Stroop干扰实验范式,证实权力词语判断的大小效应,并探索采用意识性干预和试次偏差分布策略是否会使大小效应得到控制。实验1中试次类型变量主效应极其显著。实验2中试次类型变量与提示条件变量的交互作用边缘显著。实验3中试次类型变量与提示条件变量的交互作用显著;试次类型变量与试次分布变量的交互作用显著。结果表明,权力相关词语的判断中存在大小效应,提示信息使大小效应得到控制,试次偏差分布使大小效应发生反转。     

Estimation of indifference points with an adjusting-delay procedure.

In a series of conditions, pigeons chose between 1.5 s and 3 s of access to grain, each preceded by some delay. The delay that preceded the small reinforcer was constant throughout a condition. The delay that preceded the large reinforcer was increased or decreased a number of times each session in order to estimate an "indifference point," a delay at which the subject chose each alternative about equally often. The experiment was designed to determine whether variations in any of four features of this adjusting-delay procedure would systematically alter the estimated indifference points. The four features were the total trial duration, the number of center-key responses necessary to begin a trial, the number of choice trials that preceded each change in the adjusting delay, and step size--the size of each increment and decrement in the delay. Manipulation of the first three features had no systematic effects on the indifference points. As step size was increased from 0.5 s to 6 s, within-session variability of the adjusting delay steadily increased, and the 6-s step size produced larger indifference-point estimates for some subjects. The results suggest that, within certain limits, these procedural features can be altered without affecting the indifference-point estimates, but that the use of a large step size can distort the estimates. Some theoretical implications of the relative constancy of indifference points across these procedural variations are discussed.     

Age differences in the allocation of study time account for age differences in memory performance

How aging affects the utilization of monitoring in the allocation of study time was investigated by having adults learn paired associates during multiple study-test trials. During each trial, a subject paced the presentation of individual items and later judged the likelihood of recalling each item on the upcoming test; after all items had been studied and judged, recall occurred. For both age groups in Study 1, (1) people’s judgments were highly accurate at predicting recall and (2) intraindividual correlations between judgments (or recall) on one trial, and study times on the next trial were negative, which suggests that subjects utilized monitoring to allocate study time. However, the magnitude of these correlations was less for older than for younger adults. Study 2 revealed that these differences were not due to age differences in forgetting. Results from both studies suggest that older adults do not utilize on-line monitoring to allocate study to the same degree as younger adults do, and that these differences in allocation contribute to age deficits in recall.