Role of Effect Sizes in Contemporary Research in Counseling

Effect sizes (e.g., Cohen's d, Glass's Δ, η², adjusted R², ω²) quantify the extent to which sample results diverge from the expectations specified in the null hypothesis. The present article addresses 5 related questions. First, is the advocacy for reporting and interpreting effect sizes part of the controversy over statistical significance testing? Second, why cannot p values be used as effect sizes? Third, what are the various categories of effect sizes and some commonly used examples of each type? Fourth, how should effect sizes be interpreted? Fifth, what are some recommendations for further reading?     

Ezekiel’s classic estimator of the population squared multiple correlation coefficient: Monte Carlo-based extensions and refinements

Abstract

Ezekiel’s adjusted R² is widely used in linear regression analysis. The present study examined the statistical properties of Ezekiel’s measure through a series of Monte Carlo simulations. Specifically, we examined the bias and root mean squared error (RMSE) of Ezekiel’s adjusted R² relative to (a) the sample R² statistic, and (b) the sample R² minus the expected value of R². Simulation design factors consisted of sample sizes (N?=?50, 100, 200, 400), number of predictors (2, 3, 4, 5, 6), and population squared multiple correlations (ρ² = 0, .10, .25, .40, .60). Factorially crossing these design factors resulted in 100 simulation conditions. All populations were normal/Gaussian, and for each condition, we drew 10,000 Monte Carlo samples. Regarding systematic variation (bias), results indicated that with few exceptions, Ezekiel’s adjusted R² demonstrated the lowest bias. Regarding unsystematic variation (RMSE), the performance of Ezekiel’s measure was comparable to the other statistics, suggesting that the bias-variance tradeoff is minimal for Ezekiel’s adjusted R². Additional findings indicated that sample size-to-predictor ratios of 66.67 and greater were associated with low bias and that ratios of this magnitude were accompanied by large sample sizes (N?=?200 and 400), thus suggesting that researchers using Ezekiel’s adjusted R² should aim for sample sizes of 200 or greater in order to minimize bias when estimating the population squared multiple correlation coefficient. Overall, these findings indicate that Ezekiel’s adjusted R² has desirable properties and, in addition, these findings bring needed clarity to the statistical literature on Ezekiel’s classic estimator.     

Predicting Intention to Save Water: Theory of Planned Behavior,Response Efficacy,Vulnerability, and Perceived Efficiency of Alternative Solutions1

These 2 studies attempted to predict people's intention to save water. Study 1 used a model based on Ajzen's (1991) theory of planned behavior (TPB) and other variables: vulnerability, 2 collective efficacy variables, and subjective effectiveness of alternative solutions (SEAS) to ease drought impact. Study 2 tested a model similar to that of Study 1, but with 2 personal efficacy variables added. Respondents in both studies were residents of Taiwan (Ns= 166 and 210). Analysis indicated that the modified models (R²>.32) were better than the TPB model (R²<.19), and SEAS and response efficacy had crucial effects on people's intentions to retrofit. The studies also found some significant but inconsistent effects of income, dwelling, and education.     

The relationship of children's hope to pediatric asthma treatment adherence

This study investigated the relationship between hope and adherence to a daily inhaled steroid regimen among 48 asthma patients ages 8–12 years old who participated in a 14 day adherence assessment. Participants completed the Children's Hope Scale, and parents completed a questionnaire aimed at demographic and disease-related information. Adherence was measured by electronic monitoring of the use of the participant's metered-dose inhaler. A multivariate model predicting nonadherence was built, including FEV₁ in the first step and children's hope level in the second step. This model was a significant predictor of adherence (Nagelkerke R ²?=?0.24, p?=?0.01). No other demographic or psychosocial variables were significant predictors of adherence. These findings highlight the need to attend to psychosocial predictors of adherence, specifically hope, and may help practitioners target these factors in their efforts to increase adherence among pediatric asthma patients.     

Situation Awareness and Transformational Leadership in Senior Military Leaders: An Exploratory Study

This study explored the relationship between leadership style and operational readiness in a sample of senior Norwegian military officers (N = 43), who participated in a 1-week joint staff exercise. Leadership style was measured by the Multifactor Leadership Questionnaire (MLQ-45), and indicators of operational readiness included situation awareness and interpersonal influence. Transformational leadership emerged as a predictor of situation awareness (R² = .33) and interpersonal influence (R² = .25), with intellectual stimulation as the only significant predictor among the facet subscales. Some possible theoretical and methodological implications for future research are also pointed out.     

A Two Factor ANOVA-like Test for Correlated Correlations: CORANOVA

Testing homogeneity of correlations with Fisher's Z is inappropriate when correlations are themselves correlated. Suppose measurements of brain activation and performance are taken before and during a verbal memory task. Of interest are changes in activity gradients in specific regions, R₁, R₂, R₃, and performance, V. The "correlated correlations" of interest ρ_V,R1 , ρ_V,R2 , and ρ_V,R3 , have a single variable, V, in common. We wish to compare these correlations between males and females, across regions, and to assess an interaction of the correlation. Fisher's Z can compare pairs of correlations, and Olkin and Finn's (1990) method can test homogeneity of correlated correlations across a single within factor (based on asymptotic normality), but no current procedure can test a region by gender (within by between) interaction of correlations. We propose a nonparametric method for testing this interaction and both main effects. The procedure is analogous to two-way ANOVA, but hypotheses test homogeneity of correlations, not means. The null distributions are estimated with permutations, avoiding asymptotic distributional assumptions and enhancing applicability to smaller samples and non-normal data. Simulations demonstrated maintenance of correct level (power = alpha level under the null) for normal and non-normal data and small samples. The Olkin-Finn test had inflated level for non-normal data or small samples. The Fisher's Z had inflated level for non-normal data, but not for small samples. Our method had better efficiency across contrasts and data types and sizes. Applied to correlations between regional laterality of blood flow and verbal memory performance, the method showed sensitivity to a biologically meaningful sex by region interaction in these correlations. A SAS macro for CORANOVA is available.     

The effect of covariate mean differences on the standard error and confidence interval for the comparison of treatment means

The use of covariates is commonly believed to reduce the unexplained error variance and the standard error for the comparison of treatment means, but the reduction in the standard error is neither guaranteed nor uniform over different sample sizes. The covariate mean differences between the treatment conditions can inflate the standard error of the covariate‐adjusted mean difference and can actually produce a larger standard error for the adjusted mean difference than that for the unadjusted mean difference. When the covariate observations are conceived of as randomly varying from one study to another, the covariate mean differences can be related to a Hotelling's T². Using this Hotelling's T² statistic, one can always find a minimum sample size to achieve a high probability of reducing the standard error and confidence interval width for the adjusted mean difference.     

Identifying the Source of Misfit in Item Response Theory Models

When an item response theory model fails to fit adequately, the items for which the model provides a good fit and those for which it does not must be determined. To this end, we compare the performance of several fit statistics for item pairs with known asymptotic distributions under maximum likelihood estimation of the item parameters: (a) a mean and variance adjustment to bivariate Pearson's X², (b) a bivariate subtable analog to Reiser's (1996) overall goodness-of-fit test, (c) a z statistic for the bivariate residual cross product, and (d) Maydeu-Olivares and Joe's (2006) M₂ statistic applied to bivariate subtables. The unadjusted Pearson's X² with heuristically determined degrees of freedom is also included in the comparison. For binary and ordinal data, our simulation results suggest that the z statistic has the best Type I error and power behavior among all the statistics under investigation when the observed information matrix is used in its computation. However, if one has to use the cross-product information, the mean and variance adjusted X² is recommended. We illustrate the use of pairwise fit statistics in 2 real-data examples and discuss possible extensions of the current research in various directions.     

An Algorithm to Improve Test Answer Copying Detection Using the Omega Statistic

The omega (ω) statistic is reputed to be one of the best indices for detecting answer copying on multiple choice tests, but its performance relies on the accurate estimation of copier ability, which is challenging because responses from the copiers may have been contaminated. We propose an algorithm that aims to identify and delete the suspected copied responses through probability sampling and bootstrapping. In doing so, the bias in copier ability estimation will be determined and used to update the ability estimate for calculating the modified omega (ω^m), a new statistic based on the ω. The performance of ω^m and ω were compared in a Monte Carlo simulation study under 40 typical testing conditions (2 test lengths x 4 sample sizes x 5 levels of copying). In almost all conditions, the ω^m had the same or better controlled Type I error and higher power than ω. The increase in power was particularly eminent when the source's estimated ability was higher than the copier and when 20% or 30% of items were copied. These findings support the use of the ω^m as a replacement of ω to detect answer copying in multiple choice exams.     

Situational,Interpersonal, and Intrapersonal Characteristic Associations With Adolescent Conflict Forgiveness

Given the voluntary nature of adolescent friendships, forgiveness of interpersonal transgressions has been identified as a critical aspect of maintaining these relationships. However, transgression forgiveness is related to a range of situational (e.g., transgression severity), interpersonal (e.g., friendship commitment), and intrapersonal (e.g., victim's empathy) factors. Data from 161 adolescents were used to examine the nature of the relationships between these factors and forgiveness and to examine the differential association patterns for adolescent boys and girls. Results for the overall adolescent sample indicated both situational and interpersonal factor associations with forgiveness (R ² = .52, p < .001). Examination of separate female and male forgiveness reports indicated similar interpersonal factor associations and differential situational factor associations with female (R ² = .46, p < .001), and male (R ² = .60, p < .001) forgiveness. Findings suggest the likelihood of forgiving may be contextually dependent, and that researchers should consider transgression, relationship, and intrapersonal characteristics when examining forgiveness. Further, the present study suggests the contextual factors associated with forgiveness may be further differentiated by gender.     

An Evaluation of James et al.'s (1992) VG Estimation Procedure when Artifacts and True Validity are Correlated*

James, Demaree, Muliak, and Ladd (1992) outlined a procedure for estimating the mean (M) and variance (V) of true validities (ρ's). This procedure was designed to take into account the potential nonzero intercorrelations among the three artifacts (predictor reliability, criterion reliability, and range restriction) and ρ. The accuracy of this new validity generalization procedure was compared with the accuracy of the Model 2‐based (in which correlations are individually corrected for artifacts) procedure because this latter procedure does not require the assumption of uncorrelated artifacts. The current study included two different ρ distributions, three sample sizes, and six different levels of intercorrelations among the three artifacts and ρ. Both procedures yielded relatively accurate estimates of M and V even when the intercorrelations among the three artifacts and ρ were nonzero. The Model 2‐based estimates were slightly more accurate than the James et al. estimates, and the accuracy of the Model 2‐based estimates was much more stable across sample sizes and different levels of intercorrelatedness. Sample‐based artifact data were used in this investigation.     

Religious Orientation and Ethnic Identity as Predictors of Religious Coping Among Bereaved Individuals

Religious orientation and ethnic identity inform the religious coping process, but research on this topic is scarce. The authors collected data on these constructs from a sample (N = 319) of bereaved adults. A canonical correlation analysis showed that individuals who engage in traditional spiritual practices and strive to achieve ordinary and transcendental spiritual goals are more likely to engage in positive religious coping (Wilks's Λ = .36, R_c² = .62, p < .001). Also, a multiple regression analysis revealed that individuals with higher levels of ethnic identity development are more likely to engage in positive religious coping (β = .12, t < .05). Finally, a discriminant analysis indicated that ethnic identity and a conservative religious orientation discriminated between Whites and ethnic minority individuals, Wilks's Λ = .71, χ²(4, N = 204) = 70.10, p < .001, R_c² = .26. The authors encourage counselors to strengthen their multicultural and spiritual competencies to provide effective services to a culturally and religiously diverse clientele.     

More than g: Evidence for the Incremental Validity of Performance-Based Assessments for Predicting Training Performance

Past research has consistently shown that tests measuring specific cognitive abilities provide little if any incremental validity over tests of general mental ability when predicting performance on the job. In this study, we suggest that the seeming lack of incremental validity may have been due to the type of content that has traditionally been assessed. Therefore, we hypothesised that incremental validity can be obtained using specific cognitive abilities that are less highly correlated with g and are matched to the tasks performed on the job. To test this, we examined a recently developed performance-based measure that assesses a number of cognitive abilities related to training performance. In a sample of 310 US Navy student pilots, results indicated that performance-based scores added sizeable incremental validity to a measure of g. The significant increases in R² ranged from .08 to .10 across criteria. Similar results were obtained after correcting correlations for range restriction, though the magnitude of incremental validity was slightly smaller (ΔR² ranged from .05 to .07).     

Predicting the Performance of Dental Hygiene Behaviors: An Examination of the Fishbein and Ajzen Model and Self-Efficacy Expectations1

This study examined the value of the Fishbein and Ajzen model of behavioral intentions and Bandura's concept of self-efficacy expectations as prospective predictors of the dental hygiene behaviors of young adults. All participants (73 males and 58 females) completed self-report measures of the predictor variables and 60% of that group (N = 77) then recorded brushing and flossing behaviors over a four-week period. The Fishbein and Ajzen model accounted for a significant proportion of the variance in intentions to brush (R²= .32) and intentions to floss (R²= .30). Intentions were in turn related to self-monitoring records of brushing and flossing frequency (rs= .52 and .61). Introducing self-efficacy expectations into the Fishbein and Ajzen model failed to improve the prediction of brushing and flossing frequency. However, self-efficacy was predictive of behavioral intentions, adding significantly to the variance accounted for by the attitudinal and subjective norm components of the Fishbein and Ajzen model. These data suggest that self-efficacy expectations are important in understanding protective health behaviors and that the inclusion of a self-efficacy component in the Fishbein and Ajzen model deserves consideration.     

Social dominance orientation predicts drive for muscularity among British men

The present study tested the hypothesis that men's drive for muscularity would be associated with their valuation of domination, power, status, and aggression over others. A community sample of 359 men from London, UK, completed measures of drive for muscularity, social dominance orientation, right-wing authoritarianism, trait aggression, and need for power, as well as their demographic details. Bivariate correlations showed that greater drive for muscularity was significantly correlated with most of the measures and their subscales. However, in a multiple regression analysis, the only significant predictor of drive for muscularity was support for group-based dominance hierarchies (Adj. R² = .17). These results suggest that men's drive for muscularity is associated with a socio-political ideology that favours social dominance.     

Validity of the strategic management simulations to predict real-world productivity

The performance of individual businesses and the economy as a whole hinges on the productivity of human capital, yet measuring productivity at an individual level has proved difficult. The Strategic Management Simulation (SMS) methodology comprises several decision-making assessment tools that have been used for several decades to estimate real-world productivity. Performance on productivity measures using a shorter version of the SMS methodology was gathered from a sample of 111 knowledge workers from major corporations in the U.S. and compared with their income level. In linear models, every percentile increase in SMS performance was associated with a $791–$1,113 increase in income in each of the four domains of decision-making performance provided by the SMS tool. The models had R²’s ranging from 0.65 to 0.75, indicating high levels of correlation between SMS performance and income levels. This study is the first to show linear relationships between SMS performance and other indicators of productivity, indicating that the SMS is a valid predictor of productivity and that SMS scores can be used to predict job satisfaction and performance.     

Exploratory regression analysis: A tool for selecting models and determining predictor importance

Linear regression analysis is one of the most important tools in a researcher’s toolbox for creating and testing predictive models. Although linear regression analysis indicates how strongly a set of predictor variables, taken together, will predict a relevant criterion (i.e., the multiple R), the analysis cannot indicate which predictors are the most important. Although there is no definitive or unambiguous method for establishing predictor variable importance, there are several accepted methods. This article reviews those methods for establishing predictor importance and provides a program (in Excel) for implementing them (available for direct download at . The program investigates all 2 ^p – 1 submodels and produces several indices of predictor importance. This exploratory approach to linear regression, similar to other exploratory data analysis techniques, has the potential to yield both theoretical and practical benefits.     

Predictors of preschool language environments and their relations to children's vocabulary

Despite support for the importance of early language environments, little is known about the naturally occurring experiences children have in preschool settings. The current study sample included 91 children (M_age = 4.72 years; 56% male; 67% White) from 23 preschool classrooms and nearly 1500 h of language environment data from three waves throughout the preschool year. Of the sociodemographic characteristics, family income is most closely related to children's preschool language environments. A standard deviation increase in family income was related to children hearing approximately one million more adult words in their preschool classroom. However, conversational turns were the more robust predictor of vocabulary skills with effect sizes around 0.20, depending on model specification. Theoretical and policy implications of these findings are discussed.     

Measuring the impact on relatives of caring for people with Alzheimer's disease: Quality of life,burden and well-being

Abstract

This study compared key correlates of caregiver stress in 50 Alzheimer's disease patients and their primary caregivers. in relation to three outcome measures - perceived burden, psychological well-being, and quality of life (QoL). These were evaluated using the Zarit Burden Interview. General Health Questionnaire (GHQ-30), and Schedule for the Evaluation of Individual QoL (SEIQoL-DW) respectively. Informal social support was evaluated on Vaux's Social Support Appraisal Scale. Patients' cognitive. functional, and behavioural status were rated on Mini-Mental State Examination, Blessed-Roth Dementia Scale. and Baumgarten Dementia Behaviour Disturbance Scale respectively. Standardised multiple regression analysis was used to compare the outcome measures. In this model burden was highly related to behaviour disturbance. and also to social support (adjusted R² = 0.45). Well-being was significantly related to behaviour disturbance, and also to functional status (adjusted R² = 0.40). With regard to QoL the model performed poorly as most of the variance in QoL was not accounted for by the model (adjusted R² = 0.14). These findings highlight differences in factors determining caregiver QoL. burden and well-being.     

Internal Cue Theory: Calibration and Resolution of Confidence in General Knowledge

A theory is proposed in which beliefs in the form of internal cue validities mediate the processing of ecological cue validities in the assessment of confidence. The conditions necessary for perfect calibration are specified: (a) correspondence between ecological and internal validity, (b) perfect translation of internal validity into a confidence assessment, and (c) consistent utilization of cues. Process errors are then added to these conditions to investigate how calibration is affected by error variance of confidence assessments. To accomplish this, the calibration score (C) is decomposed into three additive parts: D² = bias, i.e., the squared difference between mean confidence and proportion correct; R² = resolution, i.e., the squared difference between the standard deviations of confidence and proportion correct; L = linearity, i.e., how closely the calibration curve follows a linear function. In the equation C = D² + R² + L, R² (resolution) reflects the subject′s ability to discriminate cue validities. Selection of items is a critical factor in studies of confidence. Informal selection with a tendency to avoid easy items results in overconfidence. Internal cue theory predicts both that overconfidence should disappear (in accordance with previous research) and that resolution should improve when item selection is made representative of the natural environment. Both predictions are confirmed by data from published studies on confidence in general knowledge. It is noteworthy that resolution is still poor and accounts for the major portion of miscalibration under representative item selection.