"SOME THINGS YOU LEARN AREN'T SO":

When interpreting an interaction in the analysis of variance (ANOVA), many active researchers (and, in turn, students) often Ignore the residuals defining the interaction Although this problem has been noted previously, a appears that many users of ANOVA remain uncertain about the proper understanding of interaction effects To clear up this problem, we review the way in which the ANOVA model enables us to take apart a table of group means or the individual measurements contributing to the means to reveal the underlying components We also show how (using only published data) to compute a contrast on the question that may be of primary interest and illustrate for interpreting tables of residuals We conclude with an exercise to check on students understanding of ANOVA and to encourage increased precision in the specification of research results     

Testing for negligible interaction: A coherent and robust approach

Researchers often want to demonstrate a lack of interaction between two categorical predictors on an outcome. To justify a lack of interaction, researchers typically accept the null hypothesis of no interaction from a conventional analysis of variance (ANOVA). This method is inappropriate as failure to reject the null hypothesis does not provide statistical evidence to support a lack of interaction. This study proposes a bootstrap‐based intersection–union test for negligible interaction that provides coherent decisions between the omnibus test and post hoc interaction contrast tests and is robust to violations of the normality and variance homogeneity assumptions. Further, a multiple comparison strategy for testing interaction contrasts following a non‐significant omnibus test is proposed. Our simulation study compared the Type I error control, omnibus power and per‐contrast power of the proposed approach to the non‐centrality‐based negligible interaction test of Cheng and Shao (2007, Statistica Sinica, 17, 1441). For 2 × 2 designs, the empirical Type I error rates of the Cheng and Shao test were very close to the nominal α level when the normality and variance homogeneity assumptions were satisfied; however, only our proposed bootstrapping approach was satisfactory under non‐normality and/or variance heterogeneity. In general a × b designs, although the omnibus Cheng and Shao test, as expected, is the most powerful, it is not robust to assumption violation and results in incoherent omnibus and interaction contrast decisions that are not possible with the intersection–union approach.     

Basic statistics and the inconsistency of multiple comparison procedures.

This paper has two main themes. First, the various statistical measures used in this journal are summarized and their interrelationships described by way of a flow chart. These are the pooled standard deviation, the pooled variance or mean square error (MSE), the standard error of each treatment mean (SEM) and of the difference between two treatment means (SED), and the least difference between two means which is significant at (e.g.) the 5% level of significance (LSD(5%)). The last three measures can be displayed as vertical bars in graphs, and the relationship between the lengths of these bars is graphically illustrated. It is suggested that the LSD is the most useful of these three measures. Second, when the experimenter has no prior hypotheses to be tested using analysis of variance "contrasts," a multiple comparison procedure (MCP) that examines all pair-wise differences between treatment means, may be appropriate. In this paper a fictitious experimental data set is used to compare several well-known McPs by focussing on a particular operating characteristic, the consistency of the results between an overall analysis of all treatments and an analysis of a subset of the experimental treatments. The procedure that behaves best according to this criterion is the unrestricted least significant difference (LSD) procedure. The unrestricted LSD is therefore recommended with the proviso that it be used as a method of generating hypotheses to be tested in subsequent experimentation, not as a method that attempts to simultaneously formulate and test hypotheses.     

The insidious effects of failing to include design-driven correlated residuals in latent-variable covariance structure analysis

In practice, the inclusion of correlated residuals in latent-variable models is often regarded as a statistical sleight of hand, if not an outright form of cheating. Consequently, researchers have tended to allow only as many correlated residuals in their models as are needed to obtain a good fit to the data. The current article demonstrates that this strategy leads to the underinclusion of residual correlations that are completely justified on the basis of measurement theory and research design. In many designs, the absence of such correlations will not substantially harm the fit of the model; however, failure to include them can change the meaning of the extracted latent variables and generate potentially misleading results. Recommendations include (a) returning to the full multitrait-multimethod design when measurement theory implies the existence of shared method variance and (b) abandoning the evil-but-necessary attitude toward correlated residuals when they reflect intended features of the research design.     

Structural validity of the Mental Health Continuum-Short Form: The bifactor model of emotional,social and psychological well-being

The Mental Health Continuum-Short Form (MHC-SF) is a growingly popular questionnaire designed to assess three components of well-being: emotional, social, and psychological. The main goal of the present study was to evaluate the structural validity of the MHC-SF and test the bifactor model of the MHC-SF, which includes one general factor and three specific factors of well-being. Sample 1 consisted of 1095 Serbian students (aged 18–26 years), while Sample 2 included 325 Serbian adults (aged 27–63 years). The bifactor model of the MHC-SF yielded the best fit to the data across the two samples. The results showed that the general factor of well-being accounted for substantially greater amount of variance of the MHC-SF than three specific factors of well-being. After controlling for the general factor, three specific factors explained a small portion of variance in well-being. In addition, the three subscales of the MHC-SF showed low reliability as estimated by omega-subscale coefficients, indicating that these subscales comprise too small amount of reliable variance to interpret. The present findings suggest that researchers should not calculate separate scores for three types of well-being when using the MHC-SF and that alternative measures of specific components of well-being should be considered.     

An introduction to Bayesian model selection for evaluating informative hypotheses

Most researchers have specific expectations concerning their research questions. These may be derived from theory, empirical evidence, or both. Yet despite these expectations, most investigators still use null hypothesis testing to evaluate their data, that is, when analysing their data they ignore the expectations they have. In the present article, Bayesian model selection is presented as a means to evaluate the expectations researchers have, that is, to evaluate so called informative hypotheses. Although the methodology to do this has been described in previous articles, these are rather technical and havemainly been published in statistical journals. The main objective of thepresent article is to provide a basic introduction to the evaluation of informative hypotheses using Bayesian model selection. Moreover, what is new in comparison to previous publications on this topic is that we provide guidelines on how to interpret the results. Bayesian evaluation of informative hypotheses is illustrated using an example concerning psychosocial functioning and the interplay between personality and support from family.     

Further evaluating the conditional decision rule for comparing two independent means

Many books on statistical methods advocate a ‘conditional decision rule’ when comparing two independent group means. This rule states that the decision as to whether to use a ‘pooled variance’ test that assumes equality of variance or a ‘separate variance’ Welch t test that does not should be based on the outcome of a variance equality test. In this paper, we empirically examine the Type I error rate of the conditional decision rule using four variance equality tests and compare this error rate to the unconditional use of either of the t tests (i.e. irrespective of the outcome of a variance homogeneity test) as well as several resampling‐based alternatives when sampling from 49 distributions varying in skewness and kurtosis. Several unconditional tests including the separate variance test performed as well as or better than the conditional decision rule across situations. These results extend and generalize the findings of previous researchers who have argued that the conditional decision rule should be abandoned.     

OUTLIER DETECTION AND TREATMENT IN I/O PSYCHOLOGY: A SURVEY OF RESEARCHER BELIEFS AND AN EMPIRICAL ILLUSTRATION

Extreme data points, or outliers, can have a disproportionate influence on the conclusions drawn from a set of bivariate correlational data. This paper addresses two aspects of outlier detection. The results of a survey regarding how published researchers prefer to deal with outliers are presented, and a set of 183 test validity studies is examined to document the effects of different approaches to the detection and exclusion of outliers on effect size measures. The study indicates that: (a) there is disagreement among researchers as to the appropriateness of deleting data points from a study; (b) researchers report greater use of visual examination of data than of numeric diagnostic techniques for detecting outliers; and (c) while outlier removal influenced effect size measures in individual studies, outlying data points were not found to be a substantial source of variance in a large test validity data set.     

Studentized maximum root procedures for coherent analyses of two-factor fixed-effects designs

The authors provide generalizations of R. J. Boik's (1993) studentized maximum root (SMR) procedure that allow for simultaneous inference on families of product contrasts including simple effect contrasts and differences among simple effect contrasts in coherent analyses of data from 2-factor fixed-effects designs. Unlike the F-based simultaneous test procedures (STPs) proposed by M. A. Betz and K. R. Gabriel (1978) for coherent analyses allowing for inferences on all factorial contrasts, SMR STPs are designed for analyses where each contrast of interest is a product contrast, which is usually the case in analyses of data from factorial experiments. When both factors have more than 2 levels, SMR STPs always provide more power and precision than F STPs for inferences on product contrasts.     

Interaction in ANOVA: misconceptions

Although interaction in analysis of variance has an unequivocal theoretical meaning (and so it appears in the statistic literature), frequent misconceptions are found in empirical research, which, in many cases, lead to wrong conclusions. In this paper, 150 articles are reviewed: in 12.7% of them, no attention is paid to the interaction (either because it is not analysed or because it is analysed but not discussed); in 79.1%, interaction is studied through simple effects analysis; and only in 8.2% of the cases, interaction is correctly discussed. It could be that psychology researchers tend to analyse and interpret the interaction between factors incorrectly because the most widespread statistic packages (with SPSS in the lead) do not allow performing the comparisons needed to analyse a significant interaction in factorial designs with randomized groups. In order to contribute to eradicating this problem, we herein show how to design some of the linear comparisons that allow isolating the interaction effect, and we explain how to use SPSS to compute these comparisons.     

Assessing the validity of inferences from scores on the cognitive reflection test

Decision‐making researchers purport that a novel cognitive ability construct, cognitive reflection, explains variance in intuitive thinking processes that traditional mental ability constructs do not. However, researchers have questioned the validity of the primary measure because of poor construct conceptualization and lack of validity studies. Prior studies have not adequately aligned the analytical techniques with the theoretical basis of the construct, dual‐processing theory of reasoning. The present study assessed the validity of inferences drawn from the cognitive reflection test (CRT) scores. We analyzed response processes with an item response tree model, a method that aligns with the dual‐processing theory in order to interpret CRT scores. Findings indicate that the intuitive and reflective factors that the test purportedly measures were indistinguishable. Exploratory, post hoc analyses demonstrate that CRT scores are most likely capturing mental abilities. We suggest that future researchers recognize and distinguish between individual differences in cognitive abilities and cognitive processes.     

Using the PLUM procedure of SPSS to fit unequal variance and generalized signal detection models

The recent addition of a procedure in SPSS for the analysis of ordinal regression models offers a simple means for researchers to fit the unequal variance normal signal detection model and other extended signal detection models. The present article shows how to implement the analysis and how to interpret the SPSS output. Examples of fitting the unequal variance normal model and other generalized signal detection models are given. The approach offers a convenient means for applying signal detection theory to a variety of research.     

Misinterpretation of interaction effects: a reply to Rosnow and Rosenthal.

The structural model of analysis of variance for multiway tables, with main effects and interaction parameters, is overspecified. Only the set of estimable functions of the cell means is useful. The individual estimates of the parameters are artificial as regards the underlying scientific process. Rosnow and Rosenthal (1989a) stated that it is "absolutely necessary" (pg 146) that interactions should be interpreted by examining the usual estimates of the interaction parameters. This is incorrect.     

Contrast discrimination under temporally varying contrast conditions

Psychophysical contrast discrimination of a 0.8-cpd vertical grating was tested using a paradigm that alternated test and masking gratings at 8 Hz. Masking contrasts were lower than, equal to, or higher than the test contrasts. Six test contrasts were combined factorially with six masking contrasts to generate a series of six contrast increment threshold versus test contrast curves (tvc curves). A particularly simple relationship existed between these curves. The curves could be brought into alignment by shifting them diagonally by the ratio of their masking contrasts. It is shown that this behavior is predicted by a model in which contrast gain is set by the average of the test and masking contrasts coupled with a simple model of contrast discrimination. Contrast gain control integrates contrast over a period of at least 125 msec, and contrast discrimination is a function of this time-averaged contrast.     

The relative contribution of formal and informal organizational work-family support

Recent work-family research has proposed that informal means of organizational work-family support (e.g., managerial support) are more useful than formal means of organizational work-family support (e.g., work-family benefit availability) in explaining variance in employee affective, intentional, and behavioral outcomes. However, the relative contributions of formal and informal means of support have not yet been empirically tested against each other. The proposition is tested using data from the Families and Work Institute's 1997 National Study of the Changing Workforce (Bond, Galinsky, & Swanberg, 1998). Two relative importance procedures are used to analyze the data. The results offer compelling evidence that informal means of work-family support explain a greater share of variance in employee outcomes than formal mechanisms do. Implications for researchers and practitioners are discussed.     

Robust nonorthogonal analyses revisited: An update based on trimmed means

Three approaches to the analysis of main and interaction effect hypotheses in nonorthogonal designs were compared in a 2×2 design for data that was neither normal in form nor equal in variance. The approaches involved either least squares or robust estimators of central tendency and variability and/or a test statistic that either pools or does not pool sources of variance. Specifically, we compared the ANOVA F test which used trimmed means and Winsorized variances, the Welch-James test with the usual least squares estimators for central tendency and variability and the Welch-James test using trimmed means and Winsorized variances. As hypothesized, we found that the latter approach provided excellent Type I error control, whereas the former two did not.Financial support for this research was provided by grants to the first author from the National Sciences and Engineering Research Council of Canada (#OGP0015855) and the Social Sciences and Humanities Research Council (#410-95-0006). The authors would like to express their appreciation to the Associate Editor as well as the reviewers who provided valuable comments on an earlier version of this paper.     

Robust misinterpretation of confidence intervals

Null hypothesis significance testing (NHST) is undoubtedly the most common inferential technique used to justify claims in the social sciences. However, even staunch defenders of NHST agree that its outcomes are often misinterpreted. Confidence intervals (CIs) have frequently been proposed as a more useful alternative to NHST, and their use is strongly encouraged in the APA Manual. Nevertheless, little is known about how researchers interpret CIs. In this study, 120 researchers and 442 students—all in the field of psychology—were asked to assess the truth value of six particular statements involving different interpretations of a CI. Although all six statements were false, both researchers and students endorsed, on average, more than three statements, indicating a gross misunderstanding of CIs. Self-declared experience with statistics was not related to researchers’ performance, and, even more surprisingly, researchers hardly outperformed the students, even though the students had not received any education on statistical inference whatsoever. Our findings suggest that many researchers do not know the correct interpretation of a CI. The misunderstandings surrounding p-values and CIs are particularly unfortunate because they constitute the main tools by which psychologists draw conclusions from data.     

The Role of Theory in Testing Hypothesized Interactions: An Example from the Research on Escalation of Commitment1

Theory should guide the choice of analytic strategy in testing research hypotheses. This thesis is developed primarily with respect to the testing of interaction hypotheses, with some discussion of the implications for testing hypotheses regarding main effects. Previously published data on the escalation bias (Schoorman, 1988) are re-analyzed using planned contrasts, two-way analysis of variance (ANOVA), and regression analyses using dummy and effect coded variables, with both simultaneous and hierarchical entry. Each analysis yields different results, which demonstrates that different analytic strategies can lead to conflicting interpretations regarding the theory of interest. The reasons for these differences and the implications of these findings for the choice of analytic strategies are discussed.     

Mother and Adolescent Reports of Associations Between Child Behavior Problems and Mother-Child Relationship Qualities: Separating Shared Variance from Individual Variance

This study contrasts results from different correlational methods for examining links between mother and child (N = 72 dyads) reports of early adolescent (M = 11.5 years) behavior problems and relationship negativity and support. Simple (Pearson) correlations revealed a consistent pattern of statistically significant associations, regardless of whether scores came from the same reporter or from different reporters. When correlations between behavior problems and relationship quality differed, within-reporter correlations were always greater in magnitude than between-reporter correlations. Dyadic (common fate) analyses designed for interdependent data decomposed within-reporter correlations into variance shared across reporters (dyadic correlations) and variance unique to specific reporters (individual correlations). Dyadic correlations were responsible for most associations between adolescent behavior problems and relationship negativity; after partitioning variance shared across reporters, no individual correlations emerged as statistically significant. In contrast, adolescent behavior problems were linked to relationship support via both shared variance and variance unique to maternal perceptions. Dyadic analyses provide a parsimonious alternative to multiple contrasts in instances when identical measures have been collected from multiple reporters. Findings from these analyses indicate that same-reporter variance bias should not be assumed in the absence of dyadic statistical analyses.     

Beyond the F test: Effect size confidence intervals and tests of close fit in the analysis of variance and contrast analysis

This article presents confidence interval methods for improving on the standard F tests in the balanced, completely between-subjects, fixed-effects analysis of variance. Exact confidence intervals for omnibus effect size measures, such as or and the root-mean-square standardized effect, provide all the information in the traditional hypothesis test and more. They allow one to test simultaneously whether overall effects are (a) zero (the traditional test), (b) trivial (do not exceed some small value), or (c) nontrivial (definitely exceed some minimal level). For situations in which single-degree-of-freedom contrasts are of primary interest, exact confidence interval methods for contrast effect size measures such as the contrast correlation are also provided.