Testing for negligible interaction: A coherent and robust approach

Researchers often want to demonstrate a lack of interaction between two categorical predictors on an outcome. To justify a lack of interaction, researchers typically accept the null hypothesis of no interaction from a conventional analysis of variance (ANOVA). This method is inappropriate as failure to reject the null hypothesis does not provide statistical evidence to support a lack of interaction. This study proposes a bootstrap‐based intersection–union test for negligible interaction that provides coherent decisions between the omnibus test and post hoc interaction contrast tests and is robust to violations of the normality and variance homogeneity assumptions. Further, a multiple comparison strategy for testing interaction contrasts following a non‐significant omnibus test is proposed. Our simulation study compared the Type I error control, omnibus power and per‐contrast power of the proposed approach to the non‐centrality‐based negligible interaction test of Cheng and Shao (2007, Statistica Sinica, 17, 1441). For 2 × 2 designs, the empirical Type I error rates of the Cheng and Shao test were very close to the nominal α level when the normality and variance homogeneity assumptions were satisfied; however, only our proposed bootstrapping approach was satisfactory under non‐normality and/or variance heterogeneity. In general a × b designs, although the omnibus Cheng and Shao test, as expected, is the most powerful, it is not robust to assumption violation and results in incoherent omnibus and interaction contrast decisions that are not possible with the intersection–union approach.     

A Method for the Quantitative Recording of Eye Movements

Many writers have implicitly or explicitly stated that nonparametric tests are free from the assumption of homogeneity of variance. Nonparametric tests for difference in central tendencies generally involve the assumption of homogeneity of variance. The assumption of homogeneity of variance for the t test and for nonparametric tests serves the same purpose: it allows the user to draw more specific inferences when the null hypothesis is rejected.     

Effect of non-normality on test statistics for one-way independent groups designs

The data obtained from one‐way independent groups designs is typically non‐normal in form and rarely equally variable across treatment populations (i.e. population variances are heterogeneous). Consequently, the classical test statistic that is used to assess statistical significance (i.e. the analysis of variance F test) typically provides invalid results (e.g. too many Type I errors, reduced power). For this reason, there has been considerable interest in finding a test statistic that is appropriate under conditions of non‐normality and variance heterogeneity. Previously recommended procedures for analysing such data include the James test, the Welch test applied either to the usual least squares estimators of central tendency and variability, or the Welch test with robust estimators (i.e. trimmed means and Winsorized variances). A new statistic proposed by Krishnamoorthy, Lu, and Mathew, intended to deal with heterogeneous variances, though not non‐normality, uses a parametric bootstrap procedure. In their investigation of the parametric bootstrap test, the authors examined its operating characteristics under limited conditions and did not compare it to the Welch test based on robust estimators. Thus, we investigated how the parametric bootstrap procedure and a modified parametric bootstrap procedure based on trimmed means perform relative to previously recommended procedures when data are non‐normal and heterogeneous. The results indicated that the tests based on trimmed means offer the best Type I error control and power when variances are unequal and at least some of the distribution shapes are non‐normal.     

New heterogeneous test statistics for the unbalanced fixed‐effect nested design

When the underlying variances are unknown or/and unequal, using the conventional F test is problematic in the two‐factor hierarchical data structure. Prompted by the approximate test statistics (Welch and Alexander–Govern methods), the authors develop four new heterogeneous test statistics to test factor A and factor B nested within A for the unbalanced fixed‐effect two‐stage nested design under variance heterogeneity. The actual significance levels and statistical power of the test statistics were compared in a simulation study. The results show that the proposed procedures maintain better Type I error rate control and have greater statistical power than those obtained by the conventional F test in various conditions. Therefore, the proposed test statistics are recommended in terms of robustness and easy implementation.     

Evaluating clinical significance: Incorporating robust statistics with normative comparison tests

The purpose of this study was to evaluate a modified test of equivalence for conducting normative comparisons when distribution shapes are non‐normal and variances are unequal. A Monte Carlo study was used to compare the empirical Type I error rates and power of the proposed Schuirmann–Yuen test of equivalence, which utilizes trimmed means, with that of the previously recommended Schuirmann and Schuirmann–Welch tests of equivalence when the assumptions of normality and variance homogeneity are satisfied, as well as when they are not satisfied. The empirical Type I error rates of the Schuirmann–Yuen were much closer to the nominal α level than those of the Schuirmann or Schuirmann–Welch tests, and the power of the Schuirmann–Yuen was substantially greater than that of the Schuirmann or Schuirmann–Welch tests when distributions were skewed or outliers were present. The Schuirmann–Yuen test is recommended for assessing clinical significance with normative comparisons.     

Active or Inert? An Experimental Comparison of Creative Ideation across Incubation Periods

The primary objective of this study was to contribute to the growing research discipline investigating the effects of physical exercise on divergent thinking creativity performance. Thirty‐two students participated in this two‐visit, within‐subject intervention. Individuals consented to participate in two randomized, and counterbalanced, experimental conditions, consisting of 15 min of active treadmill walking and an inert, seated control incubation period. Creativity was assessed at baseline and post‐exercise (and control) via the Instances Creativity Task (ICT). Creativity scores for changes in fluency (F(1, 31) = 2.90, p = .10) were not statistically significant across the experimental conditions. Originality scores were higher at baseline and follow‐up when compared to the exercise condition (F(1, 31) = 6.82, p = .01). However, there was no statistically significant condition × time interaction effect (F(1, 31) = 1.78, p = .19). Further analyses demonstrated that there was no statistically significant difference between the experimental conditions on recall score (F(1, 31) = 1.04, p = .32). All models indicated statistically significant main effects for time: fluency (F(1, 31) = 131.17, p < .001); originality (F(1, 31) = 36.54, p < .001); and recall (F(1, 31) = 51.75, p < .001). These findings suggest that both active and inert creative incubation periods may similarly enhance subsequent divergent thinking performance.     

On sample size calculation for 2×2 fixed‐effect ANOVA when variances are unknown and possibly unequal

The factorial 2 × 2 fixed‐effect ANOVA is a procedure used frequently in scientific research to test mean differences between‐subjects in all of the groups. But if the assumption of homogeneity is violated, the test for the row, column, and the interaction effect might be invalid or less powerful. Therefore, for planning research in the case of unknown and possibly unequal variances, it is worth developing a sample size formula to obtain the desired power. This article suggests a simple formula to determine the sample size for 2 × 2 fixed‐effect ANOVA for heterogeneous variances across groups. We use the approximate Welch t test and consider the variance ratio to derive the formula. The sample size determination requires two‐step iterations but the approximate sample sizes needed for the main effect and the interaction effect can be determined separately with the specified power. The present study also provides an example and a SAS program to facilitate the calculation process.     

The Effectiveness of Functional Family Therapy for Adolescent Behavioral and Substance Misuse Problems: A Meta‐Analysis

A systematic review of published and unpublished English language articles identified 14 studies containing 18 comparisons between functional family therapy (FFT) and another condition in the treatment of adolescent disruptive behavior and substance use disorders. In 11 of these comparisons, assignment to conditions was random, while nonrandom assignment occurred in seven studies. For both random and nonrandom comparisons, separate meta‐analyses were conducted for subgroups of studies depending on the type of comparison group used. Data from studies of untreated control groups (CTL), treatment as usual (TAU), and well‐defined alternative treatments (ALTs) were analyzed separately. Effect sizes from these six meta‐analyses were as follows: random assignment FFT versus CTL (k = 3, d = 0.48, p < .01); random assignment FFT versus TAU (k = 3, d = .20, ns); random assignment FFT versus ALT (k = 5, d = .35, p < .05); nonrandom assignment FFT versus CTL (k = 2, d = .90, ns); nonrandom assignment FFT versus TAU (k = 2, d = .08, ns); and nonrandom assignment FFT versus ALT (k = 3, d = .75, p < .001). These results provide support for the effectiveness of FFT compared with untreated controls and well‐defined ALTs, such as cognitive behavior therapy, other models of family therapy, and individual and group therapy for adolescents.     

Further evaluating the conditional decision rule for comparing two independent means

Many books on statistical methods advocate a ‘conditional decision rule’ when comparing two independent group means. This rule states that the decision as to whether to use a ‘pooled variance’ test that assumes equality of variance or a ‘separate variance’ Welch t test that does not should be based on the outcome of a variance equality test. In this paper, we empirically examine the Type I error rate of the conditional decision rule using four variance equality tests and compare this error rate to the unconditional use of either of the t tests (i.e. irrespective of the outcome of a variance homogeneity test) as well as several resampling‐based alternatives when sampling from 49 distributions varying in skewness and kurtosis. Several unconditional tests including the separate variance test performed as well as or better than the conditional decision rule across situations. These results extend and generalize the findings of previous researchers who have argued that the conditional decision rule should be abandoned.     

An empirical study of the effect of heterogeneous within-groups variance upon certainF-tests of significance in analysis of variance

In the application of the analysis of variance to data obtained in educational methods experiments which involve several classes of several schools, one assumption is that of homogeneity in the variances of pupil scores from school to school. It is shown that such variances on representative educational achievement tests are heterogeneous. The effects of this heterogeneity upon theF-tests of significance commonly employed in methods experiments are investigated by comparing the actual distribution ofF values for a large number of experiments involving marked heterogeneity with a theoretical distribution based on the assumption of homogeneity. Although the findings, which vary somewhat with the type of variance ratio, are not entirely conclusive, they apparently demonstrate that departure from homogeneity does not invalidate the use of the customaryF-tests for evaluating results of the typical methods experiment.     

Mapping differential responses to cognitive training using machine learning

We used two simple unsupervised machine learning techniques to identify differential trajectories of change in children who undergo intensive working memory (WM) training. We used self‐organizing maps (SOMs)—a type of simple artificial neural network—to represent multivariate cognitive training data, and then tested whether the way tasks are represented changed as a result of training. The patterns of change we observed in the SOM weight matrices implied that the processes drawn upon to perform WM tasks changed following training. This was then combined with K‐means clustering to identify distinct groups of children who respond to the training in different ways. Firstly, the K‐means clustering was applied to an independent large sample (N = 616, M_age = 9.16 years, range = 5.16–17.91 years) to identify subgroups. We then allocated children who had been through cognitive training (N = 179, M_age = 9.00 years, range = 7.08–11.50 years) to these same four subgroups, both before and after their training. In doing so, we were able to map their improvement trajectories. Scores on a separate measure of fluid intelligence were predictive of a child's improvement trajectory. This paper provides an alternative approach to analysing cognitive training data that go beyond considering changes in individual tasks. This proof‐of‐principle demonstrates a potentially powerful way of distinguishing task‐specific from domain‐general changes following training and of establishing different profiles of response to training.     

How satisfied are you with your job? Estimating the reliability of scores on a single‐item job satisfaction measure

Estimating the reliability of scores on single‐item measures can be difficult because commonly used internal consistency estimates of reliability cannot be calculated. When longitudinal data is available, statistical models can be used to decompose the variability in the latent variable at each wave into trait versus state variance. Then, reliability can be estimated as a ratio of the sum of the trait variance that is captured in repeated assessments over the total variance. The current study used latent trait‐state‐error models on a nine‐year longitudinal data (N = 5,003) to estimate the test–retest reliability of scores on a single‐item measure of job satisfaction. Results showed that job satisfaction scores were somewhat unreliable (r_xx = .49–.59) and amenable to change.     

Similarities and Differences in the Functions of Nonsuicidal Self‐Injury (NSSI) and Sex as Self‐Injury (SASI)

Differences and similarities were studied in the functions of two different self‐injurious behaviors (SIB): nonsuicidal self‐injury (NSSI) and sex as self‐injury (SASI). Based on type of SIB reported, adolescents were classified in one of three groups: NSSI only (n = 910), SASI only (n = 41), and both NSSI and SASI (n = 76). There was support for functional equivalence in the two forms of SIB, with automatic functions being most commonly endorsed in all three groups. There were also functional differences, with adolescents in the SASI only group reporting more social influence functions than those with NSSI only. Adolescents reporting both NSSI and SASI endorsed the highest number of functions for both behaviors. Clinical implications are discussed, emphasizing the need for emotion regulation skills.     

Immoral,infectious, or both? How disgust sensitivity predicts judgments of violations against COVID-19 mitigation actions

Violations against mitigation actions to prevent the spreading of the SARS-CoV-2 virus causing COVID-19, such as not wearing a mask or not practicing social distancing, were seen as immoral and could also increase the likelihood of spreading the virus. In two studies (N1 = 318, N2 = 293), we found that moral and pathogen disgust sensitivity differentially predicted perceptions of such COVID-19 violations against mitigation actions, framed as a moral, pathogen, or on a good-bad dimension, albeit in a less specific way than initially hypothesized (e.g., regarding the pathogenic framed violations, not only pathogen but also moral disgust was associated with higher perceptions of infectiousness). These results suggest that individual differences, especially in pathogen disgust (and, more inconsistently, moral disgust), are important when evaluating violations against mitigation actions. Further research on the role of moral disgust is needed.     

Repression and Coping Styles in Asthmatic Patients

The objectives of this study were to (a) determine prevalence of the dispositional repressive coping style as well as other situational coping styles in a sample of asthmatic patients and (b) to analyze the capacity of these styles to predict subsequent morbidity (emergency room visits or hospitalizations due to asthma) during a 12-month follow-up. A sample of 75 adult asthmatic patients was selected and information about sociodemographics, asthma severity, and patient’s perception of illness severity was collected. Repressive coping style was defined by a combination of scores obtained on the Trait Anxiety Inventory and the Marlowe-Crowne Social Desirability Scale. Coping styles were assessed with the dispositional version of the Coping Orientation to Problems Experienced Inventory. Eighteen patients (24%) were classified as repressors. Repressor asthmatics obtained scores significantly lower on Emotion-Focused Coping compared to non-repressors (F _(1,72) = 5.15, p = .026). Patients who perceived their asthma as severe reported to use Emotion-Focused Coping more than those who judged it as mild or moderate (F _(2,71) = 4.83, p = .011). A higher use of Denial (an Emotion-Focused strategy of coping) explained 8% of variance of the frequency of emergency room visits during the 12-month follow-up. The prevalence of repressive coping style in the asthmatic population is similar to that registered in other populations of chronic patients, and it is also associated with the tendency to report a lower use of strategies traditionally considered as maladaptive. The use of Emotion-Focused Coping strategies seems to be related to a worse perception of the physical status, and among this group of strategies, Denial also could favor a poor clinical course in bronchial asthma.     

A comparative study of robust tests for spread: Asymmetric trimming strategies

We examined 633 procedures that can be used to compare the variability of scores across independent groups. The procedures, except for one, were modifications of the procedures suggested by Levene (1960) and O'Brien (1981). We modified their procedures by substituting robust measures of the typical score and variability, rather than relying on classical estimators. The robust measures that we utilized were either based on a priori or empirically determined symmetric or asymmetric trimming strategies. The Levene‐type and O'Brien‐type transformed scores were used with either the ANOVA F test, a robust test due to Lee and Fung (1985), or the Welch (1951) test. Based on four measures of robustness, we recommend a Levene‐type transformation based upon empirically determined 20% asymmetric trimmed means, involving a particular adaptive estimator, where the transformed scores are then used with the ANOVA F test.     

Trust and Involvement as Higher-Order Factors of General Attitudes Towards Politics. Testing a Structural Model Across 26 Democracies

Interindividual differences in how people think and feel about politics have been investigated for decades. However, the great number of attitudinal concepts that has been developed to describe these differences is likely to distract from their conceptual overlap and dimensional structure. In addition, not much is known about the cross-cultural invariance of their interrelation. We propose that attitudes towards politics can be structured by two broad higher-order factors, a factor of general political involvement and a factor of general political trust. In two studies (N₁ = 767; N₂ = 29,018), including representative samples from 26 democracies, we first conduct several confirmatory factor analyses (CFA) to show that the higher-order model is a well-fitting and parsimonious alternative to a baseline model without higher-order factors in most samples. Second, we present evidence from multigroup CFA that the overall patterns of factor loadings are the same across all 26 countries. We interpret this structural equivalence across different democracies as support for the assumption that general political involvement and general political trust reflect basic orientations towards politics that are based on (1) demands of democratic political systems and (2) universal principles in human trait structure.     

Student outcomes after 1 year of front line staff implementation of the PEAK curriculum

Systems‐level interventions built by behavior analysts often rely on others to implement, and this may be especially true in public education settings where behavior analysts are scarce. This study evaluated the effectiveness of the Promoting the Emergence of Advanced Knowledge Direct Training (PEAK‐DT) curriculum when implemented by school teachers and direct care staff. Thirty‐nine children with autism took part in the study (19 PEAK, 15 control), where the experimental group received applied behavior analytic instruction through the PEAK‐DT curriculum, and the quasi‐randomized control group received training as usual. The PEAK‐DT assessment was first administered to the participants at the onset of the study and again following 1 year. Participants who received PEAK training gained more skills on the PEAK‐DT assessment compared to the control group (PEAK: M = 16.0, SD = 17.8; control: M = 6.1, SD = 14.4, F(1,33) = 10.66, p < .05), suggesting that systems level implementation of behavior analytic procedures can be effective in teaching language skills as prescribed in a packaged curriculum designed by behavior analysts.     

The Experience in Personal Social Systems Questionnaire (EXIS.pers): Development and Psychometric Properties

This study presents the theoretical background, development, and psychometric properties of the German and English versions of the Experience in Personal Social Systems Questionnaire (EXIS.pers). It assesses how the members of a personal social system experience their situation within that system. It is designed as a research tool for interventions in which only one member of the system participates (e.g., Family Constellation Seminars). The EXIS.pers was created to measure change on the individual level relating to one's own important personal social system. In Study 1, we used exploratory factor analysis (EFA) for latent variable identification of the original German EXIS.pers (n = 179). In Studies 2 and 3, we used confirmatory factor analysis (CFA) to examine the dimensionality of the German (n = 634) and English (n = 310) EXIS.pers. Internal consistencies and cross‐cultural structural equivalence were assessed. EFA indicated that a four‐factor model provided best fit for the German EXIS.pers. For both the German and English EXIS.pers, CFA provided the best fit for a five‐factor bi‐level model that included a general factor (Experience In Personal Social Systems) and four dimensions (Belonging, Autonomy, Accord, Confidence). Good internal consistencies, external associations, and cross‐cultural structural equivalence were demonstrated. This study provides first evidence for the German and English EXIS.pers as an economical and reliable measure of an individual's experience within his or her personal social systems.     

The relationship between cognitive ability and personality scores in selection situations: A meta‐analysis

Several faking theories have identified applicants’ cognitive ability (CA) as a determinant of faking—the intentional distortion of answers by candidates—but the corresponding empirical findings in the area of personality tests are often ambiguous. Following the assumption that CA is important for faking, we expected applicants with high CA to show higher personality scores in selection situations, leading in this case to significant correlations between CA and personality scores, but not in nonselection situations. This meta‐analysis (66 studies, k = 115 individual samples, N = 46,265) showed this pattern of results as well as moderation effects for the study design (laboratory vs. field), the response format of the personality test, and the type of CA test.