Some developments in multivariate generalizability

This article is concerned with estimation of components of maximum generalizability in multifacet experimental designs involving multiple dependent measures. Within a Type II multivariate analysis of variance framework, components of maximum generalizability are defined as those composites of the dependent measures that maximize universe score variance for persons relative to observed score variance. The coefficient of maximum generalizability, expressed as a function of variance component matrices, is shown to equal the squared canonical correlation between true and observed scores. Emphasis is placed on estimation of variance component matrices, on the distinction between generalizability- and decision-studies, and on extension to multifacet designs involving crossed and nested facets. An example of a two-facet partially nested design is provided.Appreciation is expressed to the Office of Research in Medical Education, University of Texas Medical Branch, for permitting use of their data.     

Comment on Levy's “an empirical comparison of the Z-variance and Box-Scheffé tests for homogeneity of variance”

An alternative selection of subsample sizes for the Box-Scheffé test is compared to the singlem used in Levy, An empirical comparison of the Z-variance and Box-Scheffé tests for homogeneity of variance. Use of the alternative subsample sizes is shown to suggest greater power for the Box-Scheffé test, although the test would still be less powerful than the Z-variance test. Because of the nonrobustness of the Z-variance test and the robustness of the Box-Scheffé test, the latter is recommended as a general technique unless an experimenter has assurance that all populations in the experiment are normal. Alternative techniques are also considered.The opinions expressed do not necessarily reflect the position or policy of the U. S. Civil Service Commission.     

Best linear prediction of composite universe scores

The problem of predicting universe scores for samples of examinees based on their responses to samples of items is treated. A general measurement procedure is described in which multiple test forms are developed from a table of specifications and each form is administered to a different sample of examinees. The measurement model categorizes items according to the cells of such a table, and the linear function derived for minimizing error variance in prediction uses responses to these categories. In addition, some distinctions are drawn between aspects of the approach taken here and the familiar regressed score estimates.The author thanks Robert L. Brennan, Michael J. Kolen, and Richard Sawyer for helpful comments and corrections, and anonymous reviewers for suggested improvements.     

ANOVA and ANCOVA of pre- and post-test,ordinal data

With random assignment to treatments and standard assumptions, either a one-way ANOVA of post-test scores or a two-way, repeated measures ANOVA of pre- and post-test scores provides a legitimate test of the equal treatment effect null hypothesis for latent variable . In an ANCOVA for pre- and post-test variablesX andY which are ordinal measures of and , respectively, random assignment and standard assumptions ensure the legitimacy of inferences about the equality of treatment effects on latent variable . Sample estimates of adjustedY treatment means are ordinal estimators of adjusted post-test means on latent variable .     

Teaching Principal Components Using Correlations

Introducing principal components (PCs) to students is difficult. First, the matrix algebra and mathematical maximization lemmas are daunting, especially for students in the social and behavioral sciences. Second, the standard motivation involving variance maximization subject to unit length constraint does not directly connect to the “variance explained” interpretation. Third, the unit length and uncorrelatedness constraints of the standard motivation do not allow re-scaling or oblique rotations, which are common in practice. Instead, we propose to motivate the subject in terms of optimizing (weighted) average proportions of variance explained in the original variables; this approach may be more intuitive, and hence easier to understand because it links directly to the familiar “R-squared” statistic. It also removes the need for unit length and uncorrelatedness constraints, provides a direct interpretation of “variance explained,” and provides a direct answer to the question of whether to use covariance-based or correlation-based PCs. Furthermore, the presentation can be made without matrix algebra or optimization proofs. Modern tools from data science, including heat maps and text mining, provide further help in the interpretation and application of PCs; examples are given. Together, these techniques may be used to revise currently used methods for teaching and learning PCs in the behavioral sciences.     

The association between religion and self-reported academic honesty among college students

Current research yields inconsistent findings about the association between religious variables and academic cheating among college students. In this study, we investigated possible reasons for this disagreement by examining whether, and to what extent, three particular religious variables: religious identity, affirmation of importance and religious services attendance, are associated with academic honesty among college students. Specifically, we utilised a sample of 2503 American college-aged students from Gallup^® daily tracking survey and used analysis of variance (ANOVA) to address the proposed research question. Research findings indicate that religious service attendance is positively associated with academic honesty among college students. Specifically, students who attend religious services more frequently are less likely to be engaged in academic misconduct than students who attend less frequently. This finding remains consistent when other important factors such as student attitudes toward cheating and gender were included in the analysis.     

Personality versus traffic accidents; meta-analysis of real and method effects

ProblemThe association between personality and traffic accident involvement has been extensively researched, but the literature is difficult to summarise, because different personality instruments and statistics have been used, and effect sizes differ strongly between studies.MethodA meta-analysis of studies which had used measures of personality which could be converted into Big Five dimensions, and traffic accidents as the dependent variable, was undertaken.AnalysisOutlier values were identified and removed. Also, analyses on effects of common method variance, type of instrument, dissemination bias and restriction of variance were undertaken.ResultsOutlier problems exist in these data, which prohibit any certainty in the conclusions. Each of the 5 personality dimensions were predictors of accident involvement, but the effects were small (r < .1), which is much weaker than in a previous meta-analysis. Effect sizes were dependent upon variance in the accident variable, and the true (population) effects could therefore be larger than the present estimates, something which could be ascertained by new studies using high-risk samples over longer time periods. Newer studies and those using Big Five instruments tended to have smaller effects. No effects of common method variance could be found.ConclusionsTests of personality are weak predictors of traffic accident involvement, compared to other variables, such as previous accidents. Research into whether larger effects of personality can be found with methods other than self-reports is needed.     

Gender variance in children and adolescents with autism spectrum disorder from the National Database for Autism Research

Previous studies suggest elevated rates of gender variance (GV), the wish to be of the other gender, in those with autism spectrum disorder (ASD). This study aimed to understand the rate of GV in children and adolescents with ASD and explore differences in sex, age, and emotional-behavioral problems relative to those referred to clinical services for mental health concerns (“referred”) and to the general population (“non-referred”). A secondary analysis of data from the National Database for Autism Research was used to explore GV using a child behavior checklist, parent report, in 176 children aged 6 to 18 year with ASD compared to referred and non-referred cohorts. GV was present in 4.0% of the ASD group, higher than for the non-referred group (0.7%) but similar to the referred group (4.0%). There were no significant sex differences in GV prevalence (males 3.7%, females 6.0%) in the ASD group. That the GV rate was elevated in ASD relative to non-referred samples but similar to clinically referred samples suggests that elevated rates of GV were not specific to ASD and may be more broadly associated with neurodevelopmental and psychiatric disorders of childhood. Further population-based research using clinical assessment for gender dysphoria is required in individuals with ASD and other neurodevelopmental disorders.     

Alpha,Dimension-Free,and Model-Based Internal Consistency Reliability

As pointed out by Sijtsma (in press), coefficient alpha is inappropriate as a single summary of the internal consistency of a composite score. Better estimators of internal consistency are available. In addition to those mentioned by Sijtsma, an old dimension-free coefficient and structural equation model based coefficients are proposed to improve the routine reporting of psychometric internal consistency. The various ways to measure internal consistency are also shown to be appropriate to binary and polytomous items. Research supported in part by grants DA00017 and DA01070 from the National Institute on Drug Abuse. This paper is based in part on Bentler (2003).     

Does matching up features mess up job choice? Boundary conditions on attribute‐salience effects

Research in decision making has suggested that the degree to which features of an option are shared versus unique influences preferences in a way that violates normative rules. The generalizability of these findings to job choice was investigated. Senior‐level, undergraduate job seekers (N = 216) were presented with three jobs from which they were asked to choose one. Attributes for two of the jobs (A and B) remained invariant across conditions, and attributes for a third job (C) were manipulated such that it shared unfavorable features with one of the invariant jobs (A or B) and favorable attributes with the other job (B or A). Results suggested that jobs with unique positive features and shared negative features were preferred over those with unique negative features and shared positive features only when information was presented in a simple (versus complex) format and when participants did not rate the importance of attributes prior to the choice task. We suggest that inferences from feature‐matching research should be qualified by these boundary conditions. Copyright © 2002 John Wiley & Sons, Ltd.