The effect of nonindependence on significance testing in dyadic research

Relationship researchers regularly gather data from both members of the dyad, and these two scores are likely to be correlated. This nonindependence of observations can bias p values in significance testing if person is the unit in the statistical analysis. A method for determining how much bias results from dyadic interdependence is presented. Correction factors based on the degree of interdependence, design type, and the number of dyads are used to adjust the F statistic and its degrees of freedom to produce a corrected p value. Bias depends on the type of design and the degree of nonindependence, while the number of dyads in the study ordinarily has only a small effect on bias. Various strategies for controlling for nonindependence are briefly reviewed.     

Two SPSS programs for interpreting multiple regression results

When multiple regression is used in explanation-oriented designs, it is very important to determine both the usefulness of the predictor variables and their relative importance. Standardized regression coefficients are routinely provided by commercial programs. However, they generally function rather poorly as indicators of relative importance, especially in the presence of substantially correlated predictors. We provide two user-friendly SPSS programs that implement currently recommended techniques and recent developments for assessing the relevance of the predictors. The programs also allow the user to take into account the effects of measurement error. The first program, MIMR-Corr.sps, uses a correlation matrix as input, whereas the second program, MIMR-Raw.sps, uses the raw data and computes bootstrap confidence intervals of different statistics. The SPSS syntax, a short manual, and data files related to this article are available as supplemental materials from http:// brm.psychonomic-journals.org/content/supplemental.     

SPSS and SAS programs for generalizability theory analyses

The identification and reduction of measurement errors is a major challenge in psychological testing. Most investigators rely solely on classical test theory for assessing reliability, whereas most experts have long recommended using generalizability theory instead. One reason for the common neglect of generalizability theory is the absence of analytic facilities for this purpose in popular statistical software packages. This article provides a brief introduction to generalizability theory, describes easy to use SPSS, SAS, and MATLAB programs for conducting the recommended analyses, and provides an illustrative example, using data (N= 329) for the Rosenberg Self-Esteem Scale. Program output includes variance components, relative and absolute errors and generalizability coefficients, coefficients for D studies, and graphs of D study results.     

Simple and flexible SAS and SPSS programs for analyzing lag-sequential categorical data

This paper describes simple and flexible programs for analyzing lag-sequential categorical data, using SAS and SPSS. The programs read a stream of codes and produce a variety of lag-sequential statistics, including transitional frequencies, expected transitional frequencies, transitional probabilities, adjusted residuals, z values, Yule’s Q values, likelihood ratio tests of stationarity across time and homogeneity across groups or segments, transformed kappas for unidirectional dependence, bidirectional dependence, parallel and nonparallel dominance, and significance levels based on both parametric and randomization tests.     

SPSS and SAS programs for comparing Pearson correlations and OLS regression coefficients

Several procedures that use summary data to test hypotheses about Pearson correlations and ordinary least squares regression coefficients have been described in various books and articles. To our knowledge, however, no single resource describes all of the most common tests. Furthermore, many of these tests have not yet been implemented in popular statistical software packages such as SPSS and SAS. In this article, we describe all of the most common tests and provide SPSS and SAS programs to perform them. When they are applicable, our code also computes 100 × (1 ? α)% confidence intervals corresponding to the tests. For testing hypotheses about independent regression coefficients, we demonstrate one method that uses summary data and another that uses raw data (i.e., Potthoff analysis). When the raw data are available, the latter method is preferred, because use of summary data entails some loss of precision due to rounding.     

Some approximate tests for repeated measurement designs

Four approximate tests are considered for repeated measurement designs in which observations are multivariate normal with arbitrary covariance matrices. In these tests traditional within-subject mean square ratios are compared with critical values derived fromF distributions with adjusted degrees of freedom. Two of them—the approximate and the improved general approximate (IGA) tests—behave adequately in terms of Type I error. Generally, the IGA test functions better than the approximate test, however the latter involves less computations. In regards to power, the IGA test may compete with one multivariate procedure when the assumptions of the latter are tenable.The author wishes to thank Garrett K. Mandeville for his careful reading of the final version of the paper.     

SPSS and SAS programs for addressing interdependence and basic levels-of-analysis issues in psychological data

Levels-of-analysis issues arise whenever individual-level data are collected from more than one person from the same dyad, family, classroom, work group, or other interaction unit. Interdependence in data from individuals in the same interaction units also violates the independence-of-observations assumption that underlies commonly used statistical tests. This article describes the data analysis challenges that are presented by these issues and presents SPSS and SAS programs for conducting appropriate analyses. The programs conduct the within- and-between-analyses described by Dansereau, Alutto, and Yammarino (1984) and the dyad-level analyses describedby Gonzalez and Griffin (1999) and Griffin and Gonzalez (1995). Contrasts with general multilevel modeling procedures are then discussed.     

Power analysis for multivariate and repeated measures designs: A flexible approach using the SPSS MANOVA procedure

Although power analysis is an important component in the planning and implementation of research designs, it is often ignored. Computer programs for performing power analysis are available, but most have limitations, particularly for complex multivariate designs. An SPSS procedure is presented that can be used for calculating power for univariate, multivariate, and repeated measures models with and without time-varying and time-constant covariates. Three examples provide a framework for calculating power via this method: an ANCOVA, a MANOVA, and a repeated measures ANOVA with two or more groups. The benefits and limitations of this procedure are discussed.     

SPSS and SAS programs for addressing interdependence and basic levels-of-analysis issues in psychological data.

Levels-of-analysis issues arise whenever individual-level data are collected from more than one person from the same dyad, family, classroom, work group, or other interaction unit. Interdependence in data from individuals in the same interaction units also violates the independence-of-observations assumption that underlies commonly used statistical tests. This article describes the data analysis challenges that are presented by these issues and presents SPSS and SAS programs for conducting appropriate analyses. The programs conduct the within-and-between-analyses described by Dansereau, Alutto, and Yammarino (1984) and the dyad-level analyses described by Gonzalez and Griffin (1999) and Griffin and Gonzalez (1995). Contrasts with general multilevel modeling procedures are then discussed.     

Relational similarity and the nonindependence of features in similarity judgments

Four experiments examined the hypothesis that simple attributional features and relational features operate differently in the determination of similarity judgments. Forced choice similarity judgments ("Is X or Y more similar to Z?") and similarity rating tasks demonstrate that making the same featural change in two geometric stimuli unequally affects their judged similarity to a third stimulus (the comparison stimulus). More specifically, a featural change that causes stimuli to be more superficially similar and less relationally similar increases judged similarity if it occurs in stimuli that already share many superficial attributes, and decreases similarity if it occurs in stimuli that do not share as many superficial attributes. These results argue against an assumption of feature independence which asserts that the degree to which a feature shared by two objects affects similarity is independent of the other features shared by the objects. The MAX hypothesis is introduced, in which attributional and relational similarities are separately pooled, and shared features affect similarity more if the pool they are in is already relatively large. The results support claims that relations and attributes are psychologically distinct and that formal measures of similarity should not treat all types of matching features equally.     

A comparison of factor analysis programs in SPSS,BMDP, and SAS

Factor analysis programs in SAS, BMDP, and SPSS are discussed and compared in terms of documentation, methods and options available, internal logic, computational accuracy, and results provided. Some problems with respect to logic and output are described. Based on these comparisons, recommendations are offered which include a clear overall preference for SAS, and advice against general use of SPSS for factor analysis.     

The nonindependence of stimulus properties in human category learning

Typically, models of category learning are verified through behavioral experiments with stimuli consisting of putatively independent dimensions such as shape, size, and color. The assumption of independence is critical in both the design of behavioral experiments and the development of models and theories of learning. Using the standard classification learning paradigm and a common stimulus set, the present work demonstrates that the assumption of independence is unwarranted. Systematic relations span stimulus dimensions and govern learning performance. For example, shape is not independent of size and color, because humans quantify size and color over shape when shape is relevant to the categorization. This quantification is reflected in natural language use (e.g., "blue triangle" as opposed to "triangle and blue"). In this example, color and size are predicates and shape is the argument. Across four experiments, the difficulty of mastering a classification rule can be predicted by the number of predicates that must be unbound in order to free rule-relevant stimulus dimensions.     

Variation of the standard error of measurement

As usually interpreted, the standard error of measurement is assumed to be constant throughout the test-score range. In this investigation the standard error of measurement was assumed to be not higher than a second-degree function of the test score. By conceiving a test score to be made up of the scores on two parallel tests, an equation was derived for predicting the standard error of measurement from the test score. In the derivation the corresponding first four moments of the score distributions for the parallel tests were assumed to be identical, and certain errors of estimate involved in predicting the second test score from the first were assumed to be uncorrelated with powers of the score on the first test. An empirical verification was carried out, using nine synthetic tests and a 1000-case sample, and showed good agreement between predicted and observed results. The findings indicated that the standard error of measurement was constant only for a symmetrical, mesokurtic distribution of scores.This study was carried out while the author was a National Research Council Predoctoral Fellow in psychology at Princeton University.The author wishes to express his appreciation for the guidance given by his thesis adviser, Professor Harold Gulliksen. He wishes also to acknowledge his gratitude to the Educational Testing Service for extensive assistance in the empirical phase of the study, and to Dr. Ledyard Tucker for suggesting efficient methods of handling special computational problems.     

Ill-structured measurement designs in organizational research: implications for estimating interrater reliability

Organizational research and practice involving ratings are rife with what the authors term ill-structured measurement designs (ISMDs)--designs in which raters and ratees are neither fully crossed nor nested. This article explores the implications of ISMDs for estimating interrater reliability. The authors first provide a mock example that illustrates potential problems that ISMDs create for common reliability estimators (e.g., Pearson correlations, intraclass correlations). Next, the authors propose an alternative reliability estimator--G(q,k)--that resolves problems with traditional estimators and is equally appropriate for crossed, nested, and ill-structured designs. By using Monte Carlo simulation, the authors evaluate the accuracy of traditional reliability estimators compared with that of G(q,k) for ratings arising from ISMDs. Regardless of condition, G(q,k) yielded estimates as precise or more precise than those of traditional estimators. The advantage of G(q,k) over the traditional estimators became more pronounced with increases in the (a) overlap between the sets of raters that rated each ratee and (b) ratio of rater main effect variance to true score variance. Discussion focuses on implications of this work for organizational research and practice.     

Independence and nonindependence: A simple method for comparing groups using multiple measures and the binomial test

Existing methods for conducting analyses of small group data are either highly complicated or yield low power. Both of these limitations provide disincentives for the progress of research in this field. An alternative method modelled on the sign (binomial) test which involves comparing the differences of distributions based on multiple observations of each of the groups is presented. The calculations involved in the procedure are extremely simple. It is suggested that because the method enhances researchers' ability to make sound statistical inferences easily this should stimulate research on group‐level processes and on social interaction more generally. Copyright © 2004 John Wiley & Sons, Ltd.     

A test of the equality of standard errors of measurement

A procedure is proposed for testing the significance of group differences in the standard error of measurement of a psychological test. Wilks' criterion is used to assure that the tests used in ascertaining reliability and hence variance of errors of measurement may be assumed parallel for each group. Votaw's criterion may be used to check whether the test scores of all the groups have the same mean, variance, and covariance. It is possible, however, for the variance and reliability of the test to differ widely from group to group, so that Votaw's criterion is not satisfied even though the variance of errors of measurement stays relatively constant. For this case a modification of Neyman and Pearson's criterion is developed to test agreement among standard errors of measurement despite group differences in mean, variance, and reliability of the test.The author wishes to acknowledge the helpful criticisms of Dr. Harold Gulliksen, who suggested the problem.