Predicted differences and differences between predictions

WhenK tests are given toN individuals, and for each individual there are two criterion measures, then (1) the multiple regression weight to be applied to the standard score for each test to predict the criterion-difference score equals the difference of the weights for predicting each criterion separately; (2) the difference between the predicted scores equals the predicted difference (each test being assigned the appropriate multiple regression weight); (3) the square of the multiple correlation between predicted and actual criterion-difference scores equals the sum of squares of the multiple correlations of the battery with each criterion less the product of these correlations and the correlation between predicted scores all divided by twice the quantity one minus the criterion intercorrelation; and (4) the variance of errors of estimating the criterion-difference score equals the sum of the variances of errors of estimating each criterion score minus twice the criterion intercorrelation, plus twice the correlation between predicted scores multiplied by the product of the square root of one minus the variance of errors of estimating one criterion and the corresponding square root for the second criterion.The author wishes to express his appreciation for the suggestions and guidance given by Dr. Harold Gulliksen in the preparation of this article. He also wishes to acknowledge the helpful comments of Dr. Paul Horst and Dr. Ledyard Tucker on certain phases of the development.     

The discrimination of two racial samples

Nineteen different anthropometric measures were obtained on all members of each of two racial groups. A procedure was developed and applied to the data to give maximum differentiation between the groups. The method is applicable wherever we have a large number of independent variables and a dependent variable. In such cases, the conventional methods for determining multiple regression constants are very laborious. An iteration method is presented which is more rapid than any with which the writers are familiar. The method selects in sequence those variables which together yield the largest multiple correlation with the criterion. At each step in the procedure, rapid estimates of the regression weights and the multiple correlation at that point are available.     

The prediction of averages

Criterion measures are frequently obtained by averaging ratings, but the number and kind of ratings available may differ from individual to individual. This raises issues as to the appropriateness of any single regression equation, about the relation of variance about regression to number and kind of criterion observations, and about the preferred estimate of regression parameters. It is shown that if criterion ratings all have the same true score the regression equation for predicting the average is independent of the number and kind of criterion scores averaged.Two cases are distinguished, one where criterion measures are assumed to have the same true score, and the other where criterion measures have the same magnitude of error of measurement as well. It is further shown that the variance about regression is a function of the number and kind of criterion ratings averaged, generally decreasing as the number of measures averaged increases. Maximum likelihood estimates for the regression parameters are derived for the two cases, assuming a joint normal distribution for predictors and criterion average within each subpopulation of persons for whom the same type of criterion average is available.     

Moderation Analysis Using a Two-Level Regression Model

Moderation analysis is widely used in social and behavioral research. The most commonly used model for moderation analysis is moderated multiple regression (MMR) in which the explanatory variables of the regression model include product terms, and the model is typically estimated by least squares (LS). This paper argues for a two-level regression model in which the regression coefficients of a criterion variable on predictors are further regressed on moderator variables. An algorithm for estimating the parameters of the two-level model by normal-distribution-based maximum likelihood (NML) is developed. Formulas for the standard errors (SEs) of the parameter estimates are provided and studied. Results indicate that, when heteroscedasticity exists, NML with the two-level model gives more efficient and more accurate parameter estimates than the LS analysis of the MMR model. When error variances are homoscedastic, NML with the two-level model leads to essentially the same results as LS with the MMR model. Most importantly, the two-level regression model permits estimating the percentage of variance of each regression coefficient that is due to moderator variables. When applied to data from General Social Surveys 1991, NML with the two-level model identified a significant moderation effect of race on the regression of job prestige on years of education while LS with the MMR model did not. An R package is also developed and documented to facilitate the application of the two-level model.     

Multiple bi-serial and multiple point bi-serial correlation

Normal equations, using data in various forms, are presented for securing the regression weights for prediction of a dichotomized criterion, and a simplified equation for the estimation of the multiple bi-serial or multiple point bi-serial, depending upon the proper assumption as to the nature of the distribution of the criterion, on the basis of these maximal weights is given also. The weights, unaffected by the assumption as to the nature of the criterion, are identical (or proportional) to those found by the discriminant function approach based upon analysis of variance. The author holds that the present multiple correlation approach is both easier and more informative than the discriminant function (analysis of variance) approach and suggests that the discriminant function be abandoned in favor of multiple bi-serial and/or multiple point bi-serial correlation and regression.     

Retrieving relevant factors with exploratory SEM and principal-covariate regression: A comparison

Behavioral researchers often linearly regress a criterion on multiple predictors, aiming to gain insight into the relations between the criterion and predictors. Obtaining this insight from the ordinary least squares (OLS) regression solution may be troublesome, because OLS regression weights show only the effect of a predictor on top of the effects of other predictors. Moreover, when the number of predictors grows larger, it becomes likely that the predictors will be highly collinear, which makes the regression weights’ estimates unstable (i.e., the “bouncing beta” problem). Among other procedures, dimension-reduction-based methods have been proposed for dealing with these problems. These methods yield insight into the data by reducing the predictors to a smaller number of summarizing variables and regressing the criterion on these summarizing variables. Two promising methods are principal-covariate regression (PCovR) and exploratory structural equation modeling (ESEM). Both simultaneously optimize reduction and prediction, but they are based on different frameworks. The resulting solutions have not yet been compared; it is thus unclear what the strengths and weaknesses are of both methods. In this article, we focus on the extents to which PCovR and ESEM are able to extract the factors that truly underlie the predictor scores and can predict a single criterion. The results of two simulation studies showed that for a typical behavioral dataset, ESEM (using the BIC for model selection) in this regard is successful more often than PCovR. Yet, in 93% of the datasets PCovR performed equally well, and in the case of 48 predictors, 100 observations, and large differences in the strengths of the factors, PCovR even outperformed ESEM.     

A note on permutation tests of significance for multiple regression coefficients

In the vast majority of psychological research utilizing multiple regression analysis, asymptotic probability values are reported. This paper demonstrates that asymptotic estimates of standard errors provided by multiple regression are not always accurate. A resampling permutation procedure is used to estimate the standard errors. In some cases the results differ substantially from the traditional least squares regression estimates.     

How Many Subjects Does It Take To Do A Regression Analysis

Numerous rules-of-thumb have been suggested for determining the minimum number of subjects required to conduct multiple regression analyses. These rules-of-thumb are evaluated by comparing their results against those based on power analyses for tests of hypotheses of multiple and partial correlations. The results did not support the use of rules-of-thumb that simply specify some constant (e.g., 100 subjects) as the minimum number of subjects or a minimum ratio of number of subjects (N) to number of predictors (m). Some support was obtained for a rule-of-thumb that N ≥ 50 + 8 m for the multiple correlation and N ≥104 + m for the partial correlation. However, the rule-of-thumb for the multiple correlation yields values too large for N when m ≥ 7, and both rules-of-thumb assume all studies have a medium-size relationship between criterion and predictors. Accordingly, a slightly more complex rule-of thumb is introduced that estimates minimum sample size as function of effect size as well as the number of predictors. It is argued that researchers should use methods to determine sample size that incorporate effect size.     

Identifying criterion-related patterns of predictor scores using multiple regression

Along with examples involving vocational interests and mathematics achievement, the authors describe a multiple regression based, pattern recognition procedure that can be used to identify a pattern of predictor scores associated with high scores on a criterion variable. This pattern is called the criterion pattern. After the criterion pattern has been identified, a second regression procedure can be used to estimate the proportion of variation attributable to the criterion pattern. Cross-validation can then be used to estimate the variation attributable to a criterion pattern derived from regression weights estimated in another sample. Finally, issues of criterion pattern invariance and interpretation are discussed.     

Scale validity evaluation with congeneric measures in hierarchical designs

A procedure for validity estimation of multi-component measuring instruments in hierarchical designs is outlined. The method is developed within the framework of the popular latent variable modelling methodology. The approach is readily applicable for examining interrelationship indices between composite sum scores and criterion variables in two-level studies. The procedure is also useful for obtaining estimates and confidence intervals of criterion validity coefficients during the process of scale construction and development with hierarchical data. The method is illustrated with an empirical example.     

Iterative least squares estimates of communality: Initial estimate need not affect stabilized value

A common criticism of iterative least squares estimates of communality is that method of initial estimation may influence stabilized values. As little systematic research on this topic has been performed, the criticism appears to be based on cumulated experience with empirical data sets. In the present paper, two studies are reported in which four types of initial estimate (unities, squared multiple correlations, highestr, and zeroes) and four levels of convergence criterion were employed using four widely available computer packages (BMDP, SAS, SPSS, and SOUPAC). The results suggest that initial estimates have no effect on stabilized communality estimates when a stringent criterion for convergence is used, whereas initial estimates appear to affect stabilized values employing rather gross convergence criteria. There were no differences among the four computer packages for matrices without Heywood cases.     

Properties of the maximum likelihood solution in factor analysis regression

Algebraic properties of the normal theory maximum likelihood solution in factor analysis regression are investigated. Two commonly employed measures of the within sample predictive accuracy of the factor analysis regression function are considered: the variance of the regression residuals and the squared correlation coefficient between the criterion variable and the regression function. It is shown that this within sample residual variance and within sample squared correlation may be obtained directly from the factor loading and unique variance estimates, without use of the original observations or the sample covariance matrix.     

A program for fitting two-phase segmented-curve models with an unknown change point,with an application to the analysis of strategy shifts in a cognitive task

A program is described for fitting a regression model in which the relationship between the dependent and the independent variables is described by two regression equations, one for each of two mutually exclusive ranges of the independent variable. The point at which the change from one equation to the other occurs is often unknown, and thus must be estimated. In cognitive psychology, such models are relevant for studying the phenomenon of strategy shifts. The program uses a (weighted) least squares algorithm to estimate the regression parameters and the change point. The algorithm always finds the global minimum of the error sum of squares. The model is applied to data from a mental-rotation experiment. The program’s estimates of the point at which the strategy shift occurs are compared with estimates obtained from a nonlinear least squares minimization procedure in SPSSX.     

When is a diagnosis worth making? A statistical comparison of two prediction strategies

For a setting in which the sole goal is to predict a criterion accurately, two strategies are compared, (1) multiple linear regression directly from a set of predictors and (2) using predictors to diagnose individuals and then using only the diagnosis to predict the criterion. These are mathematical models of two common methods for making clinical predictions. This article derives equations for each statistical strategy's validity (and a formula comparing them, for a special case). Prediction accuracies are compared over a simplified but broad parameter space and four conclusions follow. Changing disorder prevalences (in the range 0.1 to 0.5) have small effect on the relative preferability of the two strategies; the effect of varying prevalence differs according to population separation on predictors and criterion. As within-population covariances of predictors with criterion decline below zero (assuming variables are scaled so that the less common population scores higher on predictors and criterion), diagnoses become increasingly preferable as a prediction strategy; as they rise above zero the opposite trend is observed. As populations become better separated on predictors or criterion, diagnoses compare more favorably. Most importantly, over almost all of the parameter space which one would expect to encounter in clinical psychology and psychiatry, multiple linear regression is much superior.     

Factor analysis by minimizing residuals (minres)

This paper is addressed to the classical problem of estimating factor loadings under the condition that the sum of squares of off-diagonal residuals be minimized. Communalities consistent with this criterion are produced as a by-product. The experimental work included several alternative algorithms before a highly efficient method was developed. The final procedure is illustrated with a numerical example. Some relationships of minres to principal-factor analysis and maximum-likelihood factor estimates are discussed, and several unresolved problems are pointed out.The authors wish to thank the Factor Analysis Work Group (supported, in part, by ONR) for valuable criticisms and suggestions made in the course of a discussion of the present work in April, 1965.     

Confidence depends on level of aggregation

The credible intervals that people set around their point estimates are typically too narrow (cf. Lichtenstein, Fischhoff, & Phillips, 1982). That is, a set of many such intervals does not contain the actual values of the criterion variables as often as it should given the probability assigned to this event for each estimate. The typical interpretation of such data is that people are overconfident about the accuracy of their judgments. This paper presents data from two studies showing the typical levels of overconfidence for individual estimates of unknown quantities. However, data from the same subjects on a different measure of confidence for the same items, their own global assessment for the set of multiple estimates as a whole, showed significantly lower levels of confidence and overconfidence than their average individual assessment for items in the set. It is argued that the event and global assessments of judgment quality are fundamentally different and are affected by unique psychological processes. Finally, we discuss the implications of a difference between confidence in single and multiple estimates for confidence research and theory.     

Regularized Partial and/or Constrained Redundancy Analysis

Methods of incorporating a ridge type of regularization into partial redundancy analysis (PRA), constrained redundancy analysis (CRA), and partial and constrained redundancy analysis (PCRA) were discussed. The usefulness of ridge estimation in reducing mean square error (MSE) has been recognized in multiple regression analysis for some time, especially when predictor variables are nearly collinear, and the ordinary least squares estimator is poorly determined. The ridge estimation method was extended to PRA, CRA, and PCRA, where the reduced rank ridge estimates of regression coefficients were obtained by minimizing the ridge least squares criterion. It was shown that in all cases they could be obtained in closed form for a fixed value of ridge parameter. An optimal value of the ridge parameter is found by G-fold cross validation. Illustrative examples were given to demonstrate the usefulness of the method in practical data analysis situations. We thank Jim Ramsay for his insightful comments on an earlier draft of this paper. The work reported in this paper is supported by Grants 10630 from the Natural Sciences and Engineering Research Council of Canada to the first author.     

Testing Differential Item Functioning in Small Samples

Abstract

Differential item functioning (DIF) is a pernicious statistical issue that can mask true group differences on a target latent construct. A considerable amount of research has focused on evaluating methods for testing DIF, such as using likelihood ratio tests in item response theory (IRT). Most of this research has focused on the asymptotic properties of DIF testing, in part because many latent variable methods require large samples to obtain stable parameter estimates. Much less research has evaluated these methods in small sample sizes despite the fact that many social and behavioral scientists frequently encounter small samples in practice. In this article, we examine the extent to which model complexity—the number of model parameters estimated simultaneously—affects the recovery of DIF in small samples. We compare three models that vary in complexity: logistic regression with sum scores, the 1-parameter logistic IRT model, and the 2-parameter logistic IRT model. We expected that logistic regression with sum scores and the 1-parameter logistic IRT model would more accurately estimate DIF because these models yielded more stable estimates despite being misspecified. Indeed, a simulation study and empirical example of adolescent substance use show that, even when data are generated from / assumed to be a 2-parameter logistic IRT, using parsimonious models in small samples leads to more powerful tests of DIF while adequately controlling for Type I error. We also provide evidence for minimum sample sizes needed to detect DIF, and we evaluate whether applying corrections for multiple testing is advisable. Finally, we provide recommendations for applied researchers who conduct DIF analyses in small samples.     

Akaike's Information Criterion and Recent Developments in Information Complexity

In this paper we briefly study the basic idea of Akaike's (1973) information criterion (AIC). Then, we present some recent developments on a new entropic or information complexity (ICOMP) criterion of Bozdogan (1988a, 1988b, 1990, 1994d, 1996, 1998a, 1998b) for model selection. A rationale for ICOMP as a model selection criterion is that it combines a badness-of-fit term (such as minus twice the maximum log likelihood) with a measure of complexity of a model differently than AIC, or its variants, by taking into account the interdependencies of the parameter estimates as well as the dependencies of the model residuals. We operationalize the general form of ICOMP based on the quantification of the concept of overall model complexity in terms of the estimated inverse-Fisher information matrix. This approach results in an approximation to the sum of two Kullback-Leibler distances. Using the correlational form of the complexity, we further provide yet another form of ICOMP to take into account the interdependencies (i.e., correlations) among the parameter estimates of the model. Later, we illustrate the practical utility and the importance of this new model selection criterion by providing several real as well as Monte Carlo simulation examples and compare its performance against AIC, or its variants. Copyright 2000 Academic Press.     

Ordinal test fidelity estimated by an item sampling model

A test theory using only ordinal assumptions is presented. It is based on the idea that the test items are a sample from a universe of items. The sum across items of the ordinal relations for a pair of persons on the universe items is analogous to a true score. Using concepts from ordinal multiple regression, it is possible to estimate the tau correlations of test items with the universe order from the taus among the test items. These in turn permit the estimation of the tau of total score with the universe. It is also possible to estimate the odds that the direction of a given observed score difference is the same as that of the true score difference. The estimates of the correlations between items and universe and between total score and universe are found to agree well with the actual values in both real and artificial data.Part of this paper was presented at the June, 1989, Meeting of the Psychometric Society. The authors wish to thank several reviewers for their suggestions. This research was mainly done while the second author was a University Fellow at the University of Southern California.