The Central Role of General Ability in Prediction

Three data sets are analyzed that permit double cross-validation of a test battery against criterion variables in a number of educational programs or jobs. The validity of the first general factor score is compared with that obtained from the set of cross-validated regression weights, and is found to account, respectively, for approximately 85, 90 and 120 percent as much criterion variance as the cross-validated regression weights. Small further contributions appear to be made by a mechanical/technical and by a psychomotor factor. However, for a wide range of criterion variables the major role in validity appears to be played by a common general factor.     

Taking the next step: combining incrementally valid indicators to improve recidivism prediction

The possibility of combining indicators to improve recidivism prediction was evaluated in a sample of released federal prisoners randomly divided into a derivation subsample (n = 550) and a cross-validation subsample (n = 551). Five incrementally valid indicators were selected from five domains: demographic (age), historical (prior convictions), adjustment (prior incident reports), rating scale (Violation scale of the Lifestyle Criminality Screening Form), and self-report (General Criminal Thinking score from the Psychological Inventory of Criminal Thinking Styles). After converting scores on the five indicators to a common scale (z score), two combined scores were calculated: a simple summed score (unweighted summed score) and a score computed using beta weights from a Cox survival analysis of the derivation subsample (weighted summed score). Correlational and receiver operating characteristic analyses revealed that the unweighted and weighted summed scores produced equivalent results and that both improved significantly on the results of the five contributing indicators.     

Methodological pitfalls in predicting counseling success

Prediction models employing multiple linear regression of raw scores, multiple linear regression of factor scores, the single best predictor, and a nine-point decision rule index were compared. The subjects were 296 clients undergoing vocational counseling and evaluation. Predictor variables included performance ratings, demographic variables, and WAIS subtest scores; the criterion was employment status upon program completion. The least statistically sophisticated model, employing the single best predictor, was the most successful approach. Considerable shrinkage in power of prediction was demonstrated upon cross-validation particularly for multiple linear regression of raw scores model, indicating the necessity of cross-validating prediction schemes. Additional suggestions are made to those designing prediction studies.     

儿童同伴关系的隐变量预测模型

摘要：以往对影响儿童同伴关系的因素的研究具有明显的元素主义倾向,为系统研究影响儿童同伴关系的因素,在总结以往研究成果的基础上,选用了4大量表和6个可测量指标,收集了478名小学四至六年级学生的有效问卷和可测量指标。统计结果表明,影响儿童同伴关系的首要因素是学校标签,然后依次是能力特质和焦虑,三个因素对模型的方差累积贡献率为74.887%;验证性因素分析模型的拟合度高,路径系数均极其显著;三个隐变量能够解释ZLM（同伴正向提名Z分数）62.2%的变异,能够解释ZLL（同伴反向提名Z分数）47.4%的变异,其中学校标签的预测力最强。     

A comparison of factor scores under conditions of factor obliquity

Six different methods of computing factor scores were investigated in a simulation study. Population scores created from oblique factor patterns selected from the psychological literature served as the bases for the simulations, and the stability of the different methods was assessed through cross-validation in a subject-sampling model. Results from 5 evaluative criteria indicated that a simplified, unit-weighting procedure based on the factor score coefficients was generally superior to several unit-weighting procedures based on the pattern or structure coefficients. This simplified method of computing factor scores also compared favorably with an exact-weighting scheme based on the full factor score coefficient matrix. Results are discussed with regard to their potential impact on current practice, and several recommendations are offered.     

Regression weights as a function of test length

A battery of pencil-and-paper tests is commonly used for predicting a single criterion. If the score on each test is the number of correct answers, the composite battery score would normally be the sum of the weighted test scores, where the weights are the raw score regression weights. Knowing the reliability of each test, it is possible to alter the lengths of the tests in a manner such that the weights will all be equal. The composite battery score would then simply be the total number of items answered correctly and scoring would be greatly simplified. Such simplification is particularly desirable where the volume of testing is large. Section I of the article outlines the procedure for altering the lengths of the tests, and Section II gives a proof of the method.     

EFFECTS OF ALTERNATIVE WEIGHTING METHODS IN A POLICY- CAPTURING APPROACH TO JOB EVALUATION: A REVIEW AND EMPIRICAL INVESTIGATION

This research was designed to examine differences in the predictive power of alternative scale weighting methods in the context of job evaluation. Two different point-factor job evaluation instruments were used to evaluate 71 managerial and service jobs in a metropolitan university, and five different weighting models were compared in terms of predictive validity and salary classification. For the job evaluation system having high multicollinearity and validity concentration, no significant differences in accuracy were found among the weighting methods. However, in the more heterogeneous system, prediction models based upon unit weights, correlational weights, and multiple regression had significantly higher predictive validity than models based upon equal raw score weights or rational weights developed by a job evaluation committee. In addition, the weighting models differed substantially in terms of the predicted policy wages and classification structures.     

Policy Capturing with Ridge Regression

A policy capturing method combining human judgment with ridge regression is offered which results in superior judgment policy models. The new method (termed smart ridge regression) was tested against four others in seven judgment policy capturing applications. Performance criteria were two cross-validation indices: cross-validated multiple correlation and mean squared error of prediction of new judgments. Smart ridge regression was found to outperform ordinary least squares regression and conventional ridge regression, as well as subjective weighting and equal weighting of cues.     

Activities, self-referent memory beliefs, and cognitive performance: evidence for direct and mediated relations

In this study, the authors investigated the role of activities and self-referent memory beliefs for cognitive performance in a life-span sample. A factor analysis identified 8 activity factors, including Developmental Activities, Experiential Activities, Social Activities, Physical Activities, Technology Use, Watching Television, Games, and Crafts. A second-order general activity factor was significantly related to a general factor of cognitive function as defined by ability tests. Structural regression models suggested that prediction of cognition by activity level was partially mediated by memory beliefs, controlling for age, education, health, and depressive affect. Models adding paths from general and specific activities to aspects of crystallized intelligence suggested additional unique predictive effects for some activities. In alternative models, nonsignificant effects of beliefs on activities were detected when cognition predicted both variables, consistent with the hypothesis that beliefs derive from monitoring cognition and have no influence on activity patterns.     

A MULTIDIMENSIONAL APPROACH FOR EVALUATING VARIABLES IN ORGANIZATIONAL RESEARCH AND PRACTICE

One of the most difficult tasks facing industrial-organizational psychologists is evaluating the importance of variables, especially new variables, to be included in the prediction of some outcome. When multiple regression is used, common practices suggest evaluating the usefulness of new variables by showing incremental validity beyond the set of existing variables. This approach assures that the new variables are not statistically redundant with this existing set, but this approach attributes any shared criterion-related validity to the existing set of variables and none to the new variables. More importantly, incremental validity alone fails to answer the question directly about the importance of variables included in a regression model—arguably the more important statistical concern for practitioners. To that end, the current article reviews 2 indices of relative importance, general dominance weights and relative weights, which may be used to complement incremental validity evidence and permit organizational decision makers to make more precise and informed decisions concerning the usefulness of predictor variables. We illustrate our approach by reanalyzing the correlation matrices from 2 published studies.     

A Meta-View of Multivariate Statistical Inference Methods in European Psychology Journals

We investigated the extent and nature of multivariate statistical inferential procedures used in eight European psychology journals covering a range of content (i.e., clinical, social, health, personality, organizational, developmental, educational, and cognitive). Multivariate methods included those found in popular texts that focused on prediction, group difference, and advanced modeling: multiple regression, logistic regression, analysis of covariance, multivariate analysis of variance, factor or principal component analysis, structural equation modeling, multilevel modeling, and other methods. Results revealed that an average of 57% of the articles from these eight journals involved multivariate analyses with a third using multiple regression, 17% using structural modeling, and the remaining methods collectively comprising about 50% of the analyses. The most frequently occurring inferential procedures involved prediction weights, dichotomous p values, figures with data, and significance tests with very few articles involving confidence intervals, statistical mediation, longitudinal analyses, power analysis, or meta-analysis. Contributions, limitations and future directions are discussed.     

Targeting Item-level Nuances Leads to Small but Robust Improvements in Personality Prediction from Digital Footprints

In the past decade, researchers have demonstrated that personality can be accurately predicted from digital footprint data, including Facebook likes, tweets, blog posts, pictures, and transaction records. Such computer-based predictions from digital footprints can complement—and in some circumstances even replace—traditional self-report measures, which suffer from well-known response biases and are difficult to scale. However, these previous studies have focused on the prediction of aggregate trait scores (i.e. a person's extroversion score), which may obscure prediction-relevant information at theoretical levels of the personality hierarchy beneath the Big 5 traits. Specifically, new research has demonstrated that personality may be better represented by so-called personality nuances—item-level representations of personality—and that utilizing these nuances can improve predictive performance. The present work examines the hypothesis that personality predictions from digital footprint data can be improved by first predicting personality nuances and subsequently aggregating to scores, rather than predicting trait scores outright. To examine this hypothesis, we employed least absolute shrinkage and selection operator regression and random forest models to predict both items and traits using out-of-sample cross-validation. In nine out of 10 cases across the two modelling approaches, nuance-based models improved the prediction of personality over the trait-based approaches to a small, but meaningful degree (4.25% or 1.69% on average, depending on method). Implications for personality prediction and personality nuances are discussed. © 2020 European Association of Personality Psychology     

An examination of the structural, discriminant, nomological, and incremental predictive validity of the MSCEIT© V2.0

We examined the structural, discriminant, nomological, and incremental predictive validity of a behavioral measure of emotional intelligence, using data from two undergraduate student samples. Covariance structure modeling indicated that the eight subscales of the MSCEIT© V2.0 were best modeled with a solution consisting of three first-order factors, and supported the existence of one higher-order factor of overall emotional intelligence. Multi-group confirmatory factor analyses indicated that the higher-order factor had discriminant validity from personality and conformity. Contrary to prediction, the higher-order factor was more highly correlated to social desirability than to general mental ability or long term affect. Finally, hierarchical regression results indicated that overall emotional intelligence did not predict incremental variance in either GPA or life satisfaction.     

Best univocal estimates of orthogonal common factors

A general formulation is presented for obtaining conditionally unbiased, univocal common-factor score estimates that have maximum validity for the true orthogonal factor scores. We note that although this expression is formally different from both Bartlett's formulation and Heermann's approximate expression, all three, while developed from very different rationales, yield identical results given that the common-factor model holds for the data. Although the true factor score validities can be raised by a different non-orthogonal transformation of orthogonalized regression estimates—as described by Mulaik—the resulting estimates lose their univocality.     

Replies to comments on "Which method delivers greater signal-to-noise ratio: Structural equation modelling or regression analysis with weighted composites?" by Yuan and Fang (2023)

Observational data typically contain measurement errors. Covariance-based structural equation modelling (CB-SEM) is capable of modelling measurement errors and yields consistent parameter estimates. In contrast, methods of regression analysis using weighted composites as well as a partial least squares approach to SEM facilitate the prediction and diagnosis of individuals/participants. But regression analysis with weighted composites has been known to yield attenuated regression coefficients when predictors contain errors. Contrary to the common belief that CB-SEM is the preferred method for the analysis of observational data, this article shows that regression analysis via weighted composites yields parameter estimates with much smaller standard errors, and thus corresponds to greater values of the signal-to-noise ratio (SNR). In particular, the SNR for the regression coefficient via the least squares (LS) method with equally weighted composites is mathematically greater than that by CB-SEM if the items for each factor are parallel, even when the SEM model is correctly specified and estimated by an efficient method. Analytical, numerical and empirical results also show that LS regression using weighted composites performs as well as or better than the normal maximum likelihood method for CB-SEM under many conditions even when the population distribution is multivariate normal. Results also show that the LS regression coefficients become more efficient when considering the sampling errors in the weights of composites than those that are conditional on weights.     

In Spite of Indeterminacy Many Common Factor Score Estimates Yield an Identical Reproduced Covariance Matrix

It was investigated whether commonly used factor score estimates lead to the same reproduced covariance matrix of observed variables. This was achieved by means of Schönemann and Steiger’s (1976) regression component analysis, since it is possible to compute the reproduced covariance matrices of the regression components corresponding to different factor score estimates. It was shown that Thurstone’s, Ledermann’s, Bartlett’s, Anderson-Rubin’s, McDonald’s, Krijnen, Wansbeek, and Ten Berge’s, as well as Takeuchi, Yanai, and Mukherjee’s score estimates reproduce the same covariance matrix. In contrast, Harman’s ideal variables score estimates lead to a different reproduced covariance matrix.     

Life History Correlates Of a Career Commitment To School Counseling

This study sought to explore the use of the scored life history as a predictive index of a career commitment to school counseling. Hypotheses concerning the relationship between life history data and a career commitment to teaching and administration were also investigated. Multiple regression analysis was used to obtain validity coefficients estimating the relationship between weighted linear combinations of life history data and the career commitment criteria. The regression weights obtained for the most promising biographical predictors were used to predict current occupational titles for a cross-validation sample. Statistical analysis of the data indicated that selected and empirically weighted life history materials are valid indices of counselor career commitment. For practical purposes, life history data for teaching and administration demonstrated a lack of consistent predictive power. The content of the most promising predictors for the three occupational groups was logically examined.     

Emotional and developmental aspects of human figure drawings in predicting school readiness

It was hypothesized that the combined use of Koppitz' developmental and emotional scores can improve the prediction of school readiness from children's Human Figure Drawings (HFD) when compared with each measure separately. The relationship of SES to HFD scores was also investigated. Multiple regression analysis was used with four factors—developmental score, emotional score, sex of the child, and SES of the school— to derive comparative correlations to test the hypotheses and to create a prediction equation to estimate the Metropolitan Readiness Test (MRT) scores of 141 kindergarten students ages 5 to 6 years representing three SES levels. Hit rate data were also presented. The results showed that the correlation for the combined developmental and emotional scores with the MRT score was not different than their separate correlations with the MRT. The SES factor was found to be significantly correlated (p < .01) with both developmental and emotional scores. The best single variable predictor of MRT scores was SES. The application of Bayes' Theorem of probability to the hit rate data showed that the nonreadiness predictions yielded by the developmental score, emotional score, and their combination are not better than chance predictions. Thus, it was found that Koppitz' developmental or emotional scoring systems are not viable screening instruments.     

Some pitfalls in the use of factor scores: The case of the Kirton adaption-innovation inventory

The use of factor analysis to generate scores on composite scales has some psychometric appeal as it allows for the partition of item variance. However, two major issues make such procedures problematic. The first concerns generalizing factor score weights from one sample to another and is essentially the problem of factor invariance. The second issue concerns the psychometric problem of factor indeterminacy. Mulligan and Martin have advocated the use of factor scores with the increasingly widely used Kirton Adaption-Innovation Inventory (KAI). This paper discusses the pitfalls of using factor scores in general and then investigates empirically whether Mulligan and Martin's proposal is justified with the KAI using an Irish sample. It is concluded that the summated rating method of scoring the KAI is superior to the factor-analytic procedure.