Quantifying Parsimony in Structural Equation Modeling

Fitting propensity (FP) is defined as a model's average ability to fit diverse data patterns, all else being equal. The relevance of FP to model selection is examined in the context of structural equation modeling (SEM). In SEM it is well known that the number of free model parameters influences FP, but other facets of FP are routinely excluded from consideration. It is shown that models possessing the same number of free parameters but different structures may exhibit different FPs. The consequences of this fact are demonstrated using illustrative examples and models culled from published research. The case is made that further attention should be given to quantifying FP in SEM and considering it in model selection. Practical approaches are suggested.     

When do individuals help close others improve? The role of information diagnosticity

On the basis of the self-evaluation maintenance model (SEM; Tesser, 1988), it was hypothesized that individuals give less improving information to relationally close (rather than distant) others, out of concern for being outperformed by close others in the future. Further, this effect only occurs if diagnostic and valid criteria for success are present. Three studies confirmed the hypotheses. In Studies 1 and 2, participants gave less improving information to familiar than to unfamiliar others in a domain (academics) in which diagnostic assessment criteria (grades) were available. This pattern was not found in a domain (social life) without diagnostic criteria. These results were replicated in Study 3, in which relative performance and diagnosticity of assessment criteria were manipulated and amount of improving information given to friends and strangers was measured. Diagnosticity of comparison information is an important addition to the SEM model.     

An Introduction to Model Selection

This paper is an introduction to model selection intended for nonspecialists who have knowledge of the statistical concepts covered in a typical first (occasionally second) statistics course. The intention is to explain the ideas that generate frequentist methodology for model selection, for example the Akaike information criterion, bootstrap criteria, and cross-validation criteria. Bayesian methods, including the Bayesian information criterion, are also mentioned in the context of the framework outlined in the paper. The ideas are illustrated using an example in which observations are available for the entire population of interest. This enables us to examine and to measure effects that are usually invisible, because in practical applications only a sample from the population is observed. The problem of selection bias, a hazard of which one needs to be aware in the context of model selection, is also discussed. Copyright 2000 Academic Press.     

Pairwise Likelihood Ratio Tests and Model Selection Criteria for Structural Equation Models with Ordinal Variables

Correlated multivariate ordinal data can be analysed with structural equation models. Parameter estimation has been tackled in the literature using limited-information methods including three-stage least squares and pseudo-likelihood estimation methods such as pairwise maximum likelihood estimation. In this paper, two likelihood ratio test statistics and their asymptotic distributions are derived for testing overall goodness-of-fit and nested models, respectively, under the estimation framework of pairwise maximum likelihood estimation. Simulation results show a satisfactory performance of type I error and power for the proposed test statistics and also suggest that the performance of the proposed test statistics is similar to that of the test statistics derived under the three-stage diagonally weighted and unweighted least squares. Furthermore, the corresponding, under the pairwise framework, model selection criteria, AIC and BIC, show satisfactory results in selecting the right model in our simulation examples. The derivation of the likelihood ratio test statistics and model selection criteria under the pairwise framework together with pairwise estimation provide a flexible framework for fitting and testing structural equation models for ordinal as well as for other types of data. The test statistics derived and the model selection criteria are used on data on ‘trust in the police’ selected from the 2010 European Social Survey. The proposed test statistics and the model selection criteria have been implemented in the R package lavaan.     

Postselection Inference in Structural Equation Modeling

Abstract

Most statistical inference methods were established under the assumption that the fitted model is known in advance. In practice, however, researchers often obtain their final model by some data-driven selection process. The selection process makes the finally fitted model random, and it also influences the sampling distribution of the estimator. Therefore, implementing naive inference methods may result in wrong conclusions—which is probably a prime source of the reproducibility crisis in psychological science. The present study accommodates three valid state-of-the-art postselection inference methods for structural equation modeling (SEM) from the statistical literature: data splitting (DS), postselection inference (PoSI), and the polyhedral (PH) method. A simulation is conducted to compare the three methods with the commonly used naive procedure under selection events made by L₁-penalized SEM. The results show that the naive method often yields incorrect inference, and that the valid methods control the coverage rate in most cases with their own pros and cons. Real world data examples show the practical use of the valid inference methods.     

On Maraun's Deconstructing of Factor Indeterminancy with Constructed Factors

The nonlinear random coefficient model has become increasingly popular as a method for describing individual differences in longitudinal research. Although promising, the nonlinear model it is not utilized as often as it might be because software options are still somewhat limited. In this article we show that a specialized version of the model can be fit to data using SEM software. The specialization is to a model in which the parameters that enter the function in a linear manner are random, whereas those that are nonlinear are common to all individuals. Although this kind of function is not as general as is the fully nonlinear model, it still is applicable to many different data sets. Two examples are presented to show how the models can be estimated using popular SEM computer programs.     

Clinical and research diagnostic criteria for developmental coordination disorder: a review and discussion

The aim of this review was to investigate the selection criteria used in the past in studies of children with developmental motor problems (excluding those suffering from neurological dysfunctions such as cerebral palsy, muscular dystrophy, etc.). We therefore conducted an extensive analysis of 176 publications. First, an overview of the main characteristics of these studies (terminology, population, type and purpose) and the selection criteria that are reported in these publications are presented. Following this, the DSM-IV selection criteria for developmental coordination disorder (DCD) are contrasted with the selection criteria reported in 41 publications that have used this terminology to classify the children. The results of this comparison show that the inclusion criteria are largely followed, albeit with little consistency concerning selection instruments and quantitative cut-offs, while adherence to the exclusion criteria is not common practice. Strengths and weaknesses of the DSM-IV criteria, complementary to the previous discussion by Henderson and Barnett in the HMS special issue on DCD in 1998 on this same topic, are discussed. The results of the review also show that many studies have used additional selection criteria related to the specific research questions of the study concerned. In the broader context of clinical practice as well as basic research, the latter result suggests the usefulness of a distinction between Clinical Diagnostic Criteria and Research Diagnostic Criteria. This distinction helps to develop a unifying view on the use of diagnostic criteria for research and clinical practice. We conclude with a number of recommendations concerning the selection criteria for children with DCD.     

Integration of Analytic Hierarchy Process with Regression Analysis to Identify Attractive Locations for Market Expansion

Selection of new geographies in which to expand is a key decision for businesses aspiring to go beyond the opportunities in the existing markets. The conventional approaches of market selection can only provide a set of systematic steps for problem solving without considering the relationships between the decision factors. Decision models based on statistical techniques are able to examine the relationship between decision factors but are unable to effectively assist decision makers in identifying the most promising market, particularly in terms of prioritizing across decision factors. Analytic Hierarchy Process (AHP) is a commonly used approach for choosing alternatives by prioritizing across multiple decision factors. The typical AHP modelling requires knowledge of criteria and/or alternatives along with their relative weights, generally elicited from field experts. Quite often, firms encounter situations where decision makers are aware of only the overall objective and a set of earmarked geographies for setting up market locations while being relatively unaware of decision criteria and relative weights. This precludes using AHP to identify promising market locations. This paper conceptualizes a market selection decision model that integrates AHP with statistical modelling techniques to identify the attractive market locations for the purpose of expansion. The model first uses principal component analysis and multiple regression to determine significant decision criteria and their weights. Thereafter, it applies AHP to prioritize the market locations across the decision criteria. This integrative approach is illustrated for identifying the attractive locations in rural markets for a steel firm in India. The major advantage of this approach is that unlike the existing models, it works in situations when firms have not enough knowledge about factors for evaluating alternative market locations. Another key advantage of the proposed model is that of economizing resources for data collection on variables representing decision factors. Copyright © 2012 John Wiley & Sons, Ltd.     

Applying Occam’s razor in modeling cognition: A Bayesian approach

In mathematical modeling of cognition, it is important to have well-justified criteria for choosing among differing explanations (i.e., models) of observed data. This paper introduces a Bayesian model selection approach that formalizes Occam’s razor, choosing the simplest model that describes the data well. The choice of a model is carried out by taking into account not only the traditional model selection criteria (i.e., a model’s fit to the data and the number of parameters) but also the extension of the parameter space, and, most importantly, the functional form of the model (i.e., the way in which the parameters are combined in the model’s equation). An advantage of the approach is that it can be applied to the comparison of non-nested models as well as nested ones. Application examples are presented and implications of the results for evaluating models of cognition are discussed.     

Toward a contemporary quantitative model of punishment

A direct‐suppression, or subtractive, model of punishment has been supported as the qualitatively and quantitatively superior matching law‐based punishment model (Critchfield, Paletz, MacAleese, & Newland, 2003; de Villiers, 1980; Farley, 1980). However, this conclusion was made without testing the model against its predecessors, including the original (Herrnstein, 1961) and generalized (Baum, 1974) matching laws, which have different numbers of parameters. To rectify this issue, we reanalyzed a set of data collected by Critchfield et al. (2003) using information theoretic model selection criteria. We found that the most advanced version of the direct‐suppression model (Critchfield et al., 2003) does not convincingly outperform the generalized matching law, an account that does not include punishment rates in its prediction of behavior allocation. We hypothesize that this failure to outperform the generalized matching law is due to significant theoretical shortcomings in model development. To address these shortcomings, we present a list of requirements that all punishment models should satisfy. The requirements include formal statements of flexibility, efficiency, and adherence to theory. We compare all past punishment models to the items on this list through algebraic arguments and model selection criteria. None of the models presented in the literature thus far meets all of the requirements.     

Stepwise variable selection in factor analysis

It is very important to choose appropriate variables to be analyzed in multivariate analysis when there are many observed variables such as those in a questionnaire. What is actually done in scale construction with factor analysis is nothing but variable selection.In this paper, we take several goodness-of-fit statistics as measures of variable selection and develop backward elimination and forward selection procedures in exploratory factor analysis. Once factor analysis is done for a certain numberp of observed variables (thep-variable model is labeled the current model), simple formulas for predicted fit measures such as chi-square, GFI, CFI, IFI and RMSEA, developed in the field of the structural equation modeling, are provided for all models obtained by adding an external variable (so that the number of variables isp + 1) and for those by deleting an internal variable (so that the number isp – 1), provided that the number of factors is held constant.A programSEFA (Stepwise variable selection in Exploratory Factor Analysis) is developed to actually obtain a list of the fit measures for all such models. The list is very useful in determining which variable should be dropped from the current model to improve the fit of the current model. It is also useful in finding a suitable variable that may be added to the current model. A model with more appropriate variables makes more stable inference in general.The criteria traditionally often used for variable selection is magnitude of communalities. This criteria gives a different choice of variables and does not improve fit of the model in most cases.The URL of the programSEFA is http://koko15.hus.osaka-u.ac.jp/~harada/factor/stepwise/.     

Candidates' Ability to Identify Criteria in Nontransparent Selection Procedures: Evidence from an assessment center and a structured interview

In selection procedures like assessment centers (ACs) and structured interviews, candidates are often not informed about the targeted criteria. Previous studies have shown that candidates' ability to identify these criteria (ATIC) is related to their performance in the respective selection procedure. However, past research has studied ATIC in only one selection procedure at a time, even though it has been assumed that ATIC is consistent across situations, which is a prerequisite for ATIC to contribute to selection procedures' criterion‐related validity. In this study, 95 candidates participated in an AC and a structured interview. ATIC scores showed cross‐situational consistency across the two procedures and accounted for part of the relationship between performance in the selection procedures. Furthermore, ATIC scores in one procedure predicted performance in the other procedure even after controlling for cognitive ability. Implications and directions for future research are discussed.     

Get the Message: Targeting Beliefs to Develop Risk Reduction Intervention Messages for African American Adolescents

Risk reduction interventions that promote condom use, a vital component of most HIV prevention interventions, have been successful in increasing condom use among African American adolescents. Understanding theoretical components that lead to behavior change and selecting relevant risk reduction messages remain important considerations for targeting new interventions and tailoring existing interventions. The present study sought to (1) identify the most important theoretical determinants of condom use intention in African American adolescent males and females, separately, using the integrative model of behavior prediction, and (2) identify underlying beliefs within the determinants that were good candidates for message development in similar interventions. Using 446 African American adolescents, multi‐group SEM indicated that the gender‐specific IM exhibited a better fit than the overall model. Specifically, the IM had a stronger capacity for predicting condom use intention and condom use behavior for adolescent boys. Using a specific criteria for message selection, specific condom use beliefs were discussed as potential candidate messages for both African American males and females.     

A Comparison of Population-Averaged and Cluster-Specific Approaches in the Context of Unequal Probabilities of Selection

Sampling designs of large-scale survey studies are typically complex, involving multiple design features such as clustering and unequal probabilities of selection. Single-level (i.e., population-averaged) methods that use adjusted variance estimators and multilevel (i.e., cluster-specific) methods provide two alternatives for modeling clustered data. Although the literature comparing these methods is vast, comparisons have been limited to the context in which all sampling units are selected with equal probabilities (thus circumventing the need for sampling weights). The goal of this study was to determine under what conditions single-level and multilevel estimators outperform one another in the context of a two-stage sampling design with unequal probabilities of selection. Monte Carlo simulation methods were used to evaluate the impact of several factors, including population model, informativeness of the design, distribution of the outcome variable, intraclass correlation coefficient, cluster size, and estimation method. Results indicated that the unweighted estimators performed similarly across conditions, whereas the weighted single-level estimators tended to outperform the weighted multilevel estimators, particularly under nonideal sample conditions. Multilevel weight approximation methods did not perform well when the design was informative. An empirical example is provided to demonstrate how researchers might investigate the implications of the simulation results in practice.     

Selecting the best unbalanced repeated measures model

This study examined the performance of selection criteria available in the major statistical packages for both mean model and covariance structure. Unbalanced designs due to missing data involving both a moderate and large number of repeated measurements and varying total sample sizes were investigated. The study also investigated the impact of using different estimation strategies for information criteria, the impact of different adjustments for calculating the criteria, and the impact of different distribution shapes. Overall, we found that the ability of consistent criteria in any of the their examined forms to select the correct model was superior under simple covariance patterns than under complex covariance patterns, and vice versa for the efficient criteria. The simulation studies covered in this paper also revealed that, regardless of method of estimation used, the consistent criteria based on number of subjects were more effective than the consistent criteria based on total number of observations, and vice versa for the efficient criteria. Furthermore, results indicated that, given a dataset with missing values, the efficient criteria were more affected than the consistent criteria by the lack of normality.     

ARE WE GETTING FOOLED AGAIN? COMING TO TERMS WITH LIMITATIONS IN THE USE OF PERSONALITY TESTS FOR PERSONNEL SELECTION

We recently published an article in which we highlighted a number of issues associated with the use of self-report personality tests in personnel selection contexts ( Morgeson et al., 2007 ). Both Ones, Dilchert, Viswesvaran, and Judge (2007) and Tett and Christiansen (2007) have written responses to this article. In our response to these articles we address many of the issues raised by Ones et al. and Tett and Christiansen. In addition to a detailed response, we make the following 4 key points: (1) Our criticisms of personality testing apply only to the selection context, not to all research on personality; (2) the observed validities of personality tests predicting job performance criteria are low and have not changed much over time; (3) when evaluating the usefulness of using personality tests to select applicants, one must not ignore the observed, uncorrected validity; and (4) when discussing the value of personality tests for selection contexts, the most important criteria are those that reflect job performance. Implications for personality testing research and practice are discussed.     

Comparing standardized coefficients in structural equation modeling: a model reparameterization approach

We propose a two-stage method for comparing standardized coefficients in structural equation modeling (SEM). At stage 1, we transform the original model of interest into the standardized model by model reparameterization, so that the model parameters appearing in the standardized model are equivalent to the standardized parameters of the original model. At stage 2, we impose appropriate linear equality constraints on the standardized model and use a likelihood ratio test to make statistical inferences about the equality of standardized coefficients. Unlike other existing methods for comparing standardized coefficients, the proposed method does not require specific modeling features (e.g., specification of nonlinear constraints), which are available only in certain SEM software programs. Moreover, this method allows researchers to compare two or more standardized coefficients simultaneously in a standard and convenient way. Three real examples are given to illustrate the proposed method, using EQS, a popular SEM software program. Results show that the proposed method performs satisfactorily for testing the equality of standardized coefficients.     

Coherent psychometric modelling with Bayesian nonparametrics

In this paper we argue that model selection, as commonly practised in psychometrics, violates certain principles of coherence. On the other hand, we show that Bayesian nonparametrics provides a coherent basis for model selection, through the use of a ‘nonparametric’ prior distribution that has a large support on the space of sampling distributions. We illustrate model selection under the Bayesian nonparametric approach, through the analysis of real questionnaire data. Also, we present ways to use the Bayesian nonparametric framework to define very flexible psychometric models, through the specification of a nonparametric prior distribution that supports all distribution functions for the inverse link, including the standard logistic distribution functions. The Bayesian nonparametric approach provides a coherent method for model selection that can be applied to any statistical model, including psychometric models. Moreover, under a ‘non‐informative’ choice of nonparametric prior, the Bayesian nonparametric approach is easy to apply, and selects the model that maximizes the log likelihood. Thus, under this choice of prior, the approach can be extended to non‐Bayesian settings where the parameters of the competing models are estimated by likelihood maximization, and it can be used with any psychometric software package that routinely reports the model log likelihood.