A case of extreme simplicity of the core matrix in three-mode principal components analysis

In three-mode Principal Components Analysis, theP ×Q ×R core matrixG can be transformed to simple structure before it is interpreted. It is well-known that, whenP=QR,G can be transformed to the identity matrix, which implies that all elements become equal to values specified a priori. In the present paper it is shown that, whenP=QR − 1,G can be transformed to have nearly all elements equal to values spectified a priori. A cllsed-form solution for this transformation is offered. Theoretical and practical implications of this simple structure transformation ofG are discussed. Constructive comments from anonymous reviewers are gratefully acknowledged.     

Newton Algorithms for Analytic Rotation: an Implicit Function Approach

In this paper implicit function-based parameterizations for orthogonal and oblique rotation matrices are proposed. The parameterizations are used to construct Newton algorithms for minimizing differentiable rotation criteria applied to m factors and p variables. The speed of the new algorithms is compared to that of existing algorithms and to that of Newton algorithms based on alternative parameterizations. Several rotation criteria were examined and the algorithms were evaluated over a range of values for m. Initial guesses for Newton algorithms were improved by subconvergence iterations of the gradient projection algorithm. Simulation results suggest that no one algorithm is fastest for minimizing all criteria for all values of m. Among competing algorithms, the gradient projection algorithm alone was faster than the implicit function algorithm for minimizing a quartic criterion over oblique rotation matrices when m is large. In all other conditions, however, the implicit function algorithms were competitive with or faster than the fastest existing algorithms. The new algorithms showed the greatest advantage over other algorithms when minimizing a nonquartic component loss criterion.     

Maximization of sums of quotients of quadratic forms and some generalizations

Monotonically convergent algorithms are described for maximizing six (constrained) functions of vectors x, or matricesX with columns x₁, ..., x _r . These functions are h₁(x)= _k (xA _kx)(xC _kx)^–1, H₁(X)= _k tr (XA _k X)(XC _k X)^–1, h₁(X)= _{k l} (x _l A _kx _l ) (x _l C _kx _l )^–1 withX constrained to be columnwise orthonormal, h₂(x)= _k (xA _kx)²(xC _kx)^–1 subject to xx=1, H₂(X)= _k tr(XA _kX)(XA_kX)(XC_kX)^–1 subject toXX=I, and h₂(X)= _{k l} (x _l A _kx _l )² (x _l C _kX _l )^–1 subject toXX=I. In these functions the matricesC _k are assumed to be positive definite. The matricesA _k can be arbitrary square matrices. The general formulation of the functions and the algorithms allows for application of the algorithms in various problems that arise in multivariate analysis. Several applications of the general algorithms are given. Specifically, algorithms are given for reciprocal principal components analysis, binormamin rotation, generalized discriminant analysis, variants of generalized principal components analysis, simple structure rotation for one of the latter variants, and set component analysis. For most of these methods the algorithms appear to be new, for the others the existing algorithms turn out to be special cases of the newly derived general algorithms.This research has been made possible by a fellowship from the Royal Netherlands Academy of Arts and Sciences to the author. The author is obliged to Jos ten Berge for stimulating this research and for helpful comments on an earlier version of this paper.     

Beyond principal component analysis: A trilinear decomposition model and least squares estimation

The paper derives sufficient conditions for the consistency and asymptotic normality of the least squares estimator of a trilinear decomposition model for multiway data analysis.     

A factor simplicity index

We propose an index for assessing the degree of factor simplicity in the context of principal components and exploratory factor analysis. The new index, which is called Loading Simplicity, is based on the idea that the communality of each variable should be related to few components, or factors, so that the loadings in each variable are either zero or as far from zero as possible. This index does not depend on the scale of the factors, and its maximum and minimum are only related to the degree of simplicity in the loading matrix. The aim of the index is to enable the degree of simplicity in loading matrices to be compared.The author would like to thank the review team for their insights and recommendations. This work was supported by a grant SEC2001-3821-C05-C02 from the Spanish Ministry of Science and Technology.     

A note on identification in the oblique and orthogonal factor analysis models

Conditions for removing the indeterminancy due to rotation are given for both the oblique and orthogonal factor analysis models. The conditions indicate why published counterexamples to conditions discussed by Jöreskog are not identifiable.The author would like to thank Gordon Bechtel and the reviewers for their comments and suggestions.     

Two-matrix orthogonal rotation procedures

Examples are presented in which it is either desirable or necessary to transform two sets of orthogonal axes to simple structure positions by means of the same transformation matrix. A solution is then outlined which represents a two-matrix extension of the general orthomax orthogonal rotation criterion. In certain circumstances, oblique two-matrix solutions are possible using the procedure outlined and the Harris-Kaiser [1964] logic. Finally, an illustrative example is presented in which the preceding technique is applied in the context of an inter-battery factor analysis.The work reported herein was supported by Grant S72-1886 from the Canada Council. The author acknowledges the helpful contributions of Nancy Reid and Lawrence Ward to parts of this paper.     

On rotating to smooth functions

Tucker has outlined an application of principal components analysis to a set of learning curves, for the purpose of identifying meaningful dimensions of individual differences in learning tasks. Since the principal components are defined in terms of a statistical criterion (maximum variance accounted for) rather than a substantive one, it is typically desirable to rotate the components to a more interpretable orientation. Simple structure is not a particularly appealing consideration for such a rotation; it is more reasonable to believe that any meaningful factor should form a (locally) smooth curve when the component loadings are plotted against trial number. Accordingly, this paper develops a procedure for transforming an arbitrary set of component reference curves to a new set which are mutually orthogonal and, subject to orthogonality, are as smooth as possible in a well defined (least squares) sense. Potential applications to learning data, electrophysiological responses, and growth data are indicated.Portions of this research were supported by the National Research Council of Canada, Grant A8615 to the second author. We thank Jagdeth Sheth for supplying his raw data.     

A note on the identification of restricted factor loading matrices

It is shown that problems of rotational equivalence of restricted factor loading matrices in orthogonal factor analysis are equivalent to problems of identification in simultaneous equations systems with covariance restrictions. A necessary (under a regularity assumption) and sufficient condition for local uniqueness is given and a counterexample is provided to a theorem by J. Algina concerning necessary and sufficient conditions for global uniqueness.     

Retrieving the correlation matrix from a truncated PCA solution: The inverse principal component problem

Whenr Principal Components are available fork variables, the correlation matrix is approximated in the least squares sense by the loading matrix times its transpose. The approximation is generally not perfect unlessr =k. In the present paper it is shown that, whenr is at or above the Ledermann bound,r principal components are enough to perfectly reconstruct the correlation matrix, albeit in a way more involved than taking the loading matrix times its transpose. In certain cases just below the Ledermann bound, recovery of the correlation matrix is still possible when the set of all eigenvalues of the correlation matrix is available as additional information.     

A model-based approach to multivariate principal component regression: Selecting principal components and estimating standard errors for unstandardized regression coefficients

Principal component regression (PCR) is a popular technique in data analysis and machine learning. However, the technique has two limitations. First, the principal components (PCs) with the largest variances may not be relevant to the outcome variables. Second, the lack of standard error estimates for the unstandardized regression coefficients makes it hard to interpret the results. To address these two limitations, we propose a model-based approach that includes two mean and covariance structure models defined for multivariate PCR. By estimating the defined models, we can obtain inferential information that will allow us to test the explanatory power of individual PCs and compute the standard error estimates for the unstandardized regression coefficients. A real example is used to illustrate our approach, and simulation studies under normality and nonnormality conditions are presented to validate the standard error estimates for the unstandardized regression coefficients. Finally, future research topics are discussed.     

Orthogonal procrustes rotation for two or more matrices

Necessary and sufficient conditions for rotating matrices to maximal agreement in the least-squares sense are discussed. A theorem by Fischer and Roppert, which solves the case of two matrices, is given a more straightforward proof. A sufficient condition for a best least-squares fit for more than two matrices is formulated and shown to be not necessary. In addition, necessary conditions suggested by Kristof and Wingersky are shown to be not sufficient. A rotation procedure that is an alternative to the one by Kristof and Wingersky is presented. Upper bounds are derived for determining the extent to which the procedure falls short of attaining the best least-squares fit. The problem of scaling matrices to maximal agreement is discussed. Modifications of Gower's method of generalized Procrustes analysis are suggested.     

Orthomax rotation and perfect simple structure

A loading matrix has perfect simple structure if each row has at most one nonzero element. It is shown that if there is an orthogonal rotation of an initial loading matrix that has perfect simple structure, then orthomax rotation with 0 1 of the initial loading matrix will produce the perfect simple structure. In particular, varimax and quartimax will produce rotations with perfect simple structure whenever they exist.     

Kernel principal component analysis combining rotation forest method for linearly inseparable data

Rotation forest (RoF) is an ensemble classifier combining linear analysis theories and decision tree algorithms. In recent existing works, RoF was widely applied to various fields with outstanding performance compared to traditional machine learning techniques, given that a reasonable number of base classifiers is provided. However, the conventional RoF algorithm suffers from classifying linearly inseparable datasets. In this study, a hybrid algorithm integrating kernel principal component analysis (KPCA) and the conventional RoF algorithm is proposed to overcome the classification difficulty for linearly inseparable datasets. The radial basis function (RBF) is selected as the kernel for the KPCA method to establish the nonlinear mapping for linearly inseparable data. Moreover, we evaluate various kernel parameters for better performance. Experimental results show that our algorithm improves the performance of RoF with linearly inseparable datasets, and therefore provides higher classification accuracy rates compared with other ensemble machine learning methods.     

Three-mode orthomax rotation

Factor analysis and principal components analysis (PCA) are often followed by an orthomax rotation to rotate a loading matrix to simple structure. The simple structure is usually defined in terms of the simplicity of the columns of the loading matrix. In Three-mode PCA, rotational freedom of the so called core (a three-way array relating components for the three different modes) can be used similarly to find a simple structure of the core. Simple structure of the core can be defined with respect to all three modes simultaneously, possibly with different emphases on the different modes. The present paper provides a fully flexible approach for orthomax rotation of the core to simple structure with respect to three modes simultaneously. Computationally, this approach relies on repeated (two-way) orthomax applied to supermatrices containing the frontal, lateral or horizontal slabs, respectively. The procedure is illustrated by means of a number of exemplary analyses. As a by-product, application of the Three-mode Orthomax procedures to two-way arrays is shown to reveal interesting relations with and interpretations of existing two-way simple structure rotation techniques.This research has been made possible by a fellowship from the Royal Netherlands Academy of Arts and Sciences to the author. The author is obliged to Jos ten Berge and two anonymous reviewers for useful comments on an earlier version of this paper.     

The optimality of the centroid method

The aim of this note is to show that the centroid method has two optimality properties. It yields loadings with the highest sum of absolute values, even in absence of the constraint that the squared component weights be equal. In addition, it yields scores with maximum variance, subject to the constraint that none of the squared component weights be larger than 1.This research is financed by NSERC of Canada. The author is grateful to Michel Tenenhaus for pointing the similarity of the procedures in the centroid method and Q-mode PCA in L₁. The author also thanks the editor and associate editor for providing shorter proofs of the theorems, along with the referees for their helpful comments.     

Rotational equivalence of factor loading matrices with specified values

Under mild assumptions, when appropriate elements of a factor loading matrix are specified to be zero, all orthogonally equivalent matrices differ at most by column sign changes. Here a variety of results are given for the more complex case when the specified values are not necessarily zero. A method is given for constructing reflections to preserve specified rows and columns. When the appropriatek(k – 1)/2 elements have been specified, sufficient conditions are stated for the existence of 2 ^k orthogonally equivalent matrices.This research was supported in part by the National Institute of Health Grant RR-3.     

Quartic rotation criteria and algorithms

Most of the currently used analytic rotation criteria for simple structure in factor analysis are summarized and identified as members of a general symmetric family of quartic criteria. A unified development of algorithms for orthogonal and direct oblique rotation using arbitrary criteria from this family is given. These algorithms represent fairly straightforward extensions of present methodology, and appear to be the best methods currently available.The research done by R. I. Jennrich was supported by NSF Grant MCS-8301587.