Derivative free gradient projection algorithms for rotation

A simple modification substantially simplifies the use of the gradient projection (GP) rotation algorithms of Jennrich (2001, 2002). These algorithms require subroutines to compute the value and gradient of any specific rotation criterion of interest. The gradient can be difficult to derive and program. It is shown that using numerical gradients gives almost precisely the same results as using exact gradients. The resulting algorithm is very easy to use because the only problem specific code required is that needed to define the rotation criterion. The computing time is increased when using numerical gradients, but it is still very modest for most purposes. While used extensively elsewhere, numerical derivatives seem to be underutilized in statistics.     

A cluster‐based factor rotation

A new oblique factor rotation method is proposed, the aim of which is to identify a simple and well‐clustered structure in a factor loading matrix. A criterion consisting of the complexity of a factor loading matrix and a between‐cluster dissimilarity is optimized using the gradient projection algorithm and the k‐means algorithm. It is shown that if there is an oblique rotation of an initial loading matrix that has a perfect simple structure, then the proposed method with Kaiser's normalization will produce the perfect simple structure. Although many rotation methods can also recover a perfect simple structure, they perform poorly when a perfect simple structure is not possible. In this case, the new method tends to perform better because it clusters the loadings without requiring the clusters to be perfect. Artificial and real data analyses demonstrate that the proposed method can give a simple structure, which the other methods cannot produce, and provides a more interpretable result than those of widely known rotation techniques.     

Newton Algorithms for Analytic Rotation: an Implicit Function Approach

In this paper implicit function-based parameterizations for orthogonal and oblique rotation matrices are proposed. The parameterizations are used to construct Newton algorithms for minimizing differentiable rotation criteria applied to m factors and p variables. The speed of the new algorithms is compared to that of existing algorithms and to that of Newton algorithms based on alternative parameterizations. Several rotation criteria were examined and the algorithms were evaluated over a range of values for m. Initial guesses for Newton algorithms were improved by subconvergence iterations of the gradient projection algorithm. Simulation results suggest that no one algorithm is fastest for minimizing all criteria for all values of m. Among competing algorithms, the gradient projection algorithm alone was faster than the implicit function algorithm for minimizing a quartic criterion over oblique rotation matrices when m is large. In all other conditions, however, the implicit function algorithms were competitive with or faster than the fastest existing algorithms. The new algorithms showed the greatest advantage over other algorithms when minimizing a nonquartic component loss criterion.     

Independent component analysis as a rotation method: A very different solution to Thurstone's box problem

In this paper we consider the well‐known Thurstone box problem in exploratory factor analysis. Initial loadings and components are extracted using principal component analysis. Rotating the components towards independence rather than rotating the loadings towards simplicity allows one to accurately recover the dimensions of each box and also produce simple loadings. It is shown how this may be done using an appropriate rotation criterion and a general rotation algorithm. Methods from independent component analysis are used, and this paper may be viewed as an introduction to independent component analysis from the perspective of factor analysis.     

Riemannian Newton and trust-region algorithms for analytic rotation in exploratory factor analysis

In exploratory factor analysis, latent factors and factor loadings are seldom interpretable until analytic rotation is performed. Typically, the rotation problem is solved by numerically searching for an element in the manifold of orthogonal or oblique rotation matrices such that the rotated factor loadings minimize a pre-specified complexity function. The widely used gradient projection (GP) algorithm, although simple to program and able to deal with both orthogonal and oblique rotation, is found to suffer from slow convergence when the number of manifest variables and/or the number of latent factors is large. The present work examines the effectiveness of two Riemannian second-order algorithms, which respectively generalize the well-established truncated Newton and trust-region strategies for unconstrained optimization in Euclidean spaces, in solving the rotation problem. When approaching a local minimum, the second-order algorithms usually converge superlinearly or even quadratically, better than first-order algorithms that only converge linearly. It is further observed in Monte Carlo studies that, compared to the GP algorithm, the Riemannian truncated Newton and trust-region algorithms require not only much fewer iterations but also much less processing time to meet the same convergence criterion, especially in the case of oblique rotation.     

Rotation to Simple Loadings Using Component Loss Functions: The Oblique Case

Component loss functions (CLFs) similar to those used in orthogonal rotation are introduced to define criteria for oblique rotation in factor analysis. It is shown how the shape of the CLF affects the performance of the criterion it defines. For example, it is shown that monotone concave CLFs give criteria that are minimized by loadings with perfect simple structure when such loadings exist. Moreover, if the CLFs are strictly concave, minimizing must produce perfect simple structure whenever it exists. Examples show that methods defined by concave CLFs perform well much more generally. While it appears important to use a concave CLF, the specific CLF used is less important. For example, the very simple linear CLF gives a rotation method that can easily outperform the most popular oblique rotation methods promax and quartimin and is competitive with the more complex simplimax and geomin methods. The author would like to thank the editor and three reviewers for helpful suggestions and for identifying numerous errors.     

Computing measures of simplicity of fit for loadings in factor-analytically derived scales

A very simple structure is sought when factor analysis is used to develop measurement scales. The SIMLOAD program computes measures of factorial simplicity for rows and columns of loading matrices (usually the factor pattern) as well as some overall measures. These include Kaiser’s (1974) index of factorial simplicity for variables (rows), the author’s scale fit index for factors (columns), Bentler’s (1977) scale-free matrix measure, and hyperplane counts. Routine use of these measures is recommended for multifactor scale development. The measures may also be useful in more general factor applications and in confirmatory as well as exploratory analyses. SIMLOAD also computes factor scale intercorrelations, scale alpha coefficients (including alpha when an item is removed), and sorted loadings for ease of interpretation.     

Rotation for simple loadings

Existing analytic oblique rotation schemes proceed by optimizing a simplicity function applied to the reference structure. This article suggests optimizing a simplicity function applied to primary loadings directly. The feasibility of the suggestion is demonstrated using the quartimin criterion. An algorithm to implement the optimization is derived and the existence of an admissible solution proved. Practical comparisons with the biquartimin method are made using Thurstone's Box Problem and Holzinger and Swineford's Twenty-Four Psychological Tests Problem.     

Orthogonal rotation algorithms

The quartimax and varimax algorithms for orthogonal rotation attempt to maximize particular simplicity criteria by a sequence of two-factor rotations. Derivations of these algorithms have been fairly complex. A simple general theory for obtaining two factor at a time algorithms for any polynomial simplicity criteria satisfying a natural symmetry condition is presented. It is shown that the degree of any symmetric criterion must be a multiple of four. A basic fourth degree algorithm, which is applicable to all symmetric fourth degree criteria, is derived and applied using a variety of criteria. When used with the quartimax and varimax criteria the algorithm is mathematically identical to the standard algorithms for these criteria. A basic eighth degree algorithm is also obtained and applied using a variety of eighth degree criteria. In general the problem of writing a basic algorithm for all symmetric criteria of any specified degree reduces to the problem of maximizing a trigonometric polynomial of degree one-fourth that of the criteria.This research was supported by the Bell Telephone Laboratories, Murray Hill, New Jersey and NIH Grant FR-3.     

Functionplane—A new approach to simple structure rotation

A new criterion for rotation to an oblique simple structure is proposed. The results obtained are similar to that obtained by Cattell and Muerle's maxplane criterion. Since the proposed criterion is smooth it is possible to locate the local maxima using simple gradient techniques. The results of the application of the Functionplane criterion to three sets of data are given. In each case a better fit to the subjective solution was obtained using the functionplane criterion than was reported for by Hakstian for the oblimax, promax, maxplane, or the Harris-Kaiser methods.This paper is contribution No. 66 from the Program in Ecology and Evolution at the State University of New York, Stony Brook, New York. This work was supported in part by a grant (GB-20496) from the National Science Foundation. The computations were performed on an IBM 360/67 computer at the State University of New York at Stony Brook.     

Three-mode orthomax rotation

Factor analysis and principal components analysis (PCA) are often followed by an orthomax rotation to rotate a loading matrix to simple structure. The simple structure is usually defined in terms of the simplicity of the columns of the loading matrix. In Three-mode PCA, rotational freedom of the so called core (a three-way array relating components for the three different modes) can be used similarly to find a simple structure of the core. Simple structure of the core can be defined with respect to all three modes simultaneously, possibly with different emphases on the different modes. The present paper provides a fully flexible approach for orthomax rotation of the core to simple structure with respect to three modes simultaneously. Computationally, this approach relies on repeated (two-way) orthomax applied to supermatrices containing the frontal, lateral or horizontal slabs, respectively. The procedure is illustrated by means of a number of exemplary analyses. As a by-product, application of the Three-mode Orthomax procedures to two-way arrays is shown to reveal interesting relations with and interpretations of existing two-way simple structure rotation techniques.This research has been made possible by a fellowship from the Royal Netherlands Academy of Arts and Sciences to the author. The author is obliged to Jos ten Berge and two anonymous reviewers for useful comments on an earlier version of this paper.     

An algorithm for extracting maximum cardinality subsets with perfect dominance or anti‐Robinson structures

A common criterion for seriation of asymmetric matrices is the maximization of the dominance index, which sums the elements above the main diagonal of a reordered matrix. Similarly, a popular seriation criterion for symmetric matrices is the maximization of an anti‐Robinson gradient index, which is associated with the patterning of elements in the rows and columns of a reordered matrix. Although perfect dominance and perfect anti‐Robinson structure are rarely achievable for empirical matrices, we can often identify a sizable subset of objects for which a perfect structure is realized. We present and demonstrate an algorithm for obtaining a maximum cardinality (i.e. the largest number of objects) subset of objects such that the seriation of the proximity matrix corresponding to the subset will have perfect structure. MATLAB implementations of the algorithm are available for dominance, anti‐Robinson and strongly anti‐Robinson structures.     

Orthogonal rotations to maximal agreement for two or more matrices of different column orders

Methods for orthogonal Procrustes rotation and orthogonal rotation to a maximal sum of inner products are examined for the case when the matrices involved have different numbers of columns. An inner product solution offered by Cliff is generalized to the case of more than two matrices. A nonrandom start for a Procrustes solution suggested by Green and Gower is shown to give better results than a random start. The Green-Gower Procrustes solution (with nonrandom start) is generalized to the case of more than two matrices. Simulation studies indicate that both the generalized inner product solution and the generalized Procrustes solution tend to attain their global optima within acceptable computation times. A simple procedure is offered for approximating simple structure for the rotated matrices without affecting either the Procrustes or the inner product criterion.The authors are obliged to Charles Lewis for helpful comments on a previous draft of this paper and to Frank Brokken for preparing a computer program that was used in this study.     

A monotonically convergent algorithm for orthogonal congruence rotation

Brokken has proposed a method for orthogonal rotation of one matrix such that its columns have a maximal sum of congruences with the columns of a target matrix. This method employs an algorithm for which convergence from every starting point is not guaranteed. In the present paper, an iterative majorization algorithm is proposed which is guaranteed to converge from every starting point. Specifically, it is proven that the function value converges monotonically, and that the difference between subsequent iterates converges to zero. In addition to the better convergence properties, another advantage of the present algorithm over Brokken's one is that it is easier to program. The algorithms are compared on 80 simulated data sets, and it turned out that the new algorithm performed well in all cases, whereas Brokken's algorithm failed in almost half the cases. The derivation of the algorithm is given in full detail because it involves a series of inequalities that can be of use to derive similar algorithms in different contexts.This research has been made possible by a fellowship from the Royal Netherlands Academy of Arts and Sciences to the first author. The authors are obliged to Willem J. Heiser and Jos M. F. ten Berge for useful comments on an earlier version of this paper.     

The effect of the observer vantage point on perceived distortions in linear perspective images

Some features of linear perspective images may look distorted. Such distortions appear in two drawings by Jan Vredeman de Vries involving perceived elliptical, instead of circular, pillars and tilted, instead of upright, columns. Distortions may be due to factors intrinsic to the images, such as violations of the so-called Perkins's laws, or factors extrinsic to them, such as observing the images from positions different from their center of projection. When the correct projection centers for the two drawings were reconstructed, it was found that they were very close to the images and, therefore, practically unattainable in normal observation. In two experiments, enlarged versions of images were used as stimuli, making the positions of the projection centers attainable for observers. When observed from the correct positions, the perceived distortions disappeared or were greatly diminished. Distortions perceived from other positions were smaller than would be predicted by geometrical analyses, possibly due to flatness cues in the images. The results are relevant for the practical purposes of creating faithful impressions of 3-D spaces using 2-D images.     

Suppressing permutations or rigid planar rotations: A remedy against nonoptimal varimax rotations

Varimax rotation consists of iteratively rotating pairs of columns of a matrix to a maximal sum (over columns) of variances of squared elements of the matrix. Without loss of optimality, the two rotated columns can be permuted and/or reflected. Although permutations and reflections are harmless for each planar rotation per se, they can be harmful in Varimax rotation. Specifically, they often give rise to the phenomenon that certain pairs of columns are consistently skipped in the iterative process, whence Varimax will be terminated at a nonstationary point. The skipping phenomenon is demonstrated, and it is shown how to prevent it.The author is obliged to Henk Kiers for commenting on a previous draft.     

Simplicity versus likelihood in visual perception: from surprisals to precisals

The likelihood principle states that the visual system prefers the most likely interpretation of a stimulus, whereas the simplicity principle states that it prefers the most simple interpretation. This study investigates how close these seemingly very different principles are by combining findings from classical, algorithmic, and structural information theory. It is argued that, in visual perception, the two principles are perhaps very different with respect to the viewpoint-independent aspects of perception but probably very close with respect to the viewpoint-dependent aspects which, moreover, seem decisive in everyday perception. This implies that either principle may have guided the evolution of visual systems and that the simplicity paradigm may provide perception models with the necessary quantitative specifications of the often plausible but also intuitive ideas provided by the likelihood paradigm.     

Factor analysis; some effects of chance error

Ignorance concerning the standard error of individual factor loadings and their differences has been a major obstacle to the proper interpretation of factorial results. The effects of three types of experimental error (selection of variables, selection of subjects and selection of scores) are here reported. In order to handle the errors of rotation systematically, it has been necessary to introduce a new semi-analytical criterion for the attainment of simple structure. Variability in results which may theoretically be eliminated is discussed under the heading of non-essential error.