Determining the number of factors to retain in an exploratory factor analysis using comparison data of known factorial structure

Exploratory factor analysis (EFA) is used routinely in the development and validation of assessment instruments. One of the most significant challenges when one is performing EFA is determining how many factors to retain. Parallel analysis (PA) is an effective stopping rule that compares the eigenvalues of randomly generated data with those for the actual data. PA takes into account sampling error, and at present it is widely considered the best available method. We introduce a variant of PA that goes even further by reproducing the observed correlation matrix rather than generating random data. Comparison data (CD) with known factorial structure are first generated using 1 factor, and then the number of factors is increased until the reproduction of the observed eigenvalues fails to improve significantly. We evaluated the performance of PA, CD with known factorial structure, and 7 other techniques in a simulation study spanning a wide range of challenging data conditions. In terms of accuracy and robustness across data conditions, the CD technique outperformed all other methods, including a nontrivial superiority to PA. We provide program code to implement the CD technique, which requires no more specialized knowledge or skills than performing PA.