A ROTATIONAL PROCEDURE FOR SEPARATION OF TRAIT,METHOD AND INTERACTION FACTORS IN MULTTTRAIT-MULTIMETHOD MATRICES

Cluster recovery indices are more important than ever, because of the necessity for comparing the large number of clustering procedures available today. Of the cluster recovery indices prominent in contemporary literature, the Hubert and Arabie (1985) adjustment to the Rand index (1971) has been demonstrated to have the most desirable properties (Milligan &; Cooper, 1986). However, use of the Hubert and Arabie adjustment to the Rand index is limited to cluster solutions involving non-overlapping, or disjoint, clusters. The present paper introduces a generalization of the Hubert and Arabie adjusted Rand index. This generalization, called the Omega index, can be applied to situations where both, one, or neither of the solutions being compared is non-disjoint. In the special case where both solutions are disjoint, the Omega index is equivalent to the Hubert and Arabie adjusted Rand index.     

Profiling local optima in K-means clustering: developing a diagnostic technique

Using the cluster generation procedure proposed by D. Steinley and R. Henson (2005), the author investigated the performance of K-means clustering under the following scenarios: (a) different probabilities of cluster overlap; (b) different types of cluster overlap; (c) varying samples sizes, clusters, and dimensions; (d) different multivariate distributions of clusters; and (e) various multidimensional data structures. The results are evaluated in terms of the Hubert-Arabie adjusted Rand index, and several observations concerning the performance of K-means clustering are made. Finally, the article concludes with the proposal of a diagnostic technique indicating when the partitioning given by a K-means cluster analysis can be trusted. By combining the information from several observable characteristics of the data (number of clusters, number of variables, sample size, etc.) with the prevalence of unique local optima in several thousand implementations of the K-means algorithm, the author provides a method capable of guiding key data-analysis decisions.     

A variable-selection heuristic for K-means clustering

One of the most vexing problems in cluster analysis is the selection and/or weighting of variables in order to include those that truly define cluster structure, while eliminating those that might mask such structure. This paper presents a variable-selection heuristic for nonhierarchical (K-means) cluster analysis based on the adjusted Rand index for measuring cluster recovery. The heuristic was subjected to Monte Carlo testing across more than 2200 datasets with known cluster structure. The results indicate the heuristic is extremely effective at eliminating masking variables. A cluster analysis of real-world financial services data revealed that using the variable-selection heuristic prior to the K-means algorithm resulted in greater cluster stability.     

A close-up comparison of the misclassification error distance and the adjusted Rand index for external clustering evaluation

The misclassification error distance and the adjusted Rand index are two of the most common criteria used to evaluate the performance of clustering algorithms. This paper provides an in-depth comparison of the two criteria, with the aim of better understand exactly what they measure, their properties and their differences. Starting from their population origins, the investigation includes many data analysis examples and the study of particular cases in great detail. An exhaustive simulation study provides insight into the criteria distributions and reveals some previous misconceptions.     

The adjusted rand statistic: A SAS macro

A macro for calculating the Hubert and Arabie (1985) adjusted Rand statistic is presented. The adjusted Rand statistic gives a measure of classification agreement between two partitions of the same set of objects. The macro is written in the SAS macro language and makes extensive use of SAS/IML software (SAS Institute, 1985a, 1985b). The macro uses two different methods of handling missing values. The default method assumes that each object that has a missing value for the classification category is in its own separate category or cluster for that classification. The optional method places all objects with a missing value for the classification category into the same category for that classification.This study was supported in part by Individual National Research Service Award F32 DA 05283 from the National Institute on Drug Abuse.Requests for the Macro code can be sent via BITNET: CUSGPXH @ UCLAMVS. A copy of the macrocode can also be obtained by sending a stamped self-addressed mailer and a PC-DOS formatted floppy diskette to Paul Hoffman, 5628 MSA, UCLA, Los Angeles, CA 90024-1557.     

A note on the expected value of the Rand index

Two expectations of the adjusted Rand index (ARI) are compared. It is shown that the expectation derived by Morey and Agresti (1984, Educational and Psychological Measurement, 44, 33) under the multinomial distribution to approximate the exact expectation from the hypergeometric distribution (Hubert & Arabie, 1985, Journal of Classification, 2, 193) provides a poor approximation, and, in some cases, the difference between the two expectations can increase with the sample size. Proofs concerning the minimum and maximum difference between the two expectations are provided, and it is shown through simulation that the ARI can differ significantly depending on which expectation is used. Furthermore, when compared in a hypothesis testing framework, multinomial approximation overly favours the null hypothesis.     

ESCOLEX: A grade-level lexical database from European Portuguese elementary to middle school textbooks

In this article, we introduce ESCOLEX, the first European Portuguese children’s lexical database with grade-level-adjusted word frequency statistics. Computed from a 3.2-million-word corpus, ESCOLEX provides 48,381 word forms extracted from 171 elementary and middle school textbooks for 6- to 11-year-old children attending the first six grades in the Portuguese educational system. Like other children’s grade-level databases (e.g., Carroll, Davies, & Richman, 1971; Corral, Ferrero, & Goikoetxea, Behavior Research Methods, 41, 1009–1017, 2009; Lété, Sprenger-Charolles, & Colé, Behavior Research Methods, Instruments, & Computers, 36, 156–166, 2004; Zeno, Ivens, Millard, Duvvuri, 1995), ESCOLEX provides four frequency indices for each grade: overall word frequency (F), index of dispersion across the selected textbooks (D), estimated frequency per million words (U), and standard frequency index (SFI). It also provides a new measure, contextual diversity (CD). In addition, the number of letters in the word and its part(s) of speech, number of syllables, syllable structure, and adult frequencies taken from P-PAL (a European Portuguese corpus-based lexical database; Soares, Comesaña, Iriarte, Almeida, Simões, Costa, …, Machado, 2010; Soares, Iriarte, Almeida, Simões, Costa, França, …, Comesaña, in press) are provided. ESCOLEX will be a useful tool both for researchers interested in language processing and development and for professionals in need of verbal materials adjusted to children’s developmental stages. ESCOLEX can be downloaded along with this article or from http://p-pal.di.uminho.pt/about/databases.     

Answer Similarity Analysis at the Group Level

Answer similarity indices were developed to detect pairs of test takers who may have worked together on an exam or instances in which one test taker copied from another. For any pair of test takers, an answer similarity index can be used to estimate the probability that the pair would exhibit the observed response similarity or a greater degree of similarity under the assumption that the test takers worked independently. To identify groups of test takers with unusually similar response patterns, Wollack and Maynes suggested conducting cluster analysis using probabilities obtained from an answer similarity index as measures of distance. However, interpretation of results at the cluster level can be challenging because the method is sensitive to the choice of clustering procedure and only enables probabilistic statements about pairwise relationships. This article addresses these challenges by presenting a statistical test that can be applied to clusters of examinees rather than pairs. The method is illustrated with both simulated and real data.     

Out-of-bag Prediction Error: A Cross Validation Index for Generalized Structured Component Analysis

Cross validation is a useful way of comparing predictive generalizability of theoretically plausible a priori models in structural equation modeling (SEM). A number of overall or local cross validation indices have been proposed for existing factor-based and component-based approaches to SEM, including covariance structure analysis and partial least squares path modeling. However, there is no such cross validation index available for generalized structured component analysis (GSCA) which is another component-based approach. We thus propose a cross validation index for GSCA, called Out-of-bag Prediction Error (OPE), which estimates the expected prediction error of a model over replications of so-called in-bag and out-of-bag samples constructed through the implementation of the bootstrap method. The calculation of this index is well-suited to the estimation procedure of GSCA, which uses the bootstrap method to obtain the standard errors or confidence intervals of parameter estimates. We empirically evaluate the performance of the proposed index through the analyses of both simulated and real data.     

What Do Implicit Measures Tell Us? Scrutinizing the Validity of Three Common Assumptions

ABSTRACT— Experimental paradigms designed to assess "implicit" representations are currently very popular in many areas of psychology. The present article addresses the validity of three widespread assumptions in research using these paradigms: that (a) implicit measures reflect unconscious or introspectively inaccessible representations; (b) the major difference between implicit measures and self-reports is that implicit measures are resistant or less susceptible to social desirability; and (c) implicit measures reflect highly stable, older representations that have their roots in long-term socialization experiences. Drawing on a review of the available evidence, we conclude that the validity of all three assumptions is equivocal and that theoretical interpretations should be adjusted accordingly. We discuss an alternative conceptualization that distinguishes between activation and validation processes.     

The search for subtypes of DCD: is cluster analysis the answer?

Studies using cluster analysis as a method to identify distinct subtypes of developmental coordination disorder (DCD) have been inconclusive leading some authors to conclude that the method of cluster analysis should be abandoned while others call for the validation of previously defined subtypes. The objective of the current study was to examine the use of cluster analysis as a method of searching for subtypes of DCD to gain a better understanding of how different samples and different measures influence the interpretation of results. The paper provides a detailed review of three commonly cited cluster analytical studies and then explores the possible reasons for the discrepant results by replicating the approach with a different clinical sample. The results highlight the impact of different measures on cluster structure and the importance of adoption of a common standard to facilitate interpretation across studies.     

不同方向视觉运动追踪的特性

对向左、向右、向上与向下4个方向的平滑运动视觉追踪的眼动特点进行了探讨,并采用了频谱分析的方法对4个方向上视觉追踪的眼动参数进行了分析。结果表明,(1)水平追踪与垂直追踪之间的差异较为普遍,几乎存在于所有的眼动参数上;(2)左右追踪之间、上下追踪之间也分别都存在差异,它们主要表现在数据分布结构上;(3)眼睛跳动距离是视觉追踪的敏感指标。另外,不同方向的差异在不同眼动参数之间并不具有一致性。这反映了视觉追踪眼动的复杂性,不同类型眼动之间存在相互关联,这种关联性还有待于进一步研究。     

A cautionary note on using internal cross validation to select the number of clusters

A highly popular method for examining the stability of a data clustering is to split the data into two parts, cluster the observations in Part A, assign the objects in Part B to their nearest centroid in Part A, and then independently cluster the Part B objects. One then examines how close the two partitions are (say, by the Rand measure). Another proposal is to split the data into k parts, and see how their centroids cluster. By means of synthetic data analyses, we demonstrate that these approaches fail to identify the appropriate number of clusters, particularly as sample size becomes large and the variables exhibit higher correlations.The authors express their thanks to the Sol C. Snider Entrepreneurial Center, Wharton School, for support of this project.     

Improving construct validity: Cronbach, Meehl, and Neurath's ship

Smith's article "On Construct Validity: Issues of Method and Measurement" is a fine tribute to L. J. Cronbach and P. E. Meehl (1955) that clarifies the current state and future directions in the understanding of construct validity. Construct validity is a dynamic process, and fit indices need to be used at the service of understanding, not in place of it. The failure of a study or set of studies to support a construct, a measure, or the theory underlying it admits of many explanations, and the ways scientists interpret such failures are prone to cognitive biases and motivated reasoning. This suggests why metrics designed to index the extent to which observations match expectations can be useful prostheses to scientific judgments. As P. E. Meehl (1954) showed decades ago, quantitative, statistical formulas and indices tend to outperform informal, qualitative judgments, and this applies as much to the way researchers evaluate constructs and measures as to judgments in the consulting room.     

Intra-individual Response Variability as an Indicator of Insufficient Effort Responding: Comparison to Other Indicators and Relationships with Individual Differences

Surveys are one of the most popular ways to collect employee information. Because of their widespread use, data quality is an increasingly important concern. The purpose of this paper is to (1) introduce the intra-individual response variability (IRV) index as an easily calculated and flexible way to detect insufficient effort responding (IER); (2) examine the extent to which various IER indices detect the same or different respondents engaging in IER behavior; and (3) investigate relationships between individual differences and commonly used IER indices to better understand systematic and theoretically relevant IER behavior. In a two-part study, 199 undergraduates responded to questionnaires online, and various IER indices were calculated. The IRV index identifies different respondents than other IER indices. Values on the IRV index (as well as other IER indices) are related to scores on theoretically meaningful individual differences in conscientiousness, agreeableness, and boredom proneness. This study provides researchers with a robust, easily calculated, and flexible means for screening questionnaire data for IER behavior. Practical recommendations for finding and making decisions about IER behavior patterns are provided. This study introduces the IRV index, an extension of the long string, used to identify survey research participants who likely engaged in one type of IER behavior. It is also one of the first studies to evaluate the extent to which IER indices identify different respondents as having engaged in IER and provides additional evidence that values on these indices are related to individual differences.     

Couples in marital distress: a study of personality factors as measured by the MMPI-2

This study examines the relationship between Minnesota Multiphasic Personality Inventory-2 (MMPI-2) measured personality characteristics and marital distress and provides empirical validation for using the MMPI-2 with a marital therapy population. Studied were 150 couples in marital therapy and 841 normal couples who participated in the MMPI-2 restandardization study. The MMPI-2, a biographical form, a partner rating form, and the Dyadic Adjustment Scale (DAS) were administered to all couples. The marital counseling group resembled previous marital counseling samples studied with the MMPI and scored significantly higher than the normative sample on several MMPI-2 scales. Relationships between the DAS and MMPI-2 clinical and content scale scores are reported. The Psychopathic Deviate (Pd) clinical scale and Family Problems (FAM) content scale were the most powerful group discriminators and strongest correlates of the DAS; their use as indices of marital distress is tested. The meaning of Pd as an index in assessing personality factors in marital distress is explored.     

Developmental Changes in Content of Movement Responses

This study examines the relationship between Minnesota Multiphasic Personality Inventory-2 (MMPI-2) measured personality characteristics and marital distress and provides empirical validation for using the MMPI-2 with a marital therapy population. Studied were 150 couples in marital therapy and 841 normal couples who participated in the MMPI-2 restandardization study. The MMPI-2, a biographical form, a partner rating form, and the Dyadic Adjustment Scale (DAS) were administered to all couples. The marital counseling group resembled previous marital counseling samples studied with the MMPI and scored significantly higher than the normative sample on several MMPI-2 scales. Relationships between the DAS and MMPI-2 clinical and content scale scores are reported. The Psychopathic Deviate (Pd) clinical scale and Family Problems (FAM) content scale were the most powerful group discriminators and strongest correlates of the DAS; their use as indices of marital distress is tested. The meaning of Pd as an index in assessing personality factors in marital distress is explored.     

International Guidelines on Computer-Based and Internet-Delivered Testing

In his (1988) article on validity and validation, Messick discussed how a proliferation of technology-mediated delivery methods in the first few decades of the 21st century will impact on our conceptions of teaching and learning, and on validity and validation practices. According to Messick (1988), our fundamental conception of validity, as expressed in his four-faceted framework, will likely remain the same, but new technology-mediated environments will render a classical, unitary approach to validation untenable. With the aid of an example, this article demonstrates how Messick's framework provides a powerful set of lenses with which to explore issues in the validation of small-scale assessments in these new environments. A goal of this article is to initiate a dialogue between measurement and testing specialists and experts in distance and distributed learning.     

External Validation of Bifactor Model of ADHD: Explaining Heterogeneity in Psychiatric Comorbidity,Cognitive Control,and Personality Trait Profiles Within DSM-IV ADHD

The current paper provides external validation of the bifactor model of ADHD by examining associations between ADHD latent factor/profile scores and external validation indices. 548 children (321 boys; 302 with ADHD), 6 to 18 years old, recruited from the community participated in a comprehensive diagnostic procedure. Mothers completed the Child Behavior Checklist, Early Adolescent Temperament Questionnaire, and California Q-Sort. Children completed the Stop and Trail-Making Task. Specific inattention was associated with depression/withdrawal, slower cognitive task performance, introversion, agreeableness, and high reactive control; specific hyperactivity-impulsivity was associated with rule-breaking/aggressive behavior, social problems, errors during set-shifting, extraversion, disagreeableness, and low reactive control. It is concluded that the bifactor model provides better explanation of heterogeneity within ADHD than DSM-IV ADHD symptom counts or subtypes.     

The Rand school of social science during the progressive era: Will to power of a stratum of the American intellectual class

Boldly asserting the existence of an intellectual class, this article details the efforts of one stratum of that class to rise to political power through the creation and development of the Rand School of Social Science in New York. The author argues that the founders of the Rand School used the social sciences—disciplines which they were themselves shaping and popularizing—to promote their political agenda. The school's founders trained an intelligentsia from the working class in the outlook and methods of the social sciences as part of their efforts to redirect the nation's political agenda toward socialism. Finding the social sciences a politically contested terrain, the author offers a history of the founding and administration of the Rand School, describes the pedagogical role of men and women in it, and details the political repression which the school endured as its influence grew. A number of notable intellectuals were associated with the school, among them Franklin H. Giddings, Charlotte Perkins Gilman, Morris Hillquit, Algernon Lee, and Scott Nearing. Previously unpublished information regarding the renowned historians Charles and Mary Beard's involvement with the American Socialist Society and the Rand School is of particular interest.