Consistent Differential Discrimination Model Estimation

A novel factor-analytic model—the differential discrimination model—for assessing individual differences in scale use has been recently introduced, together with a three-stage estimation approach for model fitting. Unfortunately, the second-stage estimator and, as a consequence, the third-stage estimator of this procedure are not consistent. In this article we show that (a) the differential discrimination model can be expressed in a structural equation model framework, and (b) consistent and simultaneous estimation of all model parameters can be achieved using standard SEM software.     

Recent Trends in the Nordic Doctrine

A review is provided of methods that estimate the magnitude of effects within experimental designs. Intraclass correlation type procedures are appropriate with any ANOVA model. Friedman's r _m and Cohen's power analysis apply to many non-ANOVA type situations, but only to the Fixed Effects ANOVA design. The Gaito utility procedure has two advantages over other intraclass correlation type measures: the coefficients sum to unity (or to 100%), and coefficients for error components are obtained.     

MCMC estimation and some model-fit analysis of multidimensional IRT models

A Bayesian procedure to estimate the three-parameter normal ogive model and a generalization of the procedure to a model with multidimensional ability parameters are presented. The procedure is a generalization of a procedure by Albert (1992) for estimating the two-parameter normal ogive model. The procedure supports analyzing data from multiple populations and incomplete designs. It is shown that restrictions can be imposed on the factor matrix for testing specific hypotheses about the ability structure. The technique is illustrated using simulated and real data. The authors would like to thank Norman Verhelst for his valuable comments and ACT, CITO group and SweSAT for the use of their data.     

Intraclass correlations: uses in assessing rater reliability

Reliability coefficients often take the form of intraclass correlation coefficients. In this article, guidelines are given for choosing among six different forms of the intraclass correlation for reliability studies in which n target are rated by k judges. Relevant to the choice of the coefficient are the appropriate statistical model for the reliability and the application to be made of the reliability results. Confidence intervals for each of the forms are reviewed.     

A coefficient alpha for test-retest data

Transient errors are caused by variations in feelings, moods, and mental states over time. If these errors are present, coefficient alpha is an inflated estimate of reliability. A true-score model is presented that incorporates transient errors for test-retest data, and a reliability estimate is derived. This estimate, referred to as the test-retest alpha, is less than coefficient alpha if transient error is present and is less susceptible to effects due to item recall than a test-retest correlation. An assumption underlying the test-retest alpha is essential tau equivalency of items. A test-retest split-half coefficient is presented as an alternative to the test-retest alpha when this assumption is violated. The test-retest alpha is the mean of all possible test-retest split-half coefficients.     

Comments on the Meehl-Waller (2002) procedure for appraisal of path analysis models

P. E. Meehl and N. G. Waller (2002) proposed an innovative method for assessing path analysis models wherein they subjected a given model, along with a set of alternatives, to risky tests using selected elements of a sample correlation matrix. Although the authors find much common ground with the perspective underlying the Meehl-Waller approach, they suggest that there are aspects of the proposed procedure that require close examination and further development. These include the selection of only one subset of correlations to estimate parameters when multiple solutions are generally available, the fact that the risky tests may test only a subset of parameters rather than the full model of interest, and the potential for different results to be obtained from analysis of equivalent models.     

Optimal sample sizes for the design of reliability studies: Power consideration

Intraclass correlation coefficients are used extensively to measure the reliability or degree of resemblance among group members in multilevel research. This study concerns the problem of the necessary sample size to ensure adequate statistical power for hypothesis tests concerning the intraclass correlation coefficient in the one-way random-effects model. In view of the incomplete and problematic numerical results in the literature, the approximate sample size formula constructed from Fisher’s transformation is reevaluated and compared with an exact approach across a wide range of model configurations. These comprehensive examinations showed that the Fisher transformation method is appropriate only under limited circumstances, and therefore it is not recommended as a general method in practice. For advance design planning of reliability studies, the exact sample size procedures are fully described and illustrated for various allocation and cost schemes. Corresponding computer programs are also developed to implement the suggested algorithms.     

Quantifying the reliability of image replication studies: The image intraclass correlation coefficient (I2C2)

This article proposes the image intraclass correlation (I2C2) coefficient as a global measure of reliability for imaging studies. The I2C2 generalizes the classic intraclass correlation (ICC) coefficient to the case when the data of interest are images, thereby providing a measure that is both intuitive and convenient. Drawing a connection with classical measurement error models for replication experiments, the I2C2 can be computed quickly, even in high-dimensional imaging studies. A nonparametric bootstrap procedure is introduced to quantify the variability of the I2C2 estimator. Furthermore, a Monte Carlo permutation is utilized to test reproducibility versus a zero I2C2, representing complete lack of reproducibility. Methodologies are applied to three replication studies arising from different brain imaging modalities and settings: regional analysis of volumes in normalized space imaging for characterizing brain morphology, seed-voxel brain activation maps based on resting-state functional magnetic resonance imaging (fMRI), and fractional anisotropy in an area surrounding the corpus callosum via diffusion tensor imaging. Notably, resting-state fMRI brain activation maps are found to have low reliability, ranging from .2 to .4. Software and data are available to provide easy access to the proposed methods.     

The primary personality factors of younger adolescent Hawaiians

A factor-analytic investigation was conducted to determine whether the primary personality factors of young adolescent Hawaiians were the same as the primary personality factors postulated by Cattell for adolescents on the United States mainland and in Germany. Responses of 584 Hawaiian adolescents (M age = 12.13 years) to a questionnaire that included items based on the High School Personality Questionnaire (IPAT, 1973) were analyzed by a procedure that adhered closely to Cattell's recommendations. The seven factors that were derived have been encountered in previous factor-analytic investigations, but, as a set, they correspond neither to the primary personality factors postulated by Cattell nor to factors postulated by other theorists who have also adopted a factorial approach to personality. They are, however, especially relevant to some psychoanalytic theories of personality.     

Sample size requirements for the design of reliability studies: precision consideration

In multilevel modeling, the intraclass correlation coefficient based on the one-way random-effects model is routinely employed to measure the reliability or degree of resemblance among group members. To facilitate the advocated practice of reporting confidence intervals in future reliability studies, this article presents exact sample size procedures for precise interval estimation of the intraclass correlation coefficient under various allocation and cost structures. Although the suggested approaches do not admit explicit sample size formulas and require special algorithms for carrying out iterative computations, they are more accurate than the closed-form formulas constructed from large-sample approximations with respect to the expected width and assurance probability criteria. This investigation notes the deficiency of existing methods and expands the sample size methodology for the design of reliability studies that have not previously been discussed in the literature.     

Integrating covariates into circumplex structures: an extension procedure for Browne’s circular stochastic process model

Circumplex structures are elements of various psychological domains. Most work focuses on assessing the circular ordering of circumplex indicators and their relationships with covariates. In this article, an extension procedure for Browne’s circumplex model is presented. Our approach models the relationships among circumplex indicators and the relationships of covariates with a latent circumplex simultaneously without affecting the circumplex indicators’ positions on the circumplex. The approach builds upon Browne’s Fourier series parameterization of a correlation function, which is used to model the latent circumplex correlation structure. It extends the shape of the correlation function to the profile of each covariate’s correlations with the circumplex. The model is specified in the framework of structural equation modeling, thereby making it possible to test various hypotheses. Procedures are presented for deriving interval estimates for the parameters that relate the covariates to the circumplex. The model is compared to other approaches for assessing the relationships of a circumplex with covariates. The results of the exemplary applications and a simulation study were in favor of the suggested model. The approach is furthermore illustrated with a real-data example, focusing on the relationships between the interpersonal circumplex and the rivalry and admiration aspects of narcissism.     

Multilevel IRT using dichotomous and polytomous response data

A structural multilevel model is presented where some of the variables cannot be observed directly but are measured using tests or questionnaires. Observed dichotomous or ordinal polytomous response data serve to measure the latent variables using an item response theory model. The latent variables can be defined at any level of the multilevel model. A Bayesian procedure Markov chain Monte Carlo (MCMC), to estimate all parameters simultaneously is presented. It is shown that certain model checks and model comparisons can be done using the MCMC output. The techniques are illustrated using a simulation study and an application involving students' achievements on a mathematics test and test results regarding management characteristics of teachers and principles.     

A double bootstrap method to analyze linear models with autoregressive error terms

A new method for the analysis of linear models that have autoregressive errors is proposed. The approach is not only relevant in the behavioral sciences for analyzing small-sample time-series intervention models, but it is also appropriate for a wide class of small-sample linear model problems in which there is interest in inferential statements regarding all regression parameters and autoregressive parameters in the model. The methodology includes a double application of bootstrap procedures. The 1st application is used to obtain bias-adjusted estimates of the autoregressive parameters. The 2nd application is used to estimate the standard errors of the parameter estimates. Theoretical and Monte Carlo results are presented to demonstrate asymptotic and small-sample properties of the method; examples that illustrate advantages of the new approach over established time-series methods are described.     

Bayesian Inference for Growth Mixture Models with Latent Class Dependent Missing Data

Growth mixture models (GMMs) with nonignorable missing data have drawn increasing attention in research communities but have not been fully studied. The goal of this article is to propose and to evaluate a Bayesian method to estimate the GMMs with latent class dependent missing data. An extended GMM is first presented in which class probabilities depend on some observed explanatory variables and data missingness depends on both the explanatory variables and a latent class variable. A full Bayesian method is then proposed to estimate the model. Through the data augmentation method, conditional posterior distributions for all model parameters and missing data are obtained. A Gibbs sampling procedure is then used to generate Markov chains of model parameters for statistical inference. The application of the model and the method is first demonstrated through the analysis of mathematical ability growth data from the National Longitudinal Survey of Youth 1997 (Bureau of Labor Statistics, U.S. Department of Labor, 1997). A simulation study considering 3 main factors (the sample size, the class probability, and the missing data mechanism) is then conducted and the results show that the proposed Bayesian estimation approach performs very well under the studied conditions. Finally, some implications of this study, including the misspecified missingness mechanism, the sample size, the sensitivity of the model, the number of latent classes, the model comparison, and the future directions of the approach, are discussed.     

Estimating the accuracy of dichotomous judgments

A reliability study is assumed to be carried out with each of a number of observers making a dichotomous judgment concerning each of a sample of subjects. A nonparametric model is proposed for the errors underlying such judgments, and conditions are given under which Cochran'sQ statistic is valid for testing the hypothesis of no systematic differences among the judgments of the different observers. Inferences concerning the probabilities of error are shown to be possible in terms of the intraclass correlation coefficient. A numerical example is given.This research was supported in part by NIMH grant MH-03546. I am indebted to Dr. E. I. Burdock, Associate Research Scientist, Biometrics Research, for his valuable suggestions and criticisms.     

Estimation of maximal reliability for multiple‐component instruments in multilevel designs

A procedure for point and interval estimation of maximal reliability of multiple‐component measuring instruments in multi‐level settings is outlined. The approach is applicable to hierarchical designs in which individuals are nested within higher‐order units and exhibit possibly related performance on components of a given homogeneous scale. The method is developed within the framework of multi‐level factor analysis. The proposed procedure is illustrated with an empirical example.     

A generalized family of coefficients of relational agreement for numerical scales

A family of coefficients of relational agreement for numerical scales is proposed. The theory is a generalization to multiple judges of the Zegers and ten Berge theory of association coefficients for two variables and is based on the premise that the choice of a coefficient depends on the scale type of the variables, defined by the class of admissible transformations. Coefficients of relational agreement that denote agreement with respect to empirically meaningful relationships are derived for absolute, ratio, interval, and additive scales. The proposed theory is compared to intraclass correlation, and it is shown that the coefficient of additivity is identical to one measure of intraclass correlation.The author thanks the Editor and anonymous reviewers for helpful suggestions.     

Interrater reliability and agreement of performance ratings: A methodological comparison

This paper demonstrates and compares methods for estimating the interrater reliability and interrater agreement of performance ratings. These methods can be used by applied researchers to investigate the quality of ratings gathered, for example, as criteria for a validity study, or as performance measures for selection or promotional purposes. While estimates of interrater reliability are frequently used for these purposes, indices of interrater agreement appear to be rarely reported for performance ratings. A recommended index of interrater agreement, theT index (Tinsley & Weiss, 1975), is compared to four methods of estimating interrater reliability (Pearsonr, coefficient alpha, mean correlation between raters, and intraclass correlation). Subordinate and superior ratings of the performance of 100 managers were used in these analyses. The results indicated that, in general, interrater agreement and reliability among subordinates were fairly high. Interrater agreement between subordinates and superiors was moderately high; however, interrater reliability between these two rating sources was very low. The results demonstrate that interrater agreement and reliability are distinct indices and that both should be reported. Reasons are discussed as to why interrater reliability should not be reported alone.This paper is based, in part, on a thesis submitted to East Carolina University by the second author. Portions of this study were presented at the American Psychological Association meeting in New Orleans, LA, August, 1989. The authors would like to thank Michael Campion and two anonymous reviewers for their comments on earlier drafts of this paper.     

The discrimination of two racial samples

Nineteen different anthropometric measures were obtained on all members of each of two racial groups. A procedure was developed and applied to the data to give maximum differentiation between the groups. The method is applicable wherever we have a large number of independent variables and a dependent variable. In such cases, the conventional methods for determining multiple regression constants are very laborious. An iteration method is presented which is more rapid than any with which the writers are familiar. The method selects in sequence those variables which together yield the largest multiple correlation with the criterion. At each step in the procedure, rapid estimates of the regression weights and the multiple correlation at that point are available.