Mokken Scale Analysis for Dichotomous Items Using Marginal Models

Scalability coefficients play an important role in Mokken scale analysis. For a set of items, scalability coefficients have been defined for each pair of items, for each individual item, and for the entire scale. Hypothesis testing with respect to these scalability coefficients has not been fully developed. This study introduces marginal modelling as a framework to derive the standard errors for the scaling coefficients and test hypotheses about these coefficients. Several examples demonstrate the possibilities of marginal modelling in Mokken scale analysis. These possibilities include testing whether Mokken’s criteria for a scale are satisfied, testing whether scalability coefficients of different items are equal, and testing whether scalability coefficients are equal across different groups.     

Covariate-free and Covariate-dependent Reliability

Classical test theory reliability coefficients are said to be population specific. Reliability generalization, a meta-analysis method, is the main procedure for evaluating the stability of reliability coefficients across populations. A new approach is developed to evaluate the degree of invariance of reliability coefficients to population characteristics. Factor or common variance of a reliability measure is partitioned into parts that are, and are not, influenced by control variables, resulting in a partition of reliability into a covariate-dependent and a covariate-free part. The approach can be implemented in a single sample and can be applied to a variety of reliability coefficients.     

Standard errors of two-level scalability coefficients

For the construction of tests and questionnaires that require multiple raters (e.g., a child behaviour checklist completed by both parents) a novel ordinal scaling technique is currently being further developed, called two-level Mokken scale analysis. The technique uses within-rater and between-rater coefficients to assess the scalability of the test. These coefficients are generalizations of Mokken's scalability coefficients. In this paper we derived standard errors for the two-level coefficients and for their ratios. The coefficients, the estimates, the estimated standard errors and the software implementation are discussed and illustrated using a real-data example, and a small-scale simulation study demonstrates the accuracy of the estimates.     

Association coefficients of identity and proportionality for metric scales

Zegers' (1986) chance-corrected coefficients of association are derived by alternative methods. A different definition of chance correction is used. It is shown that our correction and that of Zegers are identical for large samples. Three possible assumptions for the derivation of metric coefficients are examined. The first, variable reflection, formulated by Zegers and ten Berge (1985), leads to coefficients that require chance-correction. Two other assumptions, zero covariance and covariance reflection, are proposed and it is shown that the latter two assumptions lead directly to coefficients of identity and proportionality that do not require chance correction (i.e., are identical to the Zegers, 1986, corrected coefficients).We are indebted to Robyn M. Dawes, Carnegie-Mellon University, for stimulating our interest in this project, and for helpful suggestions.     

Test reliability and correction for attenuation

Evidence is cited to show that specificity, or lack of equivalence, in the comparable forms of tests has a tendency to lower the value of reliability coefficients but has no tendency to lower the value of observed trait coefficients. This implies that the greater the lack of equivalence, the higher will be coefficients corrected for attenuation. Errors of measurement are supposed to reduce the magnitude of observed trait coefficients. Since specificity does not lower the correlation between two tests and since the split-half and equivalent-form reliability coefficients treat specificity as error, it follows that these two coefficients cannot legitimately be used in Spearman's correction-for-attenuation formula.     

Comparing standardized coefficients in structural equation modeling: a model reparameterization approach

We propose a two-stage method for comparing standardized coefficients in structural equation modeling (SEM). At stage 1, we transform the original model of interest into the standardized model by model reparameterization, so that the model parameters appearing in the standardized model are equivalent to the standardized parameters of the original model. At stage 2, we impose appropriate linear equality constraints on the standardized model and use a likelihood ratio test to make statistical inferences about the equality of standardized coefficients. Unlike other existing methods for comparing standardized coefficients, the proposed method does not require specific modeling features (e.g., specification of nonlinear constraints), which are available only in certain SEM software programs. Moreover, this method allows researchers to compare two or more standardized coefficients simultaneously in a standard and convenient way. Three real examples are given to illustrate the proposed method, using EQS, a popular SEM software program. Results show that the proposed method performs satisfactorily for testing the equality of standardized coefficients.     

The Situational Specificity of Validities: Correcting for Statistical Artifacts Does Not Always Reduce the Trans-Situational Variability of Correlation Coefficients

Validity generalization research suggests that much of the variability in validity coefficients is due to statistical artifacts, such as sampling error. Correcting for these artifacts usually reduces the trans-situational variability in validity coefficients, but the reverse can happen. Conditions under which corrections for statistical artifacts can increase the variability in validity coefficients are described, examples of meta-analyses reporting larger corrected than observed variance in validity are cited, and implications for the situational specificity of validity coefficients are discussed. In general, the true variability in validity coefficients based on large samples is often larger than the observed variability in test validities.     

A note on reliability estimation for a test with components of unknown functional lengths

This research note proposes two reliability coefficients for tests with components of unknown functional lengths. The derived coefficients are extensions of the techniques devised by Kristof and Feldt and do not require a reduction of test components into parts. Simulation study indicates that the new coefficients yield reasonably stable reliability estimates when the number of test components is small.     

A comparison of two‐stage procedures for testing least‐squares coefficients under heteroscedasticity

This study explores the performance of several two‐stage procedures for testing ordinary least‐squares (OLS) coefficients under heteroscedasticity. A test of the usual homoscedasticity assumption is carried out in the first stage of the procedure. Subsequently, a test of the regression coefficients is chosen and performed in the second stage. Three recently developed methods for detecting heteroscedasticity are examined. In addition, three heteroscedastic robust tests of OLS coefficients are considered. A major finding is that performing a test of heteroscedasticity prior to applying a heteroscedastic robust test can lead to poor control over Type I errors.     

Marginal maximum likelihood estimation of item response theory (IRT) equating coefficients for the common-examinee design

A method of estimating item response theory (IRT) equating coefficients by the common-examinee design with the assumption of the two-parameter logistic model is provided. The method uses the marginal maximum likelihood estimation, in which individual ability parameters in a common-examinee group are numerically integrated out. The abilities of the common examinees are assumed to follow a normal distribution but with an unknown mean and standard deviation on one of the two tests to be equated. The distribution parameters are jointly estimated with the equating coefficients. Further, the asymptotic standard errors of the estimates of the equating coefficients and the parameters for the ability distribution are given. Numerical examples are provided to show the accuracy of the method.     

The evaluation of multiple and partial correlation coefficients from the factorial matrix

This paper shows how to compute multiple correlation coefficients, partial correlation coefficients, and regression coefficients from the factorial matrix. Special emphasis is given to computation technique and to approximation formulas. The method is extremely flexible in application since it may be applied to any subset of the original set of observed variables. It is also extremely useful when many of these coefficients are desired.     

Methods for averaging alpha coefficients in reliability generalization studies

The reliability generalization (RG) approach is a kind of meta-analysis that aims to statistically integrate a set of independent reliability coefficients obtained in several applications of a test, with the purpose of characterizing the measurement error and determining which factors related to the studies and samples can explain its variability. Diverse procedures have been proposed in the literature for averaging a set of independent alpha coefficients, and there is no consensus about which methods are best. Here, we present the results of a Monte Carlo simulation study, comparing the performance of twelve procedures proposed in Feldt and Charter, in terms of bias and mean square error. These procedures differ from each other in the transformation (or not) of the coefficients, and in the application or not of a weighting scheme based on sample size. Our results recommend using weighted methods in contrast to unweighted ones, and transforming the coefficients by the Hakstian and Whalen's proposal or by the proposal based on the square root of the inverse alpha coefficient. Lastly, we discuss the relations between the diverse procedures for averaging alpha coefficients with fixed-effects, random-effects, and varying coefficients models.     

VALCOR: A program for estimating standard error, confidence intervals, and probability of corrected validity

VALCOR is a Turbo-Basic program that corrects the observed (uncorrected) validity coefficients for criterion and predictor unreliability and range restriction in the predictor. Furthermore, using the formulas for the standard error of functions of correlations derived by Bobko and Rieck (1980), the program provides an estimation of the standard error, the confidence intervals, and the probability of the corrected validity coefficients. In this way, the probability and the boundaries of the corrected validity coefficients may be reported together with the probability of the uncorrected validity coefficients. The results are presented on the computer screen and may be saved in an external file.     

A Comparison of Modern and Popular Approaches to Calculating Reliability for Dichotomously Scored Items

Recent work on reliability coefficients has largely focused on continuous items, including critiques of Cronbach’s alpha. Although two new model-based reliability coefficients have been proposed for dichotomous items (Dimitrov, 2003a,b; Green & Yang, 2009a), these approaches have yet to be compared to each other or other popular estimates of reliability such as omega, alpha, and the greatest lower bound. We seek computational improvements to one of these model-based reliability coefficients and, in addition, conduct initial Monte Carlo simulations to compare coefficients using dichotomous data. Our results suggest that such improvements to the model-based approach are warranted, while model-based approaches were generally superior.     

Random intercept item factor analysis

The common factor model assumes that the linear coefficients (intercepts and factor loadings) linking the observed variables to the latent factors are fixed coefficients (i.e., common for all participants). When the observed variables are participants' observed responses to stimuli, such as their responses to the items of a questionnaire, the assumption of common linear coefficients may be too restrictive. For instance, this may occur if participants consistently use the response scale idiosyncratically. To account for this phenomenon, the authors partially relax the fixed coefficients assumption by allowing the intercepts in the factor model to change across participants. The model is attractive when m factors are expected on the basis of substantive theory but m + 1 factors are needed in practice to adequately reproduce the data. Also, this model for single-level data can be fitted with conventional software for structural equation modeling. The authors demonstrate the use of this model with an empirical data set on optimism in which they compare it with competing models such as the bifactor and the correlated trait-correlated method minus 1 models.     

A Turbo Pascal program for the computation of scale-dependent association coefficients

A Turbo Pascal program is described that computes scale-dependent association coefficients belonging to Zeger’s general family of association coefficients. Members of this family can be constructed that are adapted to the scale type of the variables, in that they are insensitive to admissible transformations of the variables but sensitive to inadmissible transformations. The program can compute (chance-corrected) association coefficients between variables of eight different scale types and between variables of mixed scale type.     

On Association Coefficients for 2×2 Tables and Properties That Do Not Depend on the Marginal Distributions

We discuss properties that association coefficients may have in general, e.g., zero value under statistical independence, and we examine coefficients for 2×2 tables with respect to these properties. Furthermore, we study a family of coefficients that are linear transformations of the observed proportion of agreement given the marginal probabilities. This family includes the phi coefficient and Cohen’s kappa. The main result is that the linear transformations that set the value under independence at zero and the maximum value at unity, transform all coefficients in this family into the same underlying coefficient. This coefficient happens to be Loevinger’s H.     

A family of chance-corrected association coefficients for metric scales

A chance-corrected version of the family of association coefficients for metric scales proposed by Zegers and ten Berge is presented. It is shown that a matrix with chance-corrected coefficients between a number of variables is Gramian. The members of the chance-corrected family are shown to be partially ordered. The author is obliged to Jos ten Berge for helpful comments.     

A family of association coefficients for metric scales

Four types of metric scales are distinguished: the absolute scale, the ratio scale, the difference scale and the interval scale. A general coefficient of association for two variables of the same metric scale type is developed. Some properties of this general coefficient are discussed. It is shown that the matrix containing these coefficients between any number of variables is Gramian. The general coefficient reduces to specific coefficients of association for each of the four metric scales. Two of these coefficients are well known, the product-moment correlation and Tucker's congruence coefficient. Applications of the new coefficients are discussed.     

SOME PERSONALITY TESTS APPLIED ON IDENTICAL TWINS

(1) The study concerns Rorschach, Children Apperception Test, Bender Gestalt Test and Draw-a-Man. Data have been obtained on a number of personality variables from 12 pairs of identical twins at the ages of 9;6–10;6. (2) For the Ro 17 out of 21 and for the CAT 15 out of 23 intraclass coefficients are positive. The coefficients for the Gestalt test and Draw-a-Man are high and positive. It is argued that intraclass coefficients must be considered as underestimating both the true reliability of the ratings and the correlation between true scores for identical twins. (3) The intertest correlations give support to the construct validity of the ratings on the variables studied. Especially favourable are the intercorrelations for 'General adjustment'.