Statistical properties of the rWG(J) index of agreement

L. R. James, R. G. Demaree, and G. Wolf (1984) introduced rWG(J) to estimate interrater agreement for a group. This index is calculated by comparing an observed group variance with an expected random variance. As researchers have gained experience using this index, several questions have arisen. What are the consequences of replacing values beyond the unit interval by 0? What is the dependence of rWG(J) on the group size? The authors' simulations show that a positive bias is caused by the truncation, but for large population values of rWG(J) it is negligible. Also, in this case, the group size has no effect on the expected value of rWG(J). For inference on rWG(J), researchers can exploit the availability of computers to simulate data from the hypothesized distribution and then compare the simulation results for rWG(J) with the actual values. In addition, it is shown how the bootstrap method can be used for comparing the indices of 2 groups.     

Evaluating the MSCEIT V2.0 via CFA: comment on Mayer et al. (2003)

This investigation uncovered several substantial errors in the confirmatory factor analysis results reported by J. D. Mayer, P. Salovey, D. R. Caruso, and G. Sitarenios (see record 2003-02341-015). Specifically, the values associated with the close-fit indices (normed fit index, Tucker-Lewis Index, and root-mean-square error of approximation) are inaccurate. A reanalysis of the Mayer et al. subscale intercorrelation matrix provided accurate values of the close-fit indices, which resulted in different evaluations of the models tested by J. D. Mayer et al. Contrary to J. D. Mayer et al., the 1-factor model and the 2-factor model did not provide good fit. Although the 4-factor model was still considered good fitting, the non-constrained 4-factor model yielded a non-positive definite matrix, which was interpreted to be due to the fact that two of the branch-level factors (Perceiving and Facilitating) were collinear, suggesting that a model with 4 factors was implausible.     

Screening for malingering in a criminal-forensic sample with the personality assessment inventory

In this study, the authors examined how overreporting of psychopathology indices on the Personality Assessment Inventory (PAI; L. C. Morey, 1991) performed as screening measures for malingering in a sample of 166 defendants undergoing pretrial court-ordered evaluations in the federal criminal justice system. Using results from the Structured Interview of Reported Symptoms (SIRS; R. Rogers, R. M. Bagby, & S. E. Dickens, 1992) as the criterion measure of malingering, the authors found that the Negative Impression scale (NIM) was the most effective PAI screening measure (cut score > or = 81 T). NIM performed as well as an established comparison measure from the Minnesota Multiphasic Personality Inventory--2 (J. N. Butcher, W. G. Dahlstrom, J. R. Graham, A. Tellegen, & B. Kaemmer, 1989; Infrequency [F] cut score > or = 95 T), supporting the use of either of these indices as reasonable screening measures to identify potential malingerers for subsequent evaluation.     

The Problem with Having Two Watches: Assessment of Fit When RMSEA and CFI Disagree

The root mean square error of approximation (RMSEA) and the comparative fit index (CFI) are two widely applied indices to assess fit of structural equation models. Because these two indices are viewed positively by researchers, one might presume that their values would yield comparable qualitative assessments of model fit for any data set. When RMSEA and CFI offer different evaluations of model fit, we argue that researchers are likely to be confused and potentially make incorrect research conclusions. We derive the necessary as well as the sufficient conditions for inconsistent interpretations of these indices. We also study inconsistency in results for RMSEA and CFI at the sample level. Rather than indicating that the model is misspecified in a particular manner or that there are any flaws in the data, the two indices can disagree because (a) they evaluate, by design, the magnitude of the model's fit function value from different perspectives; (b) the cutoff values for these indices are arbitrary; and (c) the meaning of “good” fit and its relationship with fit indices are not well understood. In the context of inconsistent judgments of fit using RMSEA and CFI, we discuss the implications of using cutoff values to evaluate model fit in practice and to design SEM studies.     

Adjusting Incremental Fit Indices for Nonnormality

A variety of indices are commonly used to assess model fit in structural equation modeling. However, fit indices obtained from the normal theory maximum likelihood fit function are affected by the presence of nonnormality in the data. We present a nonnormality correction for 2 commonly used incremental fit indices, the comparative fit index and the Tucker-Lewis index. This correction uses the Satorra-Bentler scaling constant to modify the sample estimate of these fit indices but does not affect the population value. We argue that this type of nonnormality correction is superior to the correction that changes the population value of the fit index implemented in some software programs. In a simulation study, we demonstrate that our correction performs well across a variety of sample sizes, model types, and misspecification types.     

Power and errors in adverse impact indices

Adverse impact indices are intended to detect potentially discriminatory employment practices, and although the number of such indices continues to increase, with a few exceptions, little is known about the comparative performance of those indices under various conditions. We examined the performance of eight such indices, including statistical significance tests, ad hoc “practical” tests proposed by the courts, and effect size measures. The power of these indices was examined under various levels of adverse impact, applicant pool sizes, and selection ratios. Although no index performed well in all circumstances, typically, Fisher’s exact test and the Z test for proportions (with correction for continuity) performed adequately most often. However, in some conditions, these indices were outperformed by some of the “practical” tests.     

Assessing differential functioning in a satisfaction scale

In this study, an item response theory-based differential functioning of items and tests (DFIT) framework (N. S. Raju, W. J. van der Linden, & P. F. Fleer, 1995) was applied to a Likert-type scale. Several differential item functioning (DIF) analyses compared the item characteristics of a 10-item satisfaction scale for Black and White examinees and for female and male examinees. F. M. Lord's (1980) chi-square and the extended signed area (SA) measures were also used. The results showed that the DFIT indices consistently performed in the expected manner. The results from Lord's chi-square and the SA procedures were somewhat varied across comparisons. A discussion of these results along with an illustration of an item with significant DIF and suggestions for future DIF research are presented.     

群体断层强度测量指标的构建与检验：一个整合的视角

群体断层概念近些年来在理论和实证研究方面都取得了重要的进展, 但有关断层强度的测量却始终呈现出滞后的态势。已有的文献表明只有FLS, Fau和PMD三个指标对群体断层强度进行了系统而完整的测量, 且三个指标都表现出相当的稳健性。尽管如此, 三大指标却都没有能够反映断层宽度这个重要维度。本研究在以往文献的基础上构建了整合群体断层强度测量指标, 很好地解决了这一问题。整合群体断层强度指标具体表现为亚群体内部一致性、亚群体间差异性和断层宽度的乘积。我们从以往研究中提炼了10个准则对新构建的指标进行了检验, 结果显示, 整合群体断层强度指标要显著优于以往的测度, 10个准则全部能够很好地拟合。本研究最后讨论了基于主观感知的断层强度的测量问题, 并提出了采用回归系数作为衡量亚群体间距离的PMD改进算法。     

The effects of payoffs and prior probabilities on indices of performance and cutoff location in recognition memory

We compared the effects of payoffs and prior probability levels on indices of performance and cutoff location in recognition memory. The performance indices considered were d’ from signal-detection theory and two corrections for guessing from simple threshold models. Shifts in cutoff location of comparable size were found for changes in priors and payoffs. The existence of a memory load and the inclusion of feedback did not affect the extent of shifts in cutoff location. Priors affected the corrections for guessing but did not affect d’. These results indicate that d’ remains the preferred index of performance and that the corrections for guessing are not criterion-free measures. The problem of finding an index of recognition performance appropriate for comparing recall and recognition is discussed, as are alternative explanations for conservative cutoff placement.     

Dynamic visuomotor synchronization: Quantification of predictive timing

When a moving target is tracked visually, spatial and temporal predictions are used to circumvent the neural delay required for the visuomotor processing. In particular, the internally generated predictions must be synchronized with the external stimulus during continuous tracking. We examined the utility of a circular visual-tracking paradigm for assessment of predictive timing, using normal human subjects. Disruptions of gaze–target synchronization were associated with anticipatory saccades that caused the gaze to be temporarily ahead of the target along the circular trajectory. These anticipatory saccades indicated preserved spatial prediction but suggested impaired predictive timing. We quantified gaze–target synchronization with several indices, whose distributions across subjects were such that instances of extremely poor performance were identifiable outside the margin of error determined by test–retest measures. Because predictive timing is an important element of attention functioning, the visual-tracking paradigm and dynamic synchronization indices described here may be useful for attention assessment.     

When fit indices and residuals are incompatible

Standard chi-square-based fit indices for factor analysis and related models have a little known property: They are more sensitive to misfit when unique variances are small than when they are large. Consequently, very small correlation residuals indicating excellent fit can be accompanied by indications of bad fit by the fit indices when unique variances are small. An empirical example of this incompatibility between residuals and fit indices is provided. For illustrative purposes, an artificial example is provided that yields exactly the same correlation residuals as the empirical example but has larger unique variances. For this example, the fit indices indicate excellent fit. A theoretical explanation for this phenomenon is provided using relationships between unique variances and eigenvalues of the fitted correlation matrix.     

Palestinian Archaeology and the New Testament

A review article on J. Finegan, The Archaeology of the New Testament, Princeton, 1969 (pp. 273, figs. 296; preface, indices, chronological guides)     

Intra-individual Response Variability as an Indicator of Insufficient Effort Responding: Comparison to Other Indicators and Relationships with Individual Differences

Surveys are one of the most popular ways to collect employee information. Because of their widespread use, data quality is an increasingly important concern. The purpose of this paper is to (1) introduce the intra-individual response variability (IRV) index as an easily calculated and flexible way to detect insufficient effort responding (IER); (2) examine the extent to which various IER indices detect the same or different respondents engaging in IER behavior; and (3) investigate relationships between individual differences and commonly used IER indices to better understand systematic and theoretically relevant IER behavior. In a two-part study, 199 undergraduates responded to questionnaires online, and various IER indices were calculated. The IRV index identifies different respondents than other IER indices. Values on the IRV index (as well as other IER indices) are related to scores on theoretically meaningful individual differences in conscientiousness, agreeableness, and boredom proneness. This study provides researchers with a robust, easily calculated, and flexible means for screening questionnaire data for IER behavior. Practical recommendations for finding and making decisions about IER behavior patterns are provided. This study introduces the IRV index, an extension of the long string, used to identify survey research participants who likely engaged in one type of IER behavior. It is also one of the first studies to evaluate the extent to which IER indices identify different respondents as having engaged in IER and provides additional evidence that values on these indices are related to individual differences.     

Multivariate information transmission

A multivariate analysis based on transmitted information is presented. It is shown that sample transmitted information provides a simple method for measuring and testing association in multi-dimensional contingency tables. Relations with analysis of variance are pointed out, and statistical tests are described.This work was supported in part by the Air Force Human Factors Operations Research Laboratories, and in part jointly by the Army, Navy, and Air Force under contract with the Massachusetts Institute of Technology.Several of the indices and tests discussed in this paper have been developed independently by J. E. Keith Smith (11) at the University of Michigan, and by W. R. Garner at Johns Hopkins University.     

Relation of person-environment fit to career certainty

This study examined the person-environment fit-career outcome relation by observing interests, self-efficacy and parental support as person aspects and major and occupational choice as separate indicators of the environment. Career certainty was selected as a career outcome. Two dimensional fit indices (People-Things and Data-Ideas) were examined as was the incremental validity of adding prestige fit indices. In addition, we examined the moderation of these PE fit-outcome relations by ethnicity (Anglos vs. Latino/a’s). Results revealed that the PE fit predictors were moderately related and none of the two dimensional PE fit predictors significantly related to career certainty, while nearly all of the Prestige PE fit-certainty relations were significant. Support was found for the spherical structure of work types and person variables [Tracey, T. J. G., & Rounds, J. (1996b). Contributions of the spherical representation of vocational interests. Journal of Vocational Behavior, 48, 85-95].     

Confidence intervals for gamma-family measures of ordinal association

This research focused on confidence intervals (CIs) for 10 measures of monotonic association between ordinal variables. Standard errors (SEs) were also reviewed because more than 1 formula was available per index. For 5 indices, an element of the formula used to compute an SE is given that is apparently new. CIs computed with different SEs were compared in simulations with small samples (N = 25, 50, 75, or 100) for variables with 4 or 5 categories. With N > 25, many CIs performed well. Performance was best for consistent CIs due to N. Cliff and colleagues (N. Cliff, 1996; N. Cliff & V. Charlin, 1991; J. D. Long & N. Cliff, 1997). CIs for Spearman's rank correlation were also examined: Parameter coverage was erratic and sometimes egregiously underestimated.     

Disentangling perceptual and response bias in unilateral neglect: a methodological proposal

Bisiach and his colleagues have developed a new version of Milner's "landmark task" for the purpose of separating "perceptual" and "response" biases in neglect patients. Subjects are required to decide which is the longer (or the shorter) of the two portions of a pre-bisected horizontal line. The authors proposed two indices to measure perceptual and response bias respectively. However, these indices are not mathematically independent of one another. Moreover, they do not exploit all of the information available in the data, since they do not consider the effect of the different transection locations across trials. We now propose an alternative means of analyzing data from the revised Landmark task, which generates independent estimates of perceptual and response biases. The method and its theoretical foundation are summarized, and illustrative data obtained from brain damaged patients and control subjects are presented.     

A theory of consistency of ordering generalizable to tailored testing

Measures of consistency and completeness of order relations derived from test-type data are proposed. The measures are generalized to apply to incomplete data such as from tailored testing. The measures are based on consideration of the items-plus-persons by items-plus-persons matrix as an adjacency matrix in which a 1 means that the row element, whether item or person, dominated the column element. From this the number of item-item and person-person dominance relations can be derived. Indices of consistency are developed which compare the number of dominance relations in a matrix to the number that would occur if it were perfectly consistent and to the number in a random matrix; relations to other indices are noted.The research reported here was supported by the Office of Naval Research, Contract N00014-75-C-0684, NR150-373. The author wishes to thank Dr. Thomas J. Reynolds for a number of helpful discussions of this material.     

Strain-dependent differences in LTP and hippocampus-dependent memory in inbred mice

Many studies have used "reverse" genetics to produce "knock-out" and transgenic mice to explore the roles of various molecules in long-term potentiation (LTP) and spatial memory. The existence of a variety of inbred strains of mice provides an additional way of exploring the genetic bases of learning and memory. We examined behavioral memory and LTP expression in area CA1 of hippocampal slices prepared from four different inbred strains of mice: C57BL/6J, CBA/J, DBA/2J, and 129/SvEms-+(Ter?)/J. We found that LTP induced by four 100-Hz trains of stimulation was robust and long-lasting in C57BL/6J and DBA/2J mice but decayed in CBA/J and 129/SvEms-+(Ter?)/J mice. LTP induced by one 100-Hz train was significantly smaller after 1 hr in the 129/SvEms-+(Ter?)/J mice than in the other three strains. Theta-burst LTP was shorter lasting in CBA/J, DBA/2J, and 129/SvEms-+(Ter?)/J mice than in C57BL/6J mice. We also observed specific memory deficits, among particular mouse strains, in spatial and nonspatial tests of hippocampus-dependent memory. CBA/J mice showed defective learning in the Morris water maze, and both DBA/2J and CBA/J strains displayed deficient long-term memory in contextual and cued fear conditioning tests. Our findings provide strong support for a genetic basis for some forms of synaptic plasticity that are linked to behavioral long-term memory and suggest that genetic background can influence the electrophysiological and behavioral phenotypes observed in genetically modified mice generated for elucidating the molecular bases of learning, memory, and LTP.