Ordinal consistency and ordinal true scores

This paper argues that test data are ordinal, that latent trait scores are only determined ordinally, and that test data are used largely for ordinal purposes. Therefore it is desirable to develop a test theory based only on ordinal assumptions. A set of ordinal assumptions is presented, including an ordinal version of local independence. From these assumptions it is first shown that the gamma-correlation between two tests is the product of their gamma-correlations with the true latent order. The theory is generalized to allow for heterogeneous tests by defining a weighted average local independence. The tau-correlations between total score and the latent order can be found in both homogeneous and heterogeneous cases, and a system of differential item weighting to maximize the tau-correlation between weighted items and the latent order is provided. Thus a purely ordinal test theory seems possible.Part of this work was done while the author was a Visiting Fellow at Macquarrie University. The paper has benefitted from discussions with Professors Thomas J. Reynolds and Roderick P. McDonald and from the comments of several anonymous reviewers.     

Reliability of test scores in nonparametric item response theory

Three methods for estimating reliability are studied within the context of nonparametric item response theory. Two were proposed originally by Mokken (1971) and a third is developed in this paper. Using a Monte Carlo strategy, these three estimation methods are compared with four classical lower bounds to reliability. Finally, recommendations are given concerning the use of these estimation methods.The authors are grateful for constructive comments from the reviewers and from Charles Lewis.     

Reactivity and unreliability of husbands as participant observers

This study examined the reliability and reactivity of participant observation. Twelve couples engaged in two discussions in a laboratory, one on a conflict topic and one on a consensus topic. For the eight couples in the experimental sequences, a baseline discussion was followed by participant observation in which the husband recorded his wife's use of I while continuing to converse. The four couples in the control sequences conversed without participant observation. Compared to observations made by a trained observer, the husbands were unreliable observers, underestimating their wives' use of I. Participant observation was reactive (P<.01), with more wives decreasing their use of I in the experimental sequences than in the control sequences (p<.094). The topic of discussion did not differentially influence reliability or reactivity.This research was conducted as the first author's independent doctoral research project. We extend gratitude to the Research Council of the University of North Carolina at Greensboro for Grant 0-2-110-218-XXXXX-7548; the Statistical Consulting Center of the University of North Carolina at Greensboro; two anonymous reviewers; and our research assistants, Suzanne Brannon, Ben Hardie, Kathy Lindamood, Amy Mitchem, Regina Pierce, and Elga Wulfert. Portions of this paper were presented at the meeting of the Association for Advancement of Behavior Therapy, Washington, D.C., December 1983.     

Premature conclusions about the signal-to-noise ratio in structural equation modeling research: A commentary on Yuan and Fang (2023)

In a recent article published in this journal, Yuan and Fang (British Journal of Mathematical and Statistical Psychology, 2023) suggest comparing structural equation modeling (SEM), also known as covariance-based SEM (CB-SEM), estimated by normal-distribution-based maximum likelihood (NML), to regression analysis with (weighted) composites estimated by least squares (LS) in terms of their signal-to-noise ratio (SNR). They summarize their findings in the statement that “[c]ontrary to the common belief that CB-SEM is the preferred method for the analysis of observational data, this article shows that regression analysis via weighted composites yields parameter estimates with much smaller standard errors, and thus corresponds to greater values of the [SNR].” In our commentary, we show that Yuan and Fang have made several incorrect assumptions and claims. Consequently, we recommend that empirical researchers not base their methodological choice regarding CB-SEM and regression analysis with composites on the findings of Yuan and Fang as these findings are premature and require further research.     

Internal consistency and reliability of alternate short forms of the college women's assertion sample: Clinical and research implications

Assertive behavior is most often assessed with self-report or role-play measures. The latter modality is preferred because it provides for the sampling of the structure of behavior and for the consideration of the situational context. MacDonald (1978) has developed such an assessment device but it is limited by the length of time for administration and scoring. Two studies were conducted to reconstruct reliable alternate short forms. The first study describes the selection of items and demonstrates the internal consistency of the alternate forms. The second study demonstrates the alternate form and retest reliability and provides normative statistics. We conclude that reliable alternate short forms have been constructed to be used in research in clinical applications.This research was supported by the Marie Wilson Howells Fund.Alternate short forms of the CWAS may be obtained from the first author.     

Some relationships between the information function of IRT and the signal/noise ratio and reliability coefficient of classical test theory

It is shown that IRTs information function for an item is functionally related to local versions of classical test theories' signal/noise ratio and reliability coefficient.     

Swedish translations of the Suicide Probability Scale, Perceived Social Support from Friends and Family Scales, and the Scale for Interpersonal Behavior: A reliability analysis

The Suicide Probability Scale (SPS), the Perceived Social Support (PSS) from Friends (PSS-Fr) and Family (PSS-Fa) scales, and the Scale for Interpersonal Behavior (SIB) were translated into Swedish and their reliability was estimated in a university student sample. The reliability coefficients indicated that both subscales and the total scales of the SPS, PSS, and SIB possess highly adequate reliabilities. The intercorrelations among the subscales and between the subscales and the total scales were found to be highly significant. The results of the study supported the use of SPS, PSS-Fr and PSS-Fa, and the SIB as reliable methods for assessing suicide risk, perceived social support from friends and family, and assertive behavior.     

Inferential procedures for multifaceted coefficients of generalizability

A two-facet measurement model with broad application in the behavioral sciences is identified, and its coefficient of generalizability (CG) is examined. A normalizing transformation is proposed, and an asymptotic variance expression is derived. Three other multifaceted measurement models and CGs are identified, and variance expressions are presented. Next, an empirical investigation of the procedures follows, and it is shown that, in most cases, Type I error control in inferential applications is precise, and that the estimates are relatively efficient compared with the correlation coefficient. Implications for further research and for practice are noted. In an Appendix, four additional models, CGs, and variance expressions are presented.The research reported herein formed part of a doctoral dissertation conducted by Marsha Schroeder (Schroeder, 1986), under the direction of Ralph Hakstian, at the University of British Columbia. We acknowledge with thanks the contributions to this research of Todd Rogers and James Steiger. We are also very indebted to an mous reviewer who provided some important clarifications in connection with two of the models considered. Some support for this research was provided by a grant to Ralph Hakstian from the Natural Sciences and Engineering Research Council of Canada.     

Type A behaviour in pre-school children: Interrater reliability, stability over six months and subcomponents

Type A behaviour was measured in 250 boys and 245 girls (ages 3-7) using a Swedish version of Matthews' Youth Test for Health (MYTH). In addition, speech and voice characteristics (speed, loudness, response latency) were measured. Each child was independently rated by two play group leaders and measurements were repeated after a six month interval. Interrater reliability (first session) was 0.76 for both sexes and did not change with the age of the child. Boys obtained significantly (p less than 0.0001) higher scores than girls. Correlations over the six month period were 0.64 and 0.60 (p less than 0.001) for the two possible combinations of different raters and slightly higher (0.75 and 0.68, p less than 0.001) when the same rater was used. Two components of Type A behaviour were identified from a factor analysis of the data: impatience and competitiveness, and they accounted for 57% of the total variance. As for the total scale, boys obtained significantly higher scores than girls on both subscales. Scores on the competitiveness scale increased systematically with age. Speech and voice characteristics correlated significantly with Type A scores (total Type A behaviour: r = 0.50, using different raters, and 0.71, using the same raters, p less than 0.001) and, once again, boys obtained significantly (p less than 0.001) higher scores than girls. The results are in close agreement with the American findings from the original scale. It was concluded that the scale provides a reliable and valid instrument for measuring Type A or Type A-like behaviours in pre-school children.