Investigating With IRT and MDS Approaches Translation and Adaptation of Rating Scales for Spanish-Speaking Populations

The goal of this study is to investigate how features of a rating scale developed for English-speaking populations interact with Spanish-speaking respondents' response styles and functional categories of judgment. A sample of 400 Spanish-speaking students took a translated scale and a scaling task developed to measure response sets and functional categories of judgment, respectively. Three response set models—extreme response, central tendency, and acquiescence—under two conditions—base and revised with respondents' functional categories—were studied with item response theory and multidimensional scaling methods. Revising the number of scale categories with the number of salient functional categories statistically improved fit of the base models. Multidimensional scaling results showed scale content features interacting with response styles and functional categories. Translation of rating scales requires adapting scale features to characteristics of target languages, such as salient response styles and respondents' functional categories of judgment.     

APPLICATION OF CONTENT VALIDITY METHODS TO THE DEVELOPMENT OF A JOB-RELATED PERFORMANCE RATING CRITERION

This study demonstrated the use of quantitative content validity procedures in the development of a job-related behavioral rating scale criterion for entry-level psychiatric aides. Work behavior items were developed by staff from 6 state psychiatric hospitals, placed in a content validity questionnaire using the Lawshe format, and given to a representative sample of 38 aides and supervisors. Seventy-eight of 83 items were found to be significantly job-relevant using the computation procedures of both Lawshe and Aiken. After the significant items were grouped into 4 categories with high interjudge agreement and placed in a rating scale format, ratings were obtained on 72 psychiatric aides from 4 hospitals. Items in the 4 categories were found to be internally consistent using coefficient alpha. Significant but low concurrent validities were established for 2 verbal ability selection tests using the rating criterion. The validities found were interpreted to be especially significant when the factors of low selection ratio, restriction in range, and limited rater training were considered.     

The Multiplicative AHP,SMART and ELECTRE in a Common Context

This paper presents a comparative study of three popular methods for multicriteria decision analysis based on a particular model of human preferential judgement. Since decisions are invariably made within a given context, we model relative preferences as ratios of increments or decrements in an interval on an axis of desirability. Next we sort the ratio magnitudes into a small number of categories, represented by numerical values on a geometric scale. We explain why the analytic hierarchy process (AHP) and the French collection of ELECTRE methods, typically based on pairwise comparison methods, are concerned with categories of ratio magnitudes, whereas the simple multiattribute rating technique (SMART) essentially uses orders of magnitude of these ratios. This phenomenon provides a common basis for the analysis of the methods in question and for a cross-validation of their results. We illustrate the approach via a well-known case study, the choice of a location for a nuclear power plant. We conclude by discussing the scope of the comparative study. © 1997 John Wiley & Sons, Ltd.     

Effects of varying numbers of Likert scale points on factor structure of the Rosenberg Self‐Esteem Scale

Likert‐type rating scales are among the most widely used tools in psychological research. Different numbers of response categories would likely affect response style, data distribution, reliability, and construct validity. There is a lack of research in factor structure invariance under Likert scales with different numbers of categories. The purpose of this study is to examine the effects of varying numbers of Likert points (4–11) on scale properties such as factor structure, external validity, and latent means based on the Rosenberg Self‐Esteem Scale (M. Rosenberg, 1989 ). The sample consists of 1,807 students from secondary schools in Macau. Confirmatory factor analysis shows that the correlated two‐factor model is the most appropriate one; longitudinal invariance analysis reveals that measurement invariance across Likert scales was satisfied at the scalar level. In addition, latent mean scores on the two factors as well as observed means on the subscales are comparable across Likert scales. Moreover, the measurement model across Likert scales exhibit similar external validity. Although psychometric properties are mostly similar among a different number of points, the 4‐point Likert scale is not recommended for its higher skewness and lower loadings; the 11‐point Likert scale from 0 to 10 is slightly preferred for its higher loadings and composite reliability.     

The Marital Communication Rating Schedule: An instrument for clinical assessment

The Marital Communication Rating Schedule (MCRaS) is presented as an observationally based clinical rating system for assessing verbal behavior in marital communication. Data from 35 response display discussions lasting from 20 to 30 min each, which took place between 11 married couples, were used to examine aspects of the reliability and validity of the instrument. Three raters made independent ratings of 37 MCRaS categories for each husband and wife for each discussion period. Reliability among the raters was shown to be high when calculated within one scale point. Concurrent validity was assessed by comparing MCRaS ratings for four categories with observationally based validation criteria independently coded and measured. Results indicated that for three categories — negative statements, overgeneralizations, and amount of talk — ratings produced results that were similar to those yielded by laborious coding of audiotapes. For one category, opinions requested, a relationship between the ratings and coded data was not found. The validation results were discussed in terms of possible differences in the basis of ratings for the categories subjected to validation. Although further research is needed, it was concluded that MCRaS has many of the desirable qualities needed in a clinically useful, observationally based rating system.This investigation was conducted in connection with the Sociobehavioral Research Project at The University of Michigan when Joyce Borkin and Claude L. Walter were affiliated with the project.     

Effects of the range and frequency of vibrations on the momentary riding comfort evaluation of a railway vehicle

When trains pass level-crossings, turnouts, and rail joints, they are momentarily subjected to extreme vibrations. In railway engineering, evaluation of the riding comfort under such occasional vibrations is called the momentary riding comfort evaluation, as distinct from the long-term evaluation, which addresses the riding comfort of passengers for certain lengths of train operation. In order to identify the effective vibrational characteristics of the momentary evaluation, an experiment was performed with a riding comfort simulator. Ten adult subjects for each condition, 80 in total, participated in the experiment. The effects of differences in the range of stimuli, frequency of each stimulus, and scores on a rating scale of discomfort were studied. Differences in the range and frequency affected the evaluation such that subjects tended to make a relative judgment on discomfort. They made almost an absolute judgment when the rating scale was well defined, with a small number of categories.     

The M5-PS-35: a five-factor personality questionnaire for preschool children

The Five-factor theory of personality (FFT) has pervaded personality research in recent years. Although many reliable and valid measurement instruments exist for use with adults, adolescents, and even elementary-age children, there is a lack of available 5-factor measurement tools for use with preschool children. This article expands on previous work developing the M5-PS, a rating form for preschool children designed to be completed by classroom teachers or caregivers. A total of 621 children were rated by their teachers on the 90-item working form of the M5-PS. Through a combination of empirical and rational scale refinement methods, the number of items has been reduced to 35, yielding a revised instrument, the M5-PS-35, with substantially improved construct validity and scale internal consistency. Potential changes in external validity were evaluated by comparative reanalysis of an existing data set.     

Innovations in Assessing ADHD: Development, Psychometric Properties, and Factor Structure of the ADHD Symptoms Rating Scale (ADHD-SRS)

This research involved the development of a behavior rating scale designed to measure ADHD and the investigation of the scale's psychometric properties and factor structure. This scale, the ADHD Symptoms Rating Scale (ADHD-SRS), was developed for the assessment of ADHD in the school-age (K–12) population. Participants were 1006 children and adolescents (in grades K–12) who were rated by their parents and/or teachers. The results indicate that the ADHD-SRS possesses strong internal consistency reliability and test–retest reliability and moderate cross-informant reliability. The data also suggest that the ADHD-SRS has strong content validity. Convergent validity of this instrument was also high, as demonstrated by correlations with three previously validated behavior rating scales. Significant age and gender differences in ADHD symptoms were found with both the parent and teacher respondent populations. Finally, the factor analysis of the ADHD-SRS suggested a two factor oblique rotation as the best fit for both the parent and the teacher data. After a visual inspection of the items which loaded on each factor, Factor 1 was named Hyperactive-Impulsive and Factor 2 was named Inattention. These two factors, along with the items which loaded on each factor, appear to be remarkably similar to the two categories listed in the DSM-IV for ADHD. Directions for future research, as well as clinical implications and limitations of the research are discussed.     

Rating Scales as Predictors—The Old Question of Scale Level and Some Answers

Rating scales as predictors in regression models are typically treated as metrically scaled variables or, alternatively, are coded in dummy variables. The first approach implies a scale level that is not justified, the latter approach results in a large number of parameters to be estimated. Therefore, when rating scales are dummy-coded, applications are often restricted to the use of a few predictors. The penalization approach advocated here takes the scale level serious by using only the ordering of categories but is shown to work in the high dimensional case. We consider the proper modeling of rating scales as predictors and selection procedures by using penalization methods that are tailored to ordinal predictors. In addition to the selection of predictors, the clustering of categories is investigated. Existing methodology is extended to the wider class of generalized linear models. Moreover, higher order differences that allow shrinkage towards a polynomial as well as monotonicity constraints and alternative penalties are introduced. The proposed penalization approaches are illustrated by use of the Motivational States Questionnaire.     

Teachers' ratings of gross motor skills suffer from low concurrent validity

In this study an attempt was made to construct a reliable and valid unifactorial teachers' rating scale for gross motor ability. Study 1 (132 children from 3 to 7 years) revealed that reliability of the scale was acceptable and that the scale represented an unifactorial dimension. Two studies on concurrent validity of the scale with an experimental gross motor task (stepping-stone crossing), the unifactorial subtest Locomotion of the Test of Gross Motor Development and the subtest Balance of the Movement Assessment Battery for Children as criterion measures, did not produce acceptable validity coefficients. In both validation studies an age effect was found. It was concluded that factor specificity does not seem the answer to the usual low validity coefficients of multifactorial teachers' rating scales. An alternative approach is suggested in which the assessment of functional activities in daily situations is stressed. Finally, the inclusion of atypical groups in random samples, which is common practice in research on concurrent validity of screening instruments for children's motor problems, is discussed.     

An extension of the rating scale model with an application to the measurement of change

The polytomous unidimensional Rasch model with equidistant scoring, also known as the rating scale model, is extended in such a way that the item parameters are linearly decomposed into certain basic parameters. The extended model is denoted as the linear rating scale model (LRSM). A conditional maximum likelihood estimation procedure and a likelihood-ratio test of hypotheses within the framework of the LRSM are presented. Since the LRSM is a generalization of both the dichotomous Rasch model and the rating scale model, the present algorithm is suited for conditional maximum likelihood estimation in these submodels as well. The practicality of the conditional method is demonstrated by means of a dichotomous Rasch example with 100 items, of a rating scale example with 30 items and 5 categories, and in the light of an empirical application to the measurement of treatment effects in a clinical study.Work supported in part by the Fonds zur Förderung der Wissenschaftlichen Forschung under Grant No. P6414.     

反应风格的测量与统计控制

反应风格是共同方法偏差的主要来源之一。本文首先讨论反应风格的定义和类型,梳理其危害,认为反应风格能使测验分数出现偏差,影响测验信效度分析和变量关系分析,有必要控制其危害。然后介绍了常用的反应风格测量方法,包括计数法和模型法两大类,对测量方法的选择给出了建议,在此基础上,就如何结合反应风格的测量方法与残差回归法、偏相关法来控制反应风格危害给出建议。     

反应风格的测量与统计控制

反应风格是共同方法偏差的主要来源之一。本文首先讨论反应风格的定义和类型,梳理其危害,认为反应风格能使测验分数出现偏差,影响测验信效度分析和变量关系分析,有必要控制其危害。然后介绍了常用的反应风格测量方法,包括计数法和模型法两大类,对测量方法的选择给出了建议,在此基础上,就如何结合反应风格的测量方法与残差回归法、偏相关法来控制反应风格危害给出建议。     

Operating characteristic analysis of attribute ratings

The analysis of receiver operating characteristics as employed in psychophysics is suggested as a way of obtaining several useful measures in the context of attribute ratings. These include the difference between two stimuli on the attribute, the tendency for Ss to favor one pole of the rating scale, width of rating categories, and equal interval properties of the rating scale. The underlying measurement model is described along with means of evaluating its basic assumptions.     

认知负荷主观评价量表比较

运用双任务实验范式,比较了三种认知负荷主观评价量表的灵敏度与效度。结果发现：在本研究任务条件下,次任务反应时的稳定性、抗干扰性较好,可以作为认知负荷主观评价的标尺;WP量表与PAAS量表的敏感性均较好,其中WP量表的敏感性、诊断性高于后者,TLX量表的敏感性较弱;WP量表与PAAS量表的效度较好,且好于TLX量表。综合各项指标,在中低难度任务下,WP量表是目前认知负荷较为理想的测量工具。     

A Systematic Review and Psychometric Evaluation of Adaptive Behavior Scales and Recommendations for Practice

Adaptive behavior scales are vital in assessing children and adolescents who experience a range of disabling conditions in school settings. This article presents the results of an evaluation of the design characteristics, norming, scale characteristics, reliability and validity evidence, and bias identification studies supporting 14 norm-referenced, informant-based interviews and rating scales designed to measure adaptive behaviors. To derive these results, the manuals for each of these scales were reviewed using a standardized coding procedure, and information about each scale was double-coded by reviewers. Findings reveal that several evidence-based adaptive behavior scales are available to school psychologists. Concluding recommendations address selection and use of adaptive behavior scales as part of a comprehensive assessment, using the optimal methods of administration of adaptive behavior scales, and interpreting resultant scores that have demonstrated the highest levels of reliability and the largest body of validity evidence.     

The reliability and validity of a system for family assessment

The Darlington Family Assessment System (DFAS) is based upon the principles of multisystem-multimethod (MSMM) assessment. In practice it consists of a structured family interview with an integrated rating scale, a number of self-report questionnaries, and a task with an integrated behaviour coding system. This article summarizes the results of a series of empirical evaluations of the DFAS, which concern evaluations of the system as an aid to clinical work with families and as a method for training (at a basic level) in family assessment. The results are presented in terms of their implications for the reliability and validity of the assessment system and discussed from the perspective of their generalizability.     

基于行为事件的履历资料评估

行为事件的履历资料评估能有效克服传统履历数据构思效度弱、情景限制多等问题。本研究结合某通信企业招聘工作,选取250名应聘者数据,结合关键事件法,通过对履历事件的行为锚定来评估应聘者的胜任特征。研究在多重比较行为履历资料、履历表数据、认知能力等多个指标组合对录用结果预测的回归模型基础上,检验新增指标的预测效度。结果表明,行为事件的履历资料评估具有效标关联效度,与其他工具组合使用时具有增量效度。