Do overall dimension ratings from assessment centres show external construct-related validity?

ABSTRACT

There have been repeated calls for an external construct validation approach to advance our understanding of the construct-related validity of assessment centre dimension ratings beyond existing internal construct-related validity findings. Following an external construct validation approach, we examined whether linking assessment centre overall dimension ratings to ratings of the same dimensions that stem from sources external to the assessment centre provides evidence for construct-related validity of assessment centre ratings. We used data from one laboratory assessment centre sample and two field samples. External ratings of the same dimensions stemmed from assessees, assessees’ supervisors, and customers. Results converged across all three samples and showed that different dimension-same source correlations within the assessment centres were larger than same dimension-different source correlations. Moreover, confirmatory factor analyses revealed source factors but no dimension factors in the latent factor structure of overall dimension ratings from the assessment centre and from external sources. Hence, consistent results across the three samples provide no support that assessment centre overall dimension ratings and ratings of the same dimensions from other sources can be attributed to dimension factors. This questions arguments that assessment centre overall dimension ratings should have construct-related validity.     

Predicting management career success from assessment center data: A longitudinal study

The authors investigated trends over a 13-year period in the predictive validity of assessment center data for objective career advancement for a sample of 456 academic graduates. Using year of entry and tenure as controls, findings from research into managerial effectiveness and development were confirmed with respect to the importance of interpersonal effectiveness, firmness, and ambition for long-term career success. Within the long-tenured group the validity of the overall assessment rating (OAR) corrected for initial differences in starting salaries and for restriction in range, was 0.35. Substantial long-term validities were obtained for ratings from the assessment center exercises. Computation of validities after three to five years of tenure of the same cohort in addition revealed some dynamic validity patterns. For instance, the OAR was important for early and late career success only for the group that was still present after 12 years, and its validity increased over time.     

Work behavior in the school psychology service: Conceptual framework and construct validity approached by two different methodologies

A conceptual framework for work behavior (WB) in the school psychology service (SPS) in Norway is discussed, focusing on differences at the individual and the systemic level. A four-factor measurement model of selected tasks was hypothesized and empirically assessed by joint application of generalizability (G) theory and confirmatory factor analysis (CFA). Self-reported WB was investigated among 190 SP counselors in Study 1 and 470 SP counselors within 121 offices in Study 2. The hypothesized four-factor solution was supported. In accordance with international findings, G studies demonstrated that the SPS as a whole allocates different priorities to aspects of WB, in favor of traditional child-centered work when approaching existing problems. A two-level model of factor analysis was tested, indicating little difference between offices in service delivery. On the contrary, the generalizable differentiation between the four hypothesized WB categories as provided by the G framework indicated individual differences among SP counselors in their priorities to aspects of WB. Because higher priority to prevention and systems intervention have been requested in general, future research should explore possible variables that may have an effect on SP counselors choice of different aspects of WB, as well as differences in the associations between aspects of WB on the one hand and important behavioral and attitudinal outcomes among jobholders on the other.     

评价中心测评的评分误差分析研究

本研究应用概化理论对评价中心中的评分误差控制问题进行了系统的分析探讨,得出了以下研究结论：(1)总体上,在评价中心(以LGD为例)测评中,评价员对研究设计的测评维度的评分基本反映了被试的真实能力水平,测量误差较小;(2)评价员对合作能力、应变能力和总体印象等维度的评价标准的理解具有较高的一致性,但对决策能力等维度的评价标准的理解存在一定的偏差;(3)在评价中心(以LGD为例)测评中,采用4名评价员能达到预期的测量目标,符合测评应用的经济性、有效性原则。     

档案袋评价中评分者信度的实证研究

档案袋评价因能充分发挥促进学生发展和教学改进的功能而受到青睐,但不佳的测评信度和效度限制了其在教学评价中的应用。本文对档案袋评分者信度的特点进行实证研究,4位评分者对152份档案袋进行了2次等级评分,运用多种统计方法计算评分者信度。结果表明,档案袋的评分有较高的关联性、中等偏弱的一致性和一定的稳定性,对档案袋整体水平的评分信度最高。本研究中,评分者个数为3时,对档案袋整体水平评分的概化系数和可靠性系数都在0.80以上。     

Identifying correlates of self-stigma in adults who stutter: Further establishing the construct validity of the Self-Stigma of Stuttering Scale (4S)

PurposeThis study was set up to further establish the construct validity of the Self-Stigma of Stuttering Scale (4S) by demonstrating its associations with other established scales and replicating its original factor structure and reliability estimates.MethodWeb surveys were completed by 354 adults who stutter recruited from Board Certified Specialists in Fluency Disorders, and adult chapters of the National Stuttering Association. Participants completed a series of psychometrically validated scales measuring self-stigma, hope, empowerment, quality of life, social support, anxiety, depression, and self-rated speech disruption.ResultsHigher subscale and total stigma scores on the 4S were associated with significantly lower levels of hope, empowerment, quality of life, and social support, and significantly higher levels of anxiety, depression, and self-rated speech disruption. The original factor structure of the 4S was replicated, and reliability estimates of the subscales ranged from adequate to excellent.ConclusionsThe findings of this study support the construct validity of the 4S and its use by clinicians and researchers intending to measure the construct of self-stigma in adults who stutter.Educational objectives: Readers should be able to: (a) distinguish between the various components of self-stigma; (b) describe how the various components of the self-stigma model relate to hope, empowerment, quality of life, and social support, self-rated speech disruption, anxiety, and depression; (c) summarize the psychometric properties of the Self-Stigma of Stuttering Scale (4S) in terms of reliability, factor structure, and construct validity; (d) discuss how the 4S could be used in research and clinical practice.     

Reliability, validity and normative data for the Danish Beck Youth Inventories

This study examines reliability and validity and establish Danish norms for the Danish version of the Beck Youth Inventories (BYI) ( Beck, Beck & Jolly, 2001 ), which consists of five self-report scales; Self-Concept (BSCI), Anxiety (BAI), Depression (BDI), Anger (BANI) and Disruptive Behavior (BDBI). A total of 1,116 school children and 128 clinical children, aged 7–14, completed BYI. Internal consistency coefficients were high. Most test-retest correlations were >0.70. A test-retest difference was found for BAI. Exploratory and confirmatory factor analysis indicated that the five factor structure of the instrument was justified. The BSCI, BAI and BDI discriminated moderately between the norming sample and the clinical group, and the latter group included more children who exceeded the 90th percentile of the norming sample. Diagnostic groups scored higher on relevant scales than norms. Only BSCI and BDI differentiated between diagnostic groups. The BYI showed acceptable internal consistency and test-retest stability, except for BAI. The BYI did not adequately differentiate between internalizing disorders.     

不同类型的测评维度对评价中心结构效度的影响研究

本研究将测评维度分为行为能力和心理特质两组,考察对评价中心结构效度的不同影响。A公司216名部门副经理级员工接受了本次管理素质评价中心测评,采用公文筐、角色扮演和无领导小组讨论三种方法,测查行为能力和心理特质两组维度,各自有三个维度被一种以上的方法测量。多质多法和验证性因素分析的结果表明,评价中心以行为能力比以心理特质为测评维度结构效度好;以行为能力为测评维度时,会聚效度和区分效度都较好。     

Factors which Improve the Construct Validity of Assessment Centers: A Review

This article reviews 21 studies which manipulated specific variables to determine their impact on the construct validity of assessment centers. This review shows that the studies regarding the impact of different observation, evaluation, and integration procedures yielded mixed results. Conversely, dimension factors (number, conceptual distinctiveness, and transparency), assessor factors (type of assessor and type of assessor training), and exercise factors (exercise form and use of role-players) were found to moderate construct validity. On the basis of the review, practical recommendations are derived to maximize the probability that practitioners design and administer an assessment center with construct validity. Finally, new perspectives for future research are identified.     

评价中心的评分维度和评分效果

对近年来国内外关于评价中心的研究进行了比较系统的介绍。首先,文章讨论了评分维度的数目对于评分结果的影响,以及评价中心中的4个元维度;其次,介绍了评价中心中衡量评分效果的指标,并讨论了评分者培训的分类及其对评分效果的影响;第三,虽然评价中心具有良好的效标关联效度,但对于其结构效度的研究至今尚无统一结论。最后,文章对评价中心未来的研究趋势进行了探讨。     

概化理论在绩效评估中的应用

概化理论借助其理论上的优势在很大程度上克服了基于经典测量理论绩效评估存在的缺陷。它可以对绩效评估中的信度进行更全面的估计,更好地预测和控制误差,并且概化理论的独特视角和方法还为360度绩效评估的效度求取提供了理论支持。     

A comparative analysis of the validity of several approaches to the scoring of the behavior of the preterm infant

A group of very low birthweight preterm infants (mean WEIGHT = 1193.6 g; mean gestational AGE = 31.1 weeks) were examined with the PREMIE. The PREMIE is a modification of the scoring system of the full term Brazelton Neonatal Behavioral Assessment (BNBAS) examination for use with preterm infants. Infants were assessed during the acute phase of cardio-respiratory illness (mean AGE = 7 days) and during the recovery phase (mean AGE = 22 days). Many modifications in procedure were required. Several analyses were performed to assess the redundancy and the comparative validity of the scoring systems of the PREMIE, the BNBAS, and the BNBAS + (the BNBAS plus a set of five additional preterm items). Results indicated that the three scoring systems were highly redundant, that they were equally related to physiologic and clinical variables, and that they were similar in their sensitivity to behavioral change and stability. Overall the results offered little in the way of differentiating among these three scoring systems with this group of infants, raising doubts about the need to use the PREMIE or other scoring systems like it for many research purposes.     

Adolescent attachment in Nepal: Testing the factorial validity of two scales

Attachment theory is an important framework in the psychology of human development and has direct relevance to the study of adolescence. The cross-cultural validity of attachment constructs and measures has been the subject of lively debate among experts. Using confirmatory factor analysis, the present study tested the factor structure of the Adolescent Attachment Questionnaire and the Adolescent Unresolved Attachment Questionnaire in a sample of 279 Nepali adolescents. The hypothesised models had a good fit and further tests established the measurement invariance of the two instruments. The cross-cultural validity of the measures was supported but areas of cultural variation were also highlighted.     

Individual differences in processing styles: validity of the Rational-Experiential Inventory

In Study 1 (N= 203) the factor structure of a Swedish translation of Pacini and Epstein's Rational-Experiential Inventory (REI-40) was investigated using confirmatory factor analysis. The hypothesized model with rationality and experientiality as orthogonal factors had satisfactory fit to the data, significantly better than alternative models (with two correlated factors or a single factor). Inclusion of "ability" and "favorability" subscales for rationality and experientiality increased fit further. It was concluded that the structural validity of the REI is adequate. In Study 2 (N= 72) the REI-factors were shown to have theoretically meaningful correlations to other personality traits, indicating convergent and discriminant validity. Finally, scores on the rationality scale were negatively related to risky choice framing effects in Kahneman and Tversky's Asian disease task, indicating concurrent validity. On the basis of these findings it was concluded that the test has satisfactory psychometric properties.     

Researchers discover what they already knew about validity: The case of the family-of-Origin Scale

This article is concerned with two recnet studies of the Family-of-Origin Scale (FOS), Gavin and Wamboldt (1992) and Ryan, Kawash, Fine, and Powel (1994). The external validity analyses conducted for the FOS in these studies assumed the scale to be a unidemsional measure. The present discussion centers on methodological decisions and interpretive strategies that may have prevented a more complete understanding of the scale's dimensional structure and theoretical meaning. It is concluded that a unidimensional view of the FOS is an oversimplification that is inconsistent with the available data. A case is made for a more comprehensive and technically adequate evaluation of the FOS. Specific suggestions are offered.