非词复述测验的编制和初步分析

本研究编制了一种用于测量儿童语音工作记忆的测验———非词复述测验,并在48名四年级小学生中初步进行信度、效度检验和项目分析。结果表明:(1)该测验有较高的重测信度;(2)该测验具有较高的结构效度和效标效度;(3)分测验二的项目难度分布比较合理,多数项目鉴别力较高,而分测验一的项目难度分布和项目鉴别力则有待于在今后的研究中进一步提高。     

Examination of the Psychometric Properties of the Spiritual Fitness Assessment

Using a sample of 196 undergraduate students, the present study investigated the psychometric properties of the Spiritual Fitness Assessment, a measure of spiritual fitness designed for use by health and fitness professionals. Examination of inter-item consistency produced satisfactory alpha coefficients for the total test and its three subscales. Correlations of the SFA with measures of spirituality and spiritual well-being provide support for convergent validity. Significant positive correlations with a measure of self-esteem suggest that the test has satisfactory criterion validity. Correlations with age, sex, and a measure of social desirability indicate that the SFA is significantly, though only moderately, affected by demographic variables and motivated response tendencies. Factor analysis of the SFA items indicate that the test is factorially complex and subscales may need to be refined to better measure their intended constructs. Overall, the results of the study suggest that the test may be useful for evaluations of spiritual fitness. Further research with samples drawn from different populations is needed on the test to better establish its reliability and validity.     

基于关系-表征复杂性模型的数学应用题表征能力测验

基于关系-表征复杂性模型,从每道应用题涉及集合关系的嵌套程度角度事前分析其关系复杂性,编制了难度序列变化的应用题测验,以考察问题表征能力。采用该测验测查了四至七年级共165名学生,考察事前分析的合理性及表征水平随年龄的变化。结果表明:(1)事前分析对两个事后难度指标(错误率和Rasch模型分析的任务难度)的解释率分别为73.7%、78.7%;该测验得分与测查思维水平层次变化的SOLO分类测验上的得分有较高相关(r=0.65)。(2)四年级的应用题表征水平显著低于五、六、七年级,其他三个年级差异不显著;而且随着问题关系复杂性的增加,年级差异增大。这说明基于关系-表征复杂性模型的事前分析是合理的,据此编制的测验能够测查表征水平随年龄的变化。     

儿童图形表征能力测验编制的初步报告

表征能力发展部分地体现为对事物中隐含关系的理解或推理的复杂程度.基于辛自强的"关系-表征复杂性模型",结合Halford的"关系复杂性"理论,通过事前分析任务复杂性确定了难度不同的项目,最终编制成儿童图形表征能力测验,该测验共50个(25类)项目.以265名小学3~6年级儿童为被试,采用计算机呈现方式施测了该测验.结果表明:项目的难度和区分度以及整个测验的重测信度、构想效度、效标关联效度均良好;基于理论的事前任务分析(确定任务的维度数与每个维度内的元素数)对事后任务难度的预测率较高(83.4%).可见,所编制的图形测验能有效测查儿童表征水平.     

The positive and negative semantic dimensions of relationship satisfaction

Semantic differential items were reconfigured to assess relationship satisfaction across separate positive and negative attitude dimensions. Study 1 (N = 1,656) supported a 2‐factor model for the Positive and Negative Semantic Differential (PN–SMD), as well as its convergent, criterion‐related, and incremental validity over the 16‐item Couples Satisfaction Index (CSI; J. L. Funk & R. D. Rogge, 2007) using known correlates of relationship satisfaction as criteria. Study 2 (N = 89) replicated the convergent, criterion‐related, and incremental validity findings of Study 1 using different criterion measures, the CSI, a bipolar semantic differential measure designed for assessing relationship satisfaction, and an existing 2‐dimensional measure of relationship satisfaction. The authors demonstrated across studies that the PN–SMD captures criterion‐relevant information about ambivalence versus indifference toward the relationship—associations that are only detectable when using a 2‐dimensional satisfaction measure.     

JOB-RELATEDNESS OF A POSTTRAINING JOB KNOWLEDGE CRITERION USE TO ASSESS VALIDITY AND TEST FAIRNESS

Significant job-relatedness was found for a posttraining job knowledge test criterion using an application of Lawshe's content validity method. The aide test was used as a criterion to assess the predictive validity of a vocabulary test and a civil service test with samples of black ( N = 43) and white ( N = 62) psychiatric aides. Significant validities were found on both tests, but a vocabulary test proved to be the better predictor of the criterion in both samples. The obtained validities were discussed in terms of differential validity, test fairness, and sample size. This study demonstrated that a content validity method could be applied to criteria as well as selection tests. It was concluded that content validity methods may be able to help solve the problem of criterion relevance in validation research by providing quantitative evidence of the job-relatedness of criteria.     

Illustration of Multilevel Explanatory IRT Model DIF Testing with the Creative Thinking Scale

Creativity has been well studied in the past several decades, and numerous measures have been developed to assess creativity. However, validity evidence associated with each measure is often mixed. In particular, the social consequence aspect of validity has received little attention. This is partly due to the difficulty of testing for differential item functioning (DIF) within the traditional classical test theory framework, which still remains the most popular approach to assessing creativity. Hence, this study provides an example of examining differential item functioning using multilevel explanatory item response theory models. The Creative Thinking Scale was tested for DIF in a sample of 1043 10th–12th graders. Results revealed significant uniform and non-uniform DIF for some items. Differentially functioning items are able to produce measurement bias and should be either deleted or modeled. The detailed implications for researchers and practitioners are discussed.     

Examination of ethnic group differential responding on a biodata instrument

Biodata is a selection device that is often cited for high criterion validity and low adverse impact, yet little research examines the latter point. Further, little is known about whether differential biodata item responding by ethnic group occurs, and if so, whether this impacts ethnic group differences, predictive bias, adverse impact, and criterion validity. Data from 5,242 applicants for a government agency position indicated that elimination of biodata item response options with differential ethnic group selection rates from scoring consideration resulted in a large decrease in ethnic group differences, no predictive bias, less potential adverse impact, and a nonsignificant decrease in criterion validity. Results are compared to a general mental ability measure. Implications and future research directions are discussed.     

Subject Matter Expert Judgments Regarding the Relative Importance of Competencies are not Useful for Choosing the Test Batteries that Best Predict Performance

Several recent articles have suggested that assessments of the relative importance of different abilities or competencies to a job have little bearing on the criterion‐related validity of these selection tests that measure those abilities. We hypothesize that selection test batteries chosen to maximize the judged importance of knowledge, skills, and abilities will not predict performance better than batteries of tests chosen at random. The results in two independent samples consistently show that the validity of test batteries chosen based on subject matter expert judgments of importance is not different from the validity of batteries of a comparable number of tests chosen at random from a set of intercorrelated tests, or even those chosen to provide the worst possible match between test content and job content.     

Emotional Intelligence in Personnel Selection: Applicant reactions,criterion, and incremental validity

In two studies, based on four samples and more than 600 participants, the authors examined applicant reactions, criterion and incremental validity, and differential prediction of emotional intelligence (EI) in personnel selection using the Mayer–Salovey–Caruso Emotional Intelligence Test. The first study examined applicant reactions in terms of face validity and fairness evaluation of the EI test. The second study examined across three samples (salespeople, front‐desk public employees, and hospital chief executive officers) the criterion validity, incremental validity, and differential prediction of EI. Results suggest that the EI test is perceived as a fair selection tool, and is predictive of performance. The EI test has incremental validity over cognitive abilities and personality (big five) when predicting subjective and objective performance criteria. Based on these results, the authors encourage further research on the use of EI in selection settings.     

An Analysis of the Convergent and Discriminant Validity of the Transformational Leadership Questionnaire

This article sets out to provide evidence of the convergent and discriminant validity of a recently developed leadership questionnaire to support the assessment and development of leadership in organizations, the Transformational Leadership Questionnaire (TLQ). The TLQ was developed on the basis of eliciting constructs of leadership from managers working at different levels (executive, top, senior, middle) in two large parts of the UK public sector, using a Grounded Theory approach. Evidence is presented that eight scales which measure different aspects of transformational leadership are significantly correlated with each of five criterion variables, even when the instrument is used in a public sector different from the one in which it was developed. Discriminant analysis indicated, however, that the relationship between the scales and the criterion variables is not consistent for the different groups and sub-groups of managers, divided by level, by sex, and by level x sex. These latter data were interpreted, in the light of the ‘range of convenience’ corollary of Kelly's Personal Construct Theory, and of the concept of ‘substitutes for leadership’, as evidence that certain TLQ scales have particular relevance to certain groups and sub-groups of managers, and thus of the discriminant validity of the instrument, and further evidence of its construct validity.     

Variation in test validity with variation in the distribution of item difficulties,number of items,and degree of their intercorrelation

The relation between item difficulty distributions and the validity and reliability of tests is computed through use of normal correlation surfaces for varying numbers of items and varying degrees of item intercorrelations. Optimal or near optimal item difficulty distributions are thus identified for various possible item difficulty distributions. The results indicate that, if a test is of conventional length, is homogeneous as to content, and has a symmetrical distribution of item difficulties, correlation with a normally distributed perfect measure of the attribute common to the items does not vary appreciably with variation in the item difficulty distribution. Greater variation was evident in correlation with a second duplicate test (reliability). The general implications of these findings and their particular significance for evaluating techniques aimed at increasing reliability are considered.     

Test anxiety and the validity of cognitive tests: A confirmatory factor analysis perspective and some empirical findings

The validity of cognitive ability tests is often interpreted solely as a function of the cognitive abilities that these tests are supposed to measure, but other factors may be at play. The effects of test anxiety on the criterion related validity (CRV) of tests was the topic of a recent study by Reeve, Heggestad, and Lievens (2009) (Reeve, C. L., Heggestad, E. D., & Lievens, F. (2009). Modeling the impact of test anxiety and test familiarity on the criterion-related validity of cognitive ability tests. Intelligence, 37, 34?41.). They proposed a model on the basis of classical test theory, and concluded on the basis of data simulations that test anxiety typically decreases the CRV. In this paper, we view the effects of test anxiety on cognitive ability test scores and its implications for validity coefficients from the perspective of confirmatory factor analysis. We argue that CRV will be increased above the effect of targeted constructs if test anxiety affects both predictor and criterion performance. This prediction is tested empirically by considering convergent validity of subtests in five experimental studies of the effect of stereotype threat on test performance. Results show that the effects of test anxiety on cognitive test performance may actually enhance the validity of tests.     

Validity of the decisional process inventory

Empirical research on career indecision has been criticized for lacking theoretical conceptualization. To address this problem we examined the validity of the decisional process inventory (DPI), a measure of career decision making derived from a model based in Gestalt psychotherapy theory. Undergraduate college students (91 women, 110 men) responded to the DPI and criterion measures of career indecision and vocational identity. Results supported hypothesized relations between and among DPI items and subscales, and the ability of the measure to differentiate individuals by expressed career choice status. Significant intercorrelations among DPI subscales and criterion measures supported the concurrent validity of the DPI. Confirmatory maximum likelihood factor analysis of the DPI items did not support the hypothesized structure of the DPI to measure three career decision-making process dimensions: readiness, action, and resistance. Although results supported the basic psychometric properties of the DPI as a brief measure of career decision making, additional research is needed to clarify the underlying factor structure of the instrument.     

VALIDATING SYNTHETIC VALIDATION: COMPARING TRADITIONAL AND SYNTHETIC VALIDITY COEFFICIENTS

We describe a unique application of a synthetic validation technique to a selection system development project in a large organization. Job analysis data were collected from 4,725 job incumbents and 619 supervisors, and were used to identify 11 job families and 27 job components. We developed 12 tests to predict performance on these job components and conducted a concurrent validation study to collect test and job component data for 1,926 incumbents. We created a test composite for each job component and then chose a test battery for each job family based on its relevant job components. Synthetic validity coefficients were computed for each test battery and compared to traditional validity coefficients that were computed within job families with large sample sizes. The synthetic validity coefficient was very close to the within-family validity coefficient for most job families and was within the bounds of sampling error for all job families. Validities tended to be highest when predictors were weighted according to the number of job components to which they were relevant and job component criterion measures were unit weighted.     

Longitudinal Predictive Validity of a Learning Potential Test

The aim of this article is to provide empirical psychometric evidence of the (longitudinal) predictive validity of a learning potential measure—the Learning Potential Computerised Adaptive Test (LPCAT)—in comparison with standard static tests with school aggregate results as the criterion measure. Participants were 79 boys (mean age 12.44, SD = 0.44) and 72 girls (mean age 11.18, SD = 0.42) attending two private schools. Correlation and regression analyses were used to evaluate the predictive validity of the learning potential and standard test scores for school aggregate academic results as criterion measure. Results indicate that learning potential scores were statistically significant predictors of aggregate academic results and provided results that were comparable to those of the standard test results—providing empirical support for the use of learning potential tests in mainstream educational settings.     

Examining the validity of self‐reports on scales measuring students' strategic processing

Background. Self‐report inventories trying to measure strategic processing at a global level have been much used in both basic and applied research. However, the validity of global strategy scores is open to question because such inventories assess strategy perceptions outside the context of specific task performance. Aims. The primary aim was to examine the criterion‐related and construct validity of the global strategy data obtained with the Cross‐Curricular Competencies (CCC) scale. Additionally, we wanted to compare the validity of these data with the validity of data obtained with a task‐specific self‐report inventory focusing on the same types of strategies. Sample. The sample included 269 10th‐grade students from 12 different junior high schools. Methods. Global strategy use as assessed with the CCC was compared with task‐specific strategy use reported in three different reading situations. Moreover, relationships between scores on the CCC and scores on measures of text comprehension were examined and compared with relationships between scores on the task‐specific strategy measure and the same comprehension measures. Results. The comparison between the CCC strategy scores and the task‐specific strategy scores suggested only modest criterion‐related validity for the data obtained with the global strategy inventory. The CCC strategy scores were also not related to the text comprehension measures, indicating poor construct validity. In contrast, the task‐specific strategy scores were positively related to the comprehension measures, indicating good construct validity. Conclusion. Attempts to measure strategic processing at a global level seem to have limited validity and utility.     

Factorial validity of the 20-item Toronto Alexithymia Scale

The 20-item Toronto Alexithymia Scale (TAS-20) was developed in previous research to measure a general dimension of alexithymia with three inter correlated factors. These three factors reflect distinct facets of the alexithymia construct: (1) difficulty identifying feelings and distinguishing them from the bodily sensations of emotion, (2) difficulty describing feelings to others, and (3) an externally orientated style of thinking. This study tested the three-factor model for the TAS-20, using confirmatory factor analysis, in separate samples of young adults from Germany, Canada, and the United States. The previously established three-factor model was found to be replicable in all three samples. In addition, the full TAS-20 and its three factors demonstrated adequate internal reliability in all three samples. Although evaluation of the convergent, discriminant, and criterion validity of the TAS-20 is required in diverse cultural groups, the present results provide evidence for the factorial validity and internal reliability of the TAS-20.     

Supervisory opportunity to observe moderates criterion‐related validity estimates

Supervisors' opportunity to observe incumbents' performance (i.e., how often a supervisor typically sees an employee's performance) has been suggested to be important for accurate performance rating and to be a moderator of criterion‐related validity. Here we test these suggestions and present empirical evidence for the effects of opportunity to observe. In Study 1, supervisors in a multi‐occupation/organization criterion‐related validation study for a biodata measure indicated the opportunity they had to observe incumbents. The data were split according to different levels of opportunity to observe. Higher validities were found when opportunity to observe was maximal. In Study 2, this finding was replicated using a cognitive ability test. These results suggest that psychologists should consider measuring opportunity to observe in criterion‐related validity studies.     

Toward a process-focused model of test score validity: improving psychological assessment in science and practice

Although definitions of validity have evolved considerably since L. J. Cronbach and P. E. Meehl's classic (1955) review, contemporary validity research continues to emphasize correlational analyses assessing predictor-criterion relationships, with most outcome criteria being self-reports. The present article describes an alternative way of operationalizing validity--the process-focused (PF) model. The PF model conceptualizes validity as the degree to which respondents can be shown to engage in a predictable set of psychological processes during testing, with those processes dictated a priori by the nature of the instrument(s) used and the context in which testing takes place. In contrast to the traditional approach wherein correlational methods are used to quantify the relationship between test score and criterion, the PF model uses experimental methods to manipulate variables that moderate test score-criterion relationships, enabling researchers to draw more definitive conclusions regarding the impact of underlying psychological processes on test scores. By complementing outcome-based validity assessment with a process-driven approach, researchers will not only improve psychology's assessment procedures but also enhance their understanding of test bias and test score misuse by illuminating the intra- and interpersonal factors that lead to differential performance (and differential prediction) in different groups.