Validity evidence for the situational judgment test paradigm in emotional intelligence measurement

To date, various measurement approaches have been proposed to assess emotional intelligence (EI). Recently, two new EI tests have been developed based on the situational judgment test (SJT) paradigm: the Situational Test of Emotional Understanding (STEU) and the Situational Test of Emotion Management (STEM). Initial attempts have been made to examine the construct‐related validity of these new tests; we extend these findings by placing the tests in a broad nomological network. To this end, 850 undergraduate students completed a personality inventory, a cognitive ability test, a self‐report EI test, a performance‐based EI measure, the STEU, and the STEM. The SJT‐based EI tests were not strongly correlated with personality and fluid cognitive ability. Regarding their relation with existing EI measures, the tests did not capture the same construct as self‐report EI measures, but corresponded rather to performance‐based EI measures. Overall, these results lend support for the SJT paradigm for measuring EI as an ability.     

Maximizing the benefits and minimizing the costs of repeated memory tests for older adults

Repeatedly trying to remember information can help people remember more but can also lead to inaccuracies. Two experiments examined whether the costs of repeated recall efforts can be minimized for older adults by using memory tests that require specification of the source of recalled items. Participants saw and imagined pictures and then took 3 successive recall tests in which they either indicated the source of each remembered item (source recall) or simply recalled the items without specification of their source (free recall). Results showed that recall increased systematically from Test 1 to Test 3, although the rate of increase was less marked for older adults, and older adults recalled less overall. After the free recall tests, older adults made more source misattributions (claiming to have seen imagined items) than did young adults, but after the source recall tests, age differences were not significant. Thus, repeatedly recalling items while considering their source was associated with benefits in terms of increased recall and fewer costs in terms of source errors.     

Psychometric properties and factor structure of the Fear Questionnaire phobia subscale items in two normative samples

Normative data for the Fear Questionnaire (Marks & Mathews, 1979), a popular self-report instrument measuring phobic concerns, were collected from both a community and a collegiate sample. The covariation and internal consistency of the blood-injury phobia (BI), social phobia (SO), and agoraphobia (AG) subscales were assessed in each sample, and the factor structure of the items from these three subscales was examined. Results indicated that, in general, community subjects reported more phobic concerns than did collegiate subjects, and females reported greater phobic concerns than did males. Results from confirmatory factor analyses suggest that the three factors of BI, SO, and AG did not emerge from either data set. Follow-up exploratory factor analyses did identify the general factors of blood-injury phobia and agoraphobia. Normative data from the present study are compared to those obtained previously, and directions for future research are provided.     

Smoking motivation: A factor-analytical study

Factor analyses were carried out on the responses of 115 cigarette smokers to the Smoking Questionnaire of Russell et al. (1974), to the Situational Smoking Questionnaire of Frith (1971) and to certain items of a biographical smoking questionnaire. In the analysis of Russell's questionnaire, a set of six factors was extracted, bearing some resemblance to the six factors found by the authors of the questionnaire, although the matching of factors was by no means perfect. However, the results did support Russell et al.'s finding of a major dimension of degree of dependence upon cigarettes, together with an orthogonal factor contrasting pharmacological with non-pharmacological dependence. The analysis of the Situational Smoking Questionnaire confirmed Frith's finding that the first principal component represented a general desire for cigarettes while the second contrasted desire to smoke in high- and low-arousal inducing (or stressful and relaxing) situations; and these two components corresponded fairly well with the first two components extracted from Russell's questionnaire, pharmacological dependence being associated with smoking in stressful situations.     

New paradigms for assessing emotional intelligence: theory and data

This article describes the development and validation of 2 measures of emotional intelligence (EI): the Situational Test of Emotional Understanding (STEU) and the Situational Test of Emotion Management (STEM). Study 1 (N=207 psychology students) examines multiple sources of validity evidence: relationships with EI, vocabulary, personality, and emotion-related criteria. Study 2 (N=149 white-collar volunteers) relates STEU and STEM scores to clinical symptoms, finding relationships to anxiety and stress for both tests, and to depression for the STEM. It is concluded that new performance-based approaches to test development, such as the present ones, might be useful in distinguishing between test and construct effects. Implications for expanding theory and for developing EI interventions are discussed.     

Performance factors and gender-related differences in spatial ability: Another assessment

To explore whether test-taking styles (performance factors)cancontribute to explaining gender. related differences on tests of spatial ability, 15 spatial tests were administered to three samples of subjects. On each test, number-correct scores and ratio scores (number of items solved divided by the number of items attempted) were computed. In accordance with previous research findings, the use ofratio scores significantly reduced the magnitude ofthe gender-related differences on the Mental Rotations Test. For most of the remaining tests, however, the reduction of the gender-related score difference was small. It was concluded that the difference reduction for the Mental Rotations Test was specific to the format of this test. In common spatial tests, performance factors may account for a small portion of gender-related variance, but the bulk of this variance must be attributed to other factors.     

应用项目反应理论创建图形推理测验题库

自编235个图形推理测验题目。采用铆测验等值设计,以72个联合型瑞文测验题目为铆题,对初中到大学各能力层次的1733名男性进行了测验。使用BILOG MG3.0（边际极大似然估计）对实测数据进行了分析,采用Logsitic 3参数模型。剔除数据与模型拟合不好的题目以及信息函数最大值小于0.3的题目,最终建立一个包含181道题目的题库。该题库可以用于淘汰智力较低的应征青年     

Validity of the Developmental Test of Visual-Motor Integration Supplemental Developmental Test of Visual Perception

Visual perceptual skills of school-age children are often assessed using the Supplemental Developmental Test of Visual Perception of the Developmental Test of Visual-Motor Integration. The study purpose was to consider the construct validity of this test by evaluating its scalability (interval level measurement), unidimensionality, differential item functioning, and hierarchical ordering of its items. Visual perceptual performance scores from a sample of 356 typically developing children (171 boys and 185 girls ages 5 to 11 years) were used to complete a Rasch analysis of the test. Seven items were discarded for poor fit, while none of the items exhibited differential item functioning by sex. The construct validity, scalability, hierarchical ordering, and lack of differential item functioning requirements were met by the final test version. Since 7 test items did not fit the Rasch analysis specifications, the clinical value of the test is questionable and limited.     

Estimating the validity of the guilty knowledge test from simulated experiments: the external validity of mock crime studies

This experiment was designed to examine the external validity of the standard mock-crime procedure used extensively to evaluate the validity of polygraph tests. The authors manipulated the type of mock-crime procedure (standard vs. a more realistic version) and the time of test (immediate vs. delayed) and examined their effects on the validity of the Guilty Knowledge Test (GKT) and the recall rate of the relevant items. The results indicated that only the type of mock-crime affected the 2 outcome variables. The realistic procedure was associated with a lower recall rate and weaker detection efficiency than the standard procedure. However, these effects were mediated by the type of GKT questions used. Practical implications of these results are discussed.     

Order Effects on Situational Judgment Test Items: A case of construct‐irrelevant difficulty

Situational judgment tests (SJTs) pose unique cognitive demands on test takers in that, when presented in written form, they require a great deal of reading and cognitive effort. Because of this cognitive demand, responses to test items toward the end of the test may be influenced by an order effect produced by responding to a large quantity of previous test items. This construct‐irrelevant order effect may increase measurement error and threaten the validity of SJT scores. To test this phenomenon, data were obtained from 1,089 applicants who had completed a lengthy SJT as part of a selection process for an hourly safety and surveillance job at a large international corporation. Results showed that local item dependence, item difficulty, and the rate of omitted responses all increased when items were placed toward the end. The order effect alone was not strong enough to influence subgroup mean score differences in the second half of the test. However, this effect did vary by race: African‐Americans were most strongly affected by the order effect, followed by Caucasians, in their number of omitted responses. Implications and future research of this effect for SJTs and similar types of assessments are discussed.     

Measurement of ability emotional intelligence: Results for two new tests

Emotional intelligence (EI) has attracted considerable interest amongst both individual differences researchers and those in other areas of psychology who are interested in how EI relates to criteria such as well‐being and career success. Both trait (self‐report) and ability EI measures have been developed; the focus of this paper is on ability EI. The associations of two new ability EI tests with psychometric intelligence, emotion perception, and the Mayer–Salovey–Caruso EI test (MSCEIT) were examined. The new EI tests were the Situational Test of Emotion Management (STEM) and the Situational Test of Emotional Understanding (STEU). Only the STEU and the MSCEIT Understanding Emotions branch were significantly correlated with psychometric intelligence, suggesting that only understanding emotions can be regarded as a candidate new intelligence component. These understanding emotions tests were also positively correlated with emotion perception tests, and STEM and STEU scores were positively correlated with MSCEIT total score and most branch scores. Neither the STEM nor the STEU were significantly correlated with trait EI tests, confirming the distinctness of trait and ability EI. Taking the present results as a starting‐point, approaches to the development of new ability EI tests and models of EI are suggested.     

Personality factors in the Eysenck personality questionnaire

Hierarchical factor analyses were carried out on raw scored EPQ data yielded from two samples of subjects, a Gallup adult quota sample of 1198 subjects and 406 university undergraduates. Each sample was split by sex providing six data sets for analysis. Results at the second order level clearly indicated the recovery of practically all the E, N and L items as three distinct factors, P item recovery differed among males and females. Adult female and student female subject groups did not show clear recovery of a substantial majority of P items. Although P, as a factor, did not appear recognisably in the male student sample, this was interpreted as being due either to the low subject: variable ratio and/or to the low subject number (N=171) and its resultant lack of definition of initial P item variance. The P factor appeared most strongly as a second order in the total Gallup, total student, and male Gallup sample.     

Test Anxiety Scale: reliability among Ethiopian students

Sarason's Test Anxiety Scale, translated into an Ethiopian language, was administered to 391 students in Grade 8 and to 422 students in preparatory school (Grades 11 and 12). In the first sample, 32 items loaded above the 0.3 criterion of acceptable item-remainder correlations and Cronbach alpha of .84. In the second sample, Cronbach alpha was .84 for the 34 items, but only 19 items had acceptable item-remainder correlations. The internal consistency reliabilities were comparable with those reported in the literature. However, the results of confirmatory factor analyses with extraction of four factors did not confirm the item loadings on factors as reported in the literature. Younger students (Grade 8) were found to have higher mean Test Anxiety than Grades 11 and 12 students. The Amharik version of the Test Anxiety Scale as a whole could be considered reliable and useful for Ethiopian students.     

Construction and validation of the South African version of the Fear Survey Schedule for Children: an exploratory factor analysis

The Fear Survey Schedule for Children-Revised (Ollendick, 1983) is an 80-item self-report instrument that has been used internationally to asses the number of fears and general level of fearfulness among children. Despite its widespread use, this instrument has not been adapted to the South African context. The present study addressed this gap by means of a 2-phase investigation aimed at developing a South African version of the instrument. In Phase 1, semistructured interviews were conducted with 40 children (7 to 13 years of age). Qualitative data obtained from these interviews were used to construct additional items for inclusion in the South African Fear Survey Schedule for Children-Revised. The modified scale, consisting of 97 items, was then administered to a sample of 646 children between the ages of 7 and 13 years. Further psychometric considerations resulted in the final version of the scale consisting of 74 items with high internal consistency (α=.97). The factor structure was explored by means of principal component analysis with varimax rotation and a 5-factor solution was found to provide the best conceptual fit. The factors identified were as follows: Fear of Death and Danger; Fear of the Unknown; Fear of Small Animals and Minor Threats to Self; Large Animal Fears; and Situational Fears. Differences between the South African version and the original Fear Survey Schedule for Children-Revised are noted and implications for the study of fear in South Africa and other countries are discussed.     

Developing an Essentially Unidimensional Test With Cognitively Designed Items

The purpose of this study was to demonstrate how cognitive and measurement principles can be integrated to create an essentially unidimensional test. Two studies were conducted. In Study 1, test questions were created by using the feature integration theory of attention to develop a cognitive model of performance and then manipulating complexity factors within the model. It was hypothesized that the complexity factors predict item difficulty. Results indicated that some complexity factors predicted difficulty in a relatively small sample. In Study 2, items developed using the cognitive model were integrated with items measuring another factor to create a multidimensional test of spatial reasoning. Results were replicated in Study 2 with a sample of 460 participants. The test met the assumption of essential unidimensionality according to DIMTEST, was moderately correlated (r = .64) with the Bennett Mechanical Comprehension Test, and showed little evidence of differential item functioning. Implications are discussed.     

LINGUISTIC AWARENESS AND EARLY READING

Children from five Year 1 classes (n = 129) were assessed with the Concepts About Print (CAP) Test and standardized reading measures. A principal component score derived from the South‐gate Word Selection Test and the ACER Primary Reading Survey Test Level AA was used to derive three groups of readers (poor, average, good). A factor analysis of the CAP indicated four clusters of items. Three of these, letter‐word concepts (.77), print direction concepts (.39) and age (.32), were found to load on a significant discriminant function separating the three groups. Univariate procedures revealed that print direction concepts, letter‐word concepts and advanced print concepts significantly separated the three groups of readers but book orientation did not. Results were discussed in relation to the validity and reliability of the CAP, the relevance of linguistic awareness skills to early reading and the need to tie instruction to concept learning.     

Psychometric g and mental processing speed on a semantic verification test

The Semantic Verification Test (SVT) consists of brief, simple statements, which are either true or false, about various arrangements of the letters ABC (e.g. B after A). Mean reaction time (RT) for confirming or disconfirming the various statements varies according to their complexity. In independent studies of university students and Navy recruits, RT and other response latency parameters (intraindividual variability and movement time) from SVT performance show significant correlations of about -0.40 with nonspeeded tests of psychometric g. The mean RTs of adults to the various SVT item types are highly related to the mean error rates on these item types when the SVT is taken by elementary school children as a nonspeeded paper-and-pencil test. RT is correlated with the general cognitive ability factor (g) and not with the test-taking speed factor that is found in speeded paper-and-pencil tests. The degree of correlation between RT and psychometric g does not show any regular relationship to differences in the SVT item-type's complexity or difficulty as indicated by mean RT.     

An Ideal Student Factor and the validity of noncognitive measures of student potential

The presence of a general factor has been found in previous research on the utilization of personality tests in selection contexts. Termed the Ideal Employee Factor (IEF), this factor is thought to represent an effort to present oneself as an ideal employee to some degree on all items in the test. Examining the responses to biodata and situational judgment items, we provide evidence for the existence of a factor similar to the IEF within the college admissions context, termed the Ideal Student Factor (ISF), but the magnitude of this factor relative to specific factors was small. Mixed support was found for the hypothesized differences in the factor loadings on the ISF for different constructs included in the model. The presence of the ISF was not found to substantially impact the validity of other predictors within this study. The overall conclusion is that this general factor would not impact the use of these biodata and situational judgment items in college admissions.     

Can learning disabled students effectively use separate answer sheets?

101 regular class and learning disabled students were administered three subtests of the Comprehensive Test of Basic Skills for which all correct answers had been identified in the students' test booklet. Analysis of the completed separate answer sheets indicated that learning disabled students answered fewer total items than their nondisabled peers but did not differ with respect to percent of items answered correctly. In addition, nonsignificant differences were found for number of answer spaces filled in outside the line. Implications for further research and training are given.     

Using Invariance to Examine Cheating in Unproctored Ability Tests

Despite their widespread use in personnel selection, there is concern that cheating could undermine the validity of unproctored Internet‐based tests. This study examined the presence of cheating in a speeded ability test used for personnel selection. The same test was administered to applicants in either proctored or unproctored conditions. Item response theory differential functioning analyses were used to evaluate the equivalence of the psychometric properties of test items across proctored and unproctored conditions. A few items displayed different psychometric properties, and the nature of these differences was not uniform. Theta scores were not reflective of widespread cheating among unproctored examinees. Thus, results were not consistent with what would be expected if cheating on unproctored tests was pervasive.