Online Testing: Mode of Administration and the Stability of OPQ 32i Scores

This study explores the equivalence of web‐based administration with no local supervision and traditional paper‐and‐pencil supervised versions of OPQ32i (the ipsative format version of the Occupational Personality Questionnaire). Samples of data were collected from a range of client projects and matched in terms of industry sector, assessment purpose (selection or development) and candidate category (graduate or managerial/professional). The analysis indicates that lack of local supervision in high‐stakes situations has little if any impact on scale scores. At worst, some scales appear to show shifts of less than quarter of an SD, with most scales showing little change, if any. Analyses in terms of the Big Five show differences of less than .2 of an SD. Scale reliabilities and scale covariances appear to be unaffected by the differences between the supervised and unsupervised administration conditions.     

Paper-and-pencil and web-based testing: the measurement invariance of the Big Five personality tests in applied settings

This study investigates the measurement equivalence of a five-factor measure of personality across two groups applying for jobs, who completed the same questionnaire using either a paper-and-pencil (n = 429) or a web online answer format (n = 651). The data were collected using the Big Five Questionnaire-2 (BFQ-2; which is a measure of the Five Factor Model) of personality traits. Multiple-group confirmatory factor analysis was used to test for the equivalence of factor covariance and mean structures of the BFQ-2. Findings suggested that the Five Factor Model scales have the same measurement unit and origin across applicants using different administration modes. However, latent means were slightly higher for applicants who responded in a web and unproctored condition than for applicants who completed a paper-and-pencil version of the same test. Practical implications for personality assessment are discussed.     

Mobile Internet Testing: An analysis of equivalence,individual differences,and reactions

Mobile internet testing (MIT) is the latest cost effective technological push in employment testing. The ability to access assessments via the internet using mobile devices provides increased speed and convenience for both administrators and respondents. In this article, we examine the equivalence of MIT compared with testing on personal computers (PCs) and whether attitudes and other individual differences influence responses and reactions to MIT. Results demonstrated equivalence for a supervisory situational judgment test across testing modes, but not for a cognitive ability test. Significant relationships were found between anxiety and both performance and reactions to mobile assessment. Respondents also reported significantly more positive reactions when tested on a PC versus a mobile device. Future research and practical implications are discussed.     

Are curmudgeon personality scales resistant to response distortion?

Curmudgeon personality, which is assessed by having participants evaluate a heterogeneous set of attitude objects, provides an applied value beyond that of more established personality traits. Recent research, for instance, suggests that curmudgeon personality is distinct from the Five Factor Model personality traits and that it predicts unique variance in important criteria, such as job attitudes and well-being. In the current paper we contribute to this literature by testing an explanation of how curmudgeon personality is distinct from other traits—their potential resistance to response distortion. We estimated response distortion for scales of curmudgeon personality and FFM traits across two quasi-experimental studies. As expected, the score shifts between nonapplicant and applicant conditions were smaller for scales of curmudgeon personality compared to scales of emotional stability and conscientiousness. We argue that curmudgeon personality scales may be more resistant to response distortion than are measures of other traits because curmudgeon personality items are semi-implicit and lack an obvious desirable response. The resistance to response distortion should serve as a call to future researchers to test the criterion validity of curmudgeon personality in regard to performance criteria.     

Comparing supervised and unsupervised category learning

Two unsupervised learning modes (incidental and intentional unsupervised learning) and their relation to supervised classification learning are examined. The approach allows for direct comparisons of unsupervised learning data with the Shepard, Hovland, and Jenkins (1961) seminal studies in supervised classification learning. Unlike supervised classification learning, unsupervised learning (especially under incidental conditions) favors linear category structures over compact nonlinear category structures. Unsupervised learning is shown to be multifaceted in that performance varies with task conditions. In comparison with incidental unsupervised learning, intentional unsupervised learning is more rule like, but is no more accurate. The acquisition and application of knowledge is also more laborious under intentional unsupervised learning.     

Crowdsourcing and personality measurement equivalence: A warning about countries whose primary language is not English

In the search to find cheaper, faster approaches for data collection, crowdsourcing methods (i.e., online labor portals that allow independent workers to complete surveys for compensation) have risen in popularity as a tool for personality researchers, despite a lack of evidence regarding the equivalence of crowdsourcing with traditional data collection methods. The purpose of this study was to evaluate crowdsourcing as a data collection tool by examining the measurement equivalence of crowdsourced data (i.e., from Amazon.com’s MTurk) with more traditional samples (i.e., an undergraduate sample and a sample of organizational employees). Our results (using a popular measure of Big Five personality) provided evidence of measurement equivalence across all three samples, with one important exception: crowdsourced data (from MTurk) only exhibited measurement invariance with traditional data collection methods when responses were restricted to participants from native-English speaking countries. Although MTurk appears to be an easy, cost-effective data collection tool, our results suggest that MTurk data are similar to traditionally-collected data only when the MTurk sample is restricted to IP addresses from English-speaking countries.     

The Five Factor Model in personnel selection: Measurement equivalence between applicant and non-applicant groups

This study investigates the measurement equivalence of a Five Factor measure of personality between applicant and non-applicant samples. The Big Five Questionnaire-2 was administered in two samples: A group of volunteers (n = 903), who completed the test for research purposes, and a group applying for jobs, who completed the test during hiring procedures (n = 401). Multiple-group confirmatory factor analysis on item composites was conducted to test for the equivalence of factor covariance and mean structures of twenty facets of the Five Factors. Item-level analyses were carried out through analysis of variance to further examine the issue of measurement invariance. Findings suggested that personality facets have the same measurement unit across applicants and non-applicants, while a lack of equivalence was found in the origin of the scales. Similar results were found at the item-level. Implications for personality assessment are advanced and discussed.     

Strategy development and learning differences in supervised and unsupervised categorization

The processes that determine unsupervised categorization, the task of classifying stimuli without guidance or feedback, are poorly understood. Two experiments examined the emergence and plasticity of unsupervised strategies using perceptual stimuli that varied along two separable dimensions. In the first experiment, participants either classified stimuli into any two categories of their choice or learned identical classifications by supervised categorization. Irrespective of the complexity of classification, supervised and unsupervised learning rates differed little when both modes of learning were maximally comparable. The second experiment examined the plasticity of unsupervised classifications by introducing novel stimuli halfway through training. Whether or not people altered their strategies, they responded to novel stimuli in a gradual manner. The gradual and continuous evolution and adaptation of strategies suggests that unsupervised categorization involves true learning which shares many properties of supervised category learning. We also show that the choice of unsupervised strategy cannot be predicted from the properties of early learning trials, but is best understood as a function of the initial distribution of dimensional attention.     

Measurement Equivalence Between Applicant and Incumbent Groups: An IRT Analysis of Personality Scales

Incumbents are often used in the development and validation of a wide variety of personnel selection instruments, including noncognitive instruments such as personality tests. However, the degree to which assumed motivational factors impact the measurement equivalence and validity of tests developed using incumbents has not been adequately addressed. This study addressed this issue by examining the measurement equivalence of 6 personality scales between a group applying for jobs as sales managers in a large retail organization (N = 999) and a group of sales managers currently employed in that organization (N = 796). A graded item response theory model (Samejima, 1969) was fit to the personality scales in each group. Results indicated that moderately large differences existed in personality scale scores (approximately 1/2 standard deviation units) but only one of the six scales contained any items that evidenced differential item functioning and no scales evidenced differential test functioning. In addition, person-level analyses showed no apparent differences across groups in aberrant responding. The results suggest that personality measures used for selection retain similar psychometric properties to those used in incumbent validation studies.     

THE PEN-BASED COMPUTER AS AN ALTERNATIVE PLATFORM FOR TEST ADMINISTRATION

Pen-based computers are similar to paper and pencil (P&P) tests in the method of responding, and thus, may more closely match paper and pencil administration in construct equivalence than keyboard-entry computers. A study was conducted comparing P&P, pen-based note-book computer, and keyboard-entry PC versions of two test batteries. Participants completed tests administered using different administration modes on separate days; construct equivalence was evaluated by comparing Day 1-Day 2 correlations across conditions. Although construct equivalence was found for the power tests, differences emerged for the speeded tests. For the pen-based computer, solid evidence of equivalence to P&P appeared for all but one of the speeded tests, whereas the keyboard PC showed borderline equivalence for only one of the three speeded tests. These findings suggested that the pen-entry computer may be more capable than the keyboard-entry computer in maintaining construct equivalence to P&P tests.     

Examination of personality-ability relationships in the elderly: The role of the contextual (interface) assessment mode

Personality-ability relationships in old age were examined in 71 elderly persons. Two modes of assessment were used. One consisted of measuring intellectual abilities (17 tests) and personality (6 dimensions) by means of established instruments. Intellectual abilities covered were drawn from Horn and Cattell's model of psychometric intelligence. Personality dimensions included three measures of locus of control and one measure each for achievement motivation, anxiety, and morale. The second mode of assessment was aimed directly at the interface between cognitive abilities and personality. Six such interface (context-specific) scales, involving self-assessment and attributions dealing with intellectual functioning in the context of aging, were developed using the six traditional (transcontextual) personality scales as conceptual criteria. These newly developed interface scales exhibit satisfactory convergent and discriminant validity with their transcontextual parent scales of personality. The six parent personality measures and the six personality-ability interface scales were correlated, separately, with performance scores of psychometric intelligence. Relationships between the two domains (personality, ability) were substantial and of greater magnitude when personality was assessed in the context of the personality-ability interface. Because of the conjoint anchoring of the interface scales to both personality and ability, the findings provide a basis for beginning to clarify the nature of personality-ability relationships in old age. In addition, the results support the continuation of a measurement approach to the study of interdomain relationships that includes the use of direct assessments of the domain interface, in addition to traditionally distinct measures of different domains.     

The Role of Parental Attitudes and Monitoring in the Risk Exposure of Young Children1

The parental monitoring literature debates the role of parental activity, as opposed to parental knowledge in childhood risk taking. We examined parents’ attitudes and behaviors regarding roadside supervision of children aged 4 to 6 years using self‐report and observational measures. Although parents perceived sons as less careful than daughters, they reported no sex differences in the age children would be allowed out unsupervised. There were no self‐report differences in supervision according to socioeconomic status. However, roadside observations showed that boys and children from a lower socioeconomic area received less supervision than girls or those from a higher socioeconomic area. Such findings contribute to our understanding of the sex and socioeconomic differences in accident involvement and the role of parental monitoring.     

Stability of the Millon Clinical Multiaxial Inventory among psychiatric inpatients

This study investigated the stability of the Millon Clinical Multiaxial Inventory (MCMI) in a sample of psychiatric inpatients over a longer time period and from different perspectives than previous reliability studies. Test-retest reliability was assessed from three perspectives: correlation/regression, equivalence of dimensional structures across testings upon different admissions, and stability of personality style across testings. Some stability from a correlational perspective was found, with higher stability estimates indicated for the basic personality scales in comparison with the clinical symptom scales which is consistent with Millon's theory. The internal structure of the MCMI was essentially identical across testings. Although source of reinforcement and instrumental or coping style were stable across testings beyond chance, the MCMI profiles of a considerable number of subjects were inconsistent across testings. Correspondingly, clinicians should be cautious in using the MCMI to make categorical decisions regarding personality style with inpatients who are tested at admission and who have had repeated hospitalizations.     

Equivalence of unproctored internet testing and proctored paper‐and‐pencil testing of the Big Five

The use of unproctored Internet‐based testing (IBT) for personality assessment is increasingly popular, especially in personnel selection. Previous studies on its equivalence to traditional proctored paper‐and‐pencil testing (PPT) have used between‐subjects designs, which makes it difficult to separate intergroup effects from format effects, and have shown mixed results. The aim of the present study was to assess the quantitative, qualitative, and auxiliary equivalence of unproctored IBT and proctored PPT of personality, using a within‐sample design. Undergraduate students (n = 407) completed both an Internet and a paper‐and‐pencil version of a measure of the Big Five, with a 1 to 3 week interval. The proctored paper‐and‐pencil assessment was completed in class and the unproctored Internet assessment was completed at the time and place chosen by the participants. Results showed effect sizes for mean differences to vary from null to small. Skewness and kurtosis indices, reliability coefficients, intercorrelations magnitudes, as well as factor solutions, were highly similar across formats. Respondents did not prefer IBT over PPT on a series of statement about their perceptions and reactions to IBT. In conclusion, results suggest that IBT and PPT of the Big Five personality traits are equivalent, and that the data obtained are comparable across formats.     

TESTAN: An integrated modular system for personality assessment and test development on MS-DOS personal computers

TESTAN (TEST and ANalysis) is an authoring shell and integrated psychometric package for development of computerized multiple-choice questionnaires, tests, and personality scales on personal computers that use MS-DOS. TESTAN allows (1) writing and editing items, interpretation messages, keys, norms, a bank of profiles, and so on, (2) collecting data in on-line or paper-and-pencil modes, (3) selecting the most discriminating items by means of correlation and factor analysis for practical use, (4) validating test scales and items according to external criteria and expert ratings, and (5) making multifactor assessment decisions after testing. TESTAN can be used to test student conceptual knowledge in any area in the form of multiple-choice questions. This report describes essential functional properties and facilities of TESTAN for psychometrists and applied psychologists.     

Forced-choice and conventional personality assessment: Each may have unique value in pre-employment testing

Forced-choice (FC) personality scales are purported to predict criteria such as counterproductive work behavior (CWB) incrementally beyond conventional personality measures in pre-employment testing situations. Unfortunately, the research upon which FC tests’ claims of superiority are founded, relies upon undergraduate participants in highly artificial scenarios. Moreover, the possible predictive advantages of FC personality scales with regards to important contextual performance (CP; Borman & Motowidlo, 1993) criteria have not been investigated. We used a sample of call center employees to investigate the incremental validity of a FC personality scale over a conventional personality scale (and vice versa) with CP measures, and CWB, as criteria. Our FC and conventional personality measures were highly correlated, but each still offered unique predictive value with regards to CP or CWB. Thus, contrary to past research with student samples, in field settings FC personality scales may not be clearly superior to conventional personality scales. Moreover, contrary to widespread concern that FC measures’ predictiveness is attributable to their overlap with general mental ability, the unique predictive value of our FC measure remained even when variance due to general mental ability was statistically controlled.     

等效性检验——结构方程模型评价和测量不变性分析的新视角

常用的结构方程模型拟合指数存在一定局限, 如χ ²以传统零假设为目标假设, 无法验证模型, 而RMSEA和CFI等描述性的拟合指数不具备推断统计性质, 等效性检验有效弥补了这些问题。首先说明等效性检验如何评价单个模型的拟合, 并解释其与零假设检验的不同, 然后介绍等效性检验如何分析测量不变性, 接着用实证数据展示了等效性检验在单个模型评价和测量不变性检验中的效果, 并与传统模型评价方法比较。     

A Five-Factor Measure of Dependent Personality Traits

This study provides convergent, discriminant, and incremental validity data for a new measure of dependent personality traits from the perspective of the five-factor model (FFM). Dependent personality trait scales were constructed as maladaptive variants of FFM facets (e.g., Gullibility as a maladaptive variant of FFM trust). Based on responses from 383 undergraduates, the convergent validity of the Five-Factor Dependency Inventory (FFDI) scales was tested with respect to 2 measures of the FFM, 6 dependency trait scales, and 4 measures of dependent personality disorder. Discriminant validity was tested with respect to FFM facets from alternative domains. Incremental validity was tested with respect to the ability of the FFM dependent personality trait scales to account for variance in 2 established measures of dependency, after variance accounted for by respective FFM facet scales and other measures of DPD was first removed. The results of this study provided support for the validity of the FFDI assessment of dependency from the perspective of the FFM.     

A five-factor measure of dependent personality traits

This study provides convergent, discriminant, and incremental validity data for a new measure of dependent personality traits from the perspective of the five-factor model (FFM). Dependent personality trait scales were constructed as maladaptive variants of FFM facets (e.g., Gullibility as a maladaptive variant of FFM trust). Based on responses from 383 undergraduates, the convergent validity of the Five-Factor Dependency Inventory (FFDI) scales was tested with respect to 2 measures of the FFM, 6 dependency trait scales, and 4 measures of dependent personality disorder. Discriminant validity was tested with respect to FFM facets from alternative domains. Incremental validity was tested with respect to the ability of the FFM dependent personality trait scales to account for variance in 2 established measures of dependency, after variance accounted for by respective FFM facet scales and other measures of DPD was first removed. The results of this study provided support for the validity of the FFDI assessment of dependency from the perspective of the FFM.     

A Comparative Study of Integrity Tests: The Criterion-Related Validity of Personality-Based and Overt Measures of Integrity

In the field of personality assessment, one construct or set of constructs that is emerging as important for both publishers and users of psychological testing has been labelled variously as integrity, conscientiousness, reliability, delinquency, or responsibility. The associated organizational outcome has been broadly labelled counterproductivity. In the present study, four integrity scales, selected scales from mainstream personality inventories, derived molar integrity factors and optimal scale linear combinations were correlated with admissions of counterproductivity. It was found that (a) all four integrity scales were significantly correlated with the counterproductivity criterion, (b) individual personality scales from normal personality inventories, as well as unit-weighted linear combinations of them, were approximately as highly correlated with counterproductivity as were the integrity tests, and (c) personality traits conceptually distinct from the domain of integrity were also related to counterproductivity.