Social Networking Websites,Personality Ratings,and the Organizational Context: More Than Meets the Eye?1

We examined the psychometric properties of the Big Five personality traits assessed through social networking profiles in 2 studies consisting of 274 and 244 social networking website (SNW) users. First, SNW ratings demonstrated sufficient interrater reliability and internal consistency. Second, ratings via SNWs demonstrated convergent validity with self‐ratings of the Big Five traits. Third, SNW ratings correlated with job performance, hirability, and academic performance criteria; and the magnitude of these correlations was generally larger than for self‐ratings. Finally, SNW ratings accounted for significant variance in the criterion measures beyond self‐ratings of personality and cognitive ability. We suggest that SNWs may provide useful information for potential use in organizational research and practice, taking into consideration various legal and ethical issues.     

Are supervisors and coworkers likely to witness employee counterproductive work behavior? An investigation of observability and self–observer convergence

Supervisor and coworker ratings (i.e., “observer ratings”) remain a common manner of measuring counterproductive work behavior (CWB) despite long‐standing doubts that observers have the opportunity to witness the work behaviors they are expected to rate. We conducted 2 studies that evaluated the observability of CWB items and consequences of observability. First, we show that many CWBs are unlikely to be witnessed by supervisors or coworkers—specifically, behaviors such as “spends too much time fantasizing or daydreaming instead of working” and “discussed confidential company information with an unauthorized person” were found to be lowest in observability, whereas “cursed at someone at work” or “acted rudely toward someone at work” were relatively higher in observability (though observability was generally low). Second, a meta‐analysis revealed variability in item‐level relationships (correlations and mean differences) between self‐ratings and observer ratings for specific CWB scale items (i.e., items from Bennett & Robinson, 2,000). Important, this variability was partially explained by observability—behaviors with low self–observer convergence tend to have low levels of observability, whereas behaviors with higher levels of convergence tend to have higher levels of observability. This study demonstrates that supervisor and coworker ratings of CWB may be susceptible to an observability bias resulting from rating behaviors they have not likely witnessed.     

Low accuracy in judgments of others' psychological well-being as seen from a phenomenological perspective

The generally low degree of agreement between self-ratings on personality traits and ratings by others may be interpreted from the viewpoint that self-reports reflect people's experience of themselves but not necessarily their behaviors. A detailed analysis of self and other ratings on subjective well-being as a central dimension of human experience is consistent with this phenomenological view. Ratings of well-being were not significantly correlated with rated behaviors either in self-ratings or in ratings by others. Screening subjects in terms of avowed consistency and observability on a trait did not improve self-other agreement for well-being, nor did it replicate the individual trait effects reported by Kenrick and Stringfield (1980). Judgments by others were found to have poor interjudge reliability and to reflect biases associated with projection of own well-being and a halo effect organized around the subject's perceived friendliness or likability. It was demonstrated that pooling the judgments of several observers should not and does not lead to accurate prediction of the phenomenal personality, and that accuracy may generally depend on the level of self-disclosure.     

The impact of job complexity and performance measurement on the temporal consistency, stability, and test-retest reliability of employee job performance ratings

Although research has shown that individual job performance changes over time, the extent of such changes is unknown. In this article, the authors define and distinguish between the concepts of temporal consistency, stability, and test-retest reliability when considering individual job performance ratings over time. Furthermore, the authors examine measurement type (i.e., subjective and objective measures) and job complexity in relation to temporal consistency, stability, and test-retest reliability. On the basis of meta-analytic results, the authors found that the test-retest reliability of these ratings ranged from .83 for subjective measures in low-complexity jobs to .50 for objective measures in high-complexity jobs. The stability of these ratings over a 1-year time lag ranged from .85 to .67. The analyses also reveal that correlations between performance measures decreased as the time interval between performance measurements increased, but the estimates approached values greater than zero.     

Reliability and convergent validity of the Job Components Inventory

Further evidence is presented on the reliability and validity of the Job Components Inventory (JCI), a recently developed job analysis technique. The JCI covers the use of tools and equipment, physical and perceptual requirements, mathematical requirements, communication requirements, decision-making and responsibility requirements. Results of a study of 75 postmen and postwomen and seven supervisors are reported. Inter-rater reliability, supervisor-job incumbent agreement and inter-office agreement were all found to be high. This study indicates that the JCI can be used to analyse jobs other than low skill youth jobs for which it was originally developed     

The accuracy of roommate ratings of behaviors versus beliefs

We evaluated the accuracy of peer ratings of roommates’ personality characteristics, against roommate self-ratings, as a function of rating domain observability. Instead of the usual ratings of broad personality traits, however, our domains represented peer ratings of narrow exemplars of personality traits. Specifically, we compared roommate ratings on (a) observable trait-related behaviors with (b) unobservable trait-related attitudes or beliefs. We observed greater self-peer agreement in rating behaviors, in general, than in rating beliefs. We also observed greater tendency of raters to adopt an assumed similarity heuristic when judging their roommates’ attitudes and beliefs than their behaviors. We discuss the contribution of these findings to understanding the determinants of accuracy in personality judgments and developing best practices for personality assessment.     

Worker traits and worker functions in DOT

Some users of the worker traits ratings and worker function hierarchies in DOT do not seem to be fully aware that the two systems are in large part redundant: the worker traits can be efficiently summarized by the ratings of worker functions (level of complexity of working with data, people and things). This paper, a by-product of an extension of DOT to the Australian Census Classification of Occupations, attempts to validate the worker function hierarchies in terms of the worker traits required by different jobs in DOT. It shows empirically that the variation in worker traits across the 21,741 occupations listed in DOT is closely reflected in the 197 worker function profiles, which have the better research potential.     

Supervisor estimation of abilities required in jobs

The Minnesota Job Requirements Questionnaire (MJRQ) was developed and administered to supervisors of 11 selected jobs who rated the ability requirements of the job they supervised. The MJRQ represented each of nine GATB aptitudes by five items in a Likert-type format. MJRQ ratings were compared with expert ratings in the Dictionary of Occupational Titles (DOT) and Occupational Ability Patterns (OAPs) derived from administration of the General Aptitude Test Battery (GATB). Reliability of MJRQ ratings was shown by high Hoyt reliability coefficients. Comparison of mean ratings and variabilities of ratings by supervisors in a job showed construct validity for the MJRQ approach, as did intercorrelations of MJRQ ratings. OAPs derived from the MJRQ compared favorably with those derived from the GATB and DOT, showing promise for the parsimonious MJRQ approach to the development of OAPs.     

Worker estimation of ability requirements of their jobs

A modified version of the Minnesota Job Requirements Questionnaire (MJRQ), representing the GATB aptitudes, was used by workers in 11 selected jobs to rate the ability requirements of their jobs. Worker MJRQ ratings were compared with supervisor MJRQ ratings. Modified Occupational Ability Patterns (OAPs) derived from worker ratings were compared with OAPs based on supervisor ratings, OAPs from expert ratings in the DOT, and OAPs from the empirical GATB approach. Reliability of worker MJRQ ratings was comparable to that of supervisor ratings. Comparisons of mean ratings and variability of ratings of workers in a job category with workers in other jobs, and with supervisors rating the same job, showed construct validity for worker MJRQ ratings, as did intercorrelations of the worker ratings. OAPs derived from worker MJRQ ratings compared favorably with OAPs derived from supervisor MJRQ ratings, from the DOT, and from the GATB. Differences among workers on variables such as satisfaction, age and tenure did not significantly affect their MJRQ ratings. Implications of the worker MJRQ ratings are discussed.     

CHILDREN'S PERSON PERCEPTION: THE GENERALIZATION FROM TELEVISION PEOPLE TO REAL PEOPLE

This study investigated children's perceptions of television people and the extent to which these perceptions are generalized to impressions of peers. Elementary school children (W=172) completed questionnaires measuring viewing, knowledge of, identification with, and the perceived reality of television characters, and ratings of peers on 11 traits (e.g., funny, attractive, kind, nice). Two criterion variables were studied: children's mean rating of peers on each of the traits and the variance in the ratings of peers on each trait. Multiple regression analyses (controlling for grade and sex) revealed that: (1) identification with television characters was a significant predictor of the mean evaluation of peers on the traits; (2) television viewing was a significant predictor of the variance in children's ratings of peers; and (3) the relationship between television viewing and the variance in trait ratings was lower for attributes found in past studies to be suggested by television presentations. Implications of these findings for the traditional concern that television standardizes perceptions of people through stereotyping are considered.     

INTERRATER CORRELATIONS DO NOT ESTIMATE THE RELIABILITY OF JOB PERFORMANCE RATINGS

Interrater correlations are widely interpreted as estimates of the reliability of supervisory performance ratings, and are frequently used to correct the correlations between ratings and other measures (e.g., test scores) for attenuation. These interrater correlations do provide some useful information, but they are not reliability coefficients. There is clear evidence of systematic rater effects in performance appraisal, and variance associated with raters is not a source of random measurement error. We use generalizability theory to show why rater variance is not properly interpreted as measurement error, and show how such systematic rater effects can influence both reliability estimates and validity coefficients. We show conditions under which interrater correlations can either overestimate or underestimate reliability coefficients, and discuss reasons other than random measurement error for low interrater correlations.     

Using Personality Item Characteristics to Predict Single‐Item Internal Reliability,Retest Reliability,and Self–Other Agreement

The use of reliability estimates is increasingly scrutinized as scholars become more aware that test–retest stability and self–other agreement provide a better approximation of the theoretical and practical usefulness of an instrument than its internal reliability. In this study, we investigate item characteristics that potentially impact single‐item internal reliability, retest reliability, and self–other agreement. Across two large samples (N = 6690 and N = 4396), two countries (Estonia and The Netherlands), and two personality inventories (the NEO PI‐3 and the HEXACO‐PI‐R), results show that (i) item variance is a strong predictor of self–other agreement and retest reliability but not of single‐item internal reliability; (ii) item variance mediates the relations between evaluativeness and self–other agreement; and (iii) self–other agreement is predicted by observability and item domain. On the whole, weak relations between item length, negations, and item position (indicating effects of questionnaire length) on the one hand, and single‐item internal reliability, retest reliability, and self–other agreement on the other, were observed. In order to increase the predictive validity of personality scales, our findings suggest that during the construction of questionnaire items, researchers are advised to pay close attention especially to item variance, but also to evaluativeness and observability. Copyright © 2016 European Association of Personality Psychology     

EXAMINING RATING SOURCE VARIATION IN WORK BEHAVIOR TO KSA LINKAGES

We examined Work Behavior to knowledge, skill, or ability linkage ratings for 9 jobs to determine the degree to which differences in the ratings were due to rater type. We collected ratings from incumbents and 2 types of job analysts: project job analysts (analysts knowledgeable of the job) and nonproject job analysts (analysts with very little or no knowledge of the job). In our analyses of the data, we calculated means, standard deviations, effect sizes, and correlations for each rater type, as well as compared the reliability of the ratings. We also estimated variance components for each job by conducting generalizability analyses ( Brennan, 1983 ; Shavelson, Webb, & Rowley, 1989 ). Our findings indicate that the level of linkage ratings is similar across rater types, that it is important to obtain ratings from multiple raters regardless of rater type, and that ratings from job analysts may be more reliable than those of incumbents.     

Job Type and Recruiters' Inferences of Applicant Personality Drawn from Resume Biodata: Their Relationships with Hiring Recommendations

Using data collected from 244 recruiters who reviewed resumes and made dispositional inferences and hiring recommendations for 122 entry‐level job applicants, we found that type of job opening (Holland's Conventional vs. Enterprising jobs) moderated relationships between recruiter inferences of applicant personality traits and recruiter judgments of applicant employability. Results showed Conscientiousness correlated with employability ratings for Conventional jobs (e.g., accounting) while extraversion was most associated with employability ratings for Enterprising jobs (e.g., marketing).     

Ill-structured measurement designs in organizational research: implications for estimating interrater reliability

Organizational research and practice involving ratings are rife with what the authors term ill-structured measurement designs (ISMDs)--designs in which raters and ratees are neither fully crossed nor nested. This article explores the implications of ISMDs for estimating interrater reliability. The authors first provide a mock example that illustrates potential problems that ISMDs create for common reliability estimators (e.g., Pearson correlations, intraclass correlations). Next, the authors propose an alternative reliability estimator--G(q,k)--that resolves problems with traditional estimators and is equally appropriate for crossed, nested, and ill-structured designs. By using Monte Carlo simulation, the authors evaluate the accuracy of traditional reliability estimators compared with that of G(q,k) for ratings arising from ISMDs. Regardless of condition, G(q,k) yielded estimates as precise or more precise than those of traditional estimators. The advantage of G(q,k) over the traditional estimators became more pronounced with increases in the (a) overlap between the sets of raters that rated each ratee and (b) ratio of rater main effect variance to true score variance. Discussion focuses on implications of this work for organizational research and practice.     

Core Self-Evaluations and Work Success

ABSTRACT— Core self-evaluations (CSE) is a broad, integrative trait indicated by self-esteem, locus of control, generalized self-efficacy, and (low) neuroticism (high emotional stability). While only a decade old, research on CSE suggests that it explains much of the overlap among these trait measures, while also predicting many work and other applied outcomes better than the individual traits. Individuals with high levels of CSE perform better on their jobs, are more successful in their careers, are more satisfied with their jobs and lives, report lower levels of stress and conflict, cope more effectively with setbacks, and better capitalize on advantages and opportunities. Though research on individual self-concept traits such as self-esteem and locus of control should continue, researchers interested in these traits should consider the advantages of CSE in its relation to success in work and in life.