PERSONALITY AND JOB PERFORMANCE: A CRITIQUE OF THE TETT, JACKSON, AND ROTHSTEIN (1991) META-ANALYSIS

Tett, Jackson, and Rothstein (1991) recently presented a meta-analysis of the relationship between personality and job performance. Many of their findings, particularly those pertaining to the Big Five personality dimensions, are at odds with one other large scale meta-analytic study (Barrick & Mount, 1991) investigating the relation between personality and performance. In order to reconcile these new results with previous findings, we examined differences in the sample sizes used, the process for assigning pre-existing scales to personality dimensions, and the nature of the jobs investigated. In addition, we found four technical errors in the Tett et al. moderator meta-analyses in computing sampling error, the bias correction, sampling error for bias corrected correlations, and computing sampling error variance across studies. These errors raise serious questions about the interpretation of their results for various moderators of the personality-job performance relationship.     

Statistical inferences about the error variance

This paper is a presentation of an essential part of the sampling theory of the error variance and the standard error of measurement. An experimental assumption is that several equivalent tests with equal variances are available. These may be either final forms of the same test or obtained by dividing one test into several parts. The simple model of independent and normally distributed errors of measurement with zero mean is employed. No assumption is made about the form of the distributions of true and observed scores. This implies unrestricted freedom in defining the population. First, maximum-likelihood estimators of the error variance and the standard error of measurement are obtained, their sampling distributions given, and their properties investigated. Then unbiased estimators are defined and their distributions derived. The accuracy of estimation is given special consideration from various points of view. Next, rigorous statistical tests are developed to test hypotheses about error variances on the basis of one and two samples. Also the construction of confidence intervals is treated. Finally, Bartlett's test of homogeneity of variances is used to provide a multi-sample test of equality of error variances.     

Variance Estimation of Nominal-Scale Inter-Rater Reliability with Random Selection of Raters

Most inter-rater reliability studies using nominal scales suggest the existence of two populations of inference: the population of subjects (collection of objects or persons to be rated) and that of raters. Consequently, the sampling variance of the inter-rater reliability coefficient can be seen as a result of the combined effect of the sampling of subjects and raters. However, all inter-rater reliability variance estimators proposed in the literature only account for the subject sampling variability, ignoring the extra sampling variance due to the sampling of raters, even though the latter may be the biggest of the variance components. Such variance estimators make statistical inference possible only to the subject universe. This paper proposes variance estimators that will make it possible to infer to both universes of subjects and raters. The consistency of these variance estimators is proved as well as their validity for confidence interval construction. These results are applicable only to fully crossed designs where each rater must rate each subject. A small Monte Carlo simulation study is presented to demonstrate the accuracy of large-sample approximations on reasonably small samples.     

Note on use of extreme criterion groups in item discrimination analysis

The purpose of this note is to reconsider the Kelley-Cureton definition of optimal extreme groups for estimating item-criterion correlations. Optimal tail per cents are derived, using the criterion of minimum sampling variance of the tetrachoric correlation coefficient, and the findings are related to earlier work of Mosteller. It is shown that upper and lower 27 per cent groups yield the most precise estimate of the tetrachoric coefficient only when the population correlation is close to zero. When the population value is .4, extreme 20 per cent groups provide estimates with the smallest sampling error variance. It is further shown, however, that 27 per cent extremes yield highly efficient estimates. Thus no change is recommended in traditional item analysis procedures.     

Further evaluating the conditional decision rule for comparing two independent means

Many books on statistical methods advocate a ‘conditional decision rule’ when comparing two independent group means. This rule states that the decision as to whether to use a ‘pooled variance’ test that assumes equality of variance or a ‘separate variance’ Welch t test that does not should be based on the outcome of a variance equality test. In this paper, we empirically examine the Type I error rate of the conditional decision rule using four variance equality tests and compare this error rate to the unconditional use of either of the t tests (i.e. irrespective of the outcome of a variance homogeneity test) as well as several resampling‐based alternatives when sampling from 49 distributions varying in skewness and kurtosis. Several unconditional tests including the separate variance test performed as well as or better than the conditional decision rule across situations. These results extend and generalize the findings of previous researchers who have argued that the conditional decision rule should be abandoned.     

Methods for second order meta-analysis and illustrative applications

This paper presents methods for second order meta-analysis along with several illustrative applications. A second order meta-analysis is a meta-analysis of a number of statistically independent and methodologically comparable first order meta-analyses examining ostensibly the same relationship in different contexts. First order meta-analysis greatly reduces sampling error variance but does not eliminate it. The residual sampling error is called second order sampling error. The purpose of a second order meta-analysis is to estimate the proportion of the variance in mean meta-analytic effect sizes across multiple first order meta-analyses attributable to second order sampling error and to use this information to improve accuracy of estimation for each first order meta-analytic estimate. We present equations and methods based on the random effects model for second order meta-analysis for three situations and three empirical applications of second order meta-analysis to illustrate the potential value of these methods to the pursuit of cumulative knowledge.     

INDIVIDUAL DIFFERENCES IN INTERVIEWER RATINGS: THE IMPACT OF STANDARDIZATION, CONSENSUS DISCUSSION, AND SAMPLING ERROR ON THE VALIDITY OF A STRUCTURED INTERVIEW

This research investigated empirical issues regarding the validity of individual interviewers'( N = 62) ratings collected after a structured interview. Each interviewer rated an average of 25 interviewees. One hypothesis examined but not supported was that systematic interviewer errors will attenuate interview validity when data are aggregated across interviewers. Also investigated was the validity of ratings averaged across interviewers compared to consensus ratings; consensus ratings were shown to have significantly but probably not practically higher validities. Third, a meta-analysis of individual interviewer validities revealed that all of the variance in validities could be attributed to sampling error. Results and implications are discussed.     

基于概化理论的方差分量变异量估计

概化理论广泛应用于心理与教育测量实践中, 方差分量估计是进行概化理论分析的关键。方差分量估计受限于抽样, 需要对其变异量进行探讨。采用蒙特卡洛(Monte Carlo)数据模拟技术, 在正态分布下讨论不同方法对基于概化理论的方差分量变异量估计的影响。结果表明: Jackknife方法在方差分量变异量估计上不足取; 不采取Bootstrap方法的“分而治之”策略, 从总体上看, Traditional方法和有先验信息的MCMC方法在标准误及置信区间这两个变异量估计上优势明显。     

VALIDITY GENERALIZATION: A REVIEW AND CRITIQUE OF THE CORRELATION MODEL

This paper presents a review and critique of the validity generalization research based on the correlation model. Emphasis was placed on integrating the applied validity generalization studies as well as the computer simulation studies which have tested the accuracy of the various validity generalization procedures in estimating the mean and variance of true validity coefficients. In general, this review indicated that the validity generalization procedures are fairly accurate in estimating the mean and variance of true validities, that the preponderance (e.g., 90%) of corrected validity coefficients for a single predictor-criterion relationship, based on a series of cumulative studies, when placed in distributional form are positive, and that a substantial proportion of observed validity coefficient variation can be attributed to statistical artifacts with sampling error accounting for the majority of the artifactual variance. Moreover, potential problems with validity generalization/meta analysis based on criterion-related validity coefficients were discussed. It was concluded that validity general-ization/meta-analysis is an important development in summarizing cumulative research, however, caution should be exercised in utilizing and interpreting the findings of such analyses. Directions for future research aimed at establishing trait-performance relationships were discussed.     

Movement trajectory smoothness is not associated with the endpoint accuracy of rapid multi-joint arm movements in young and older adults

The minimum variance theory proposes that motor commands are corrupted by signal-dependent noise and smooth trajectories with low noise levels are selected to minimize endpoint error and endpoint variability. The purpose of the study was to determine the contribution of trajectory smoothness to the endpoint accuracy and endpoint variability of rapid multi-joint arm movements. Young and older adults performed arm movements (4 blocks of 25 trials) as fast and as accurately as possible to a target with the right (dominant) arm. Endpoint accuracy and endpoint variability along with trajectory smoothness and error were quantified for each block of trials. Endpoint error and endpoint variance were greater in older adults compared with young adults, but decreased at a similar rate with practice for the two age groups. The greater endpoint error and endpoint variance exhibited by older adults were primarily due to impairments in movement extent control and not movement direction control. The normalized jerk was similar for the two age groups, but was not strongly associated with endpoint error or endpoint variance for either group. However, endpoint variance was strongly associated with endpoint error for both the young and older adults. Finally, trajectory error was similar for both groups and was weakly associated with endpoint error for the older adults. The findings are not consistent with the predictions of the minimum variance theory, but support and extend previous observations that movement trajectories and endpoints are planned independently.     

Measures of sensitivity based on a single hit rate and false alarm rate: the accuracy, precision, and robustness of d', Az, and A'

Signal detection theory offers several indexes of sensitivity (d', Az, and A') that are appropriate for two-choice discrimination when data consist of one hit rate and one false alarm rate per condition. These measures require simplifying assumptions about how target and lure evidence is distributed. We examine three statistical properties of these indexes: accuracy (good agreement between the parameter and the sampling distribution mean), precision (small variance of the sampling distribution), and robustness (small influence of violated assumptions on accuracy). We draw several conclusions from the results. First, a variety of parameters (sample size, degree of discriminability, and magnitude of hits and false alarms) influence statistical bias in these indexes. Comparing conditions that differ in these parameters entails discrepancies that can be reduced by increasing N. Second, unequal variance of the evidence distributions produces significant bias that cannot be reduced by increasing N-a serious drawback to the use of these sensitivity indexes when variance is unknown. Finally, their relative statistical performances suggest that Az is preferable to A'.     

Modeling error variance in job specification ratings: the influence of rater, job, and organization-level factors

The authors modeled sources of error variance in job specification ratings collected from 3 levels of raters across 5 organizations (N=381). Variance components models were used to estimate the variance in ratings attributable to true score (variance between knowledge, skills, abilities, and other characteristics [KSAOs]) and error (KSAO-by-rater and residual variance). Subsequent models partitioned error variance into components related to the organization, position level, and demographic characteristics of the raters. Analyses revealed that the differential ordering of KSAOs by raters was not a function of these characteristics but rather was due to unexplained rating differences among the raters. The implications of these results for job specification and validity transportability are discussed.     

Age effects may be global, not local: comment on Fisk and Rogers (1991)

A series of analyses of variance on target search times allowed Fisk and Rogers (1991) to reject the null hypothesis that age had a uniform, additive effect across search conditions. It does not, however, follow that age affected some conditions in an exceptional way, as Fisk and Rogers concluded. Age may have had a uniform but nonadditive effect across conditions. In this article, it is shown that age had a uniform linear, or perhaps slightly curvilinear, effect on search times. This "null hypothesis" adequately accounted for the age effects in all 27 search conditions. Indeed, it accounted for the age effects in 107 conditions abstracted from other visual search studied and for the age effects in 154 conditions abstracted from a miscellaneous collection of nonsearch processing-time studies. The only variation in age outcomes across studies was consistent with sampling error, given the known variance in response times. It is concluded that age is experienced as a generalized slowing of the central nervous system uniformly affecting all information processes.     

Adult visual choice-reaction time, age, sex and preparedness: A test of Welford's problem in a large population sample

The relationship between age, sex and visual three-choice reaction time was investigated with a representative sample of adult subjects who participated in a large health survey in Finland (the Mini-Finland Survey). Subjects' age varied from 30 to 79 years. Reaction time increased reliably with increasing age. Men were faster than women across all age levels. The shortest of the irregularly presented inter-stimulus intervals (ISI) slowed down the performance of elderly subjects in particular. This suggests that they had more difficulty than younger subjects in getting prepared for an uncertain event. They also committed most errors at the longest ISI. This was interpreted as a result of their relative inability to maintain the optimum level of preparation. On the whole, men displayed a higher error rate than women, which suggests that the observed male superiority is at least in part due to response strategies. The importance for aging research of sampling the subjects and task types is briefly discussed.     

非正态分布下概化理论方差分量变异量估计

方差分量估计是概化理论的必用技术,但受限于抽样,需要对其变异量进行探讨。采用Monte Carlo数据模拟技术,探讨非正态数据分布对四种方法估计概化理论方差分量变异量的影响。结果表明：（1）不同非正态数据分布下,各种估计方法的“性能”表现出差异性;（2）数据分布对方差分量变异量估计有影响,适合于非正态分布数据的方差分量变异量估计方法不一定适合于正态分布数据。     

The retention of nouns,adjectives, and verbs of sentences

This research compared the retention of nouns, adjectives and verbs of sentences. A model of the structure underlying the retention of sentences and in which deep structure constituent propositions are hierarchically related, was used to make predictions concerning the retention of these word classes. The model was not supported by the data and it was determined that latencies to recognize the words of a previously presented sentence were the same for all classes of words. This result indicated that for recognition (but not necessarily for other memory tasks), each word of a sentence is seperately or equally accessible (as long as inter-word imagery is the same). However, a further experiment showed that, although word class is not a variable, the image-arousing capacity of a word is. Latencies and error rates were much lower for high-imagery words than for low-imagery ones. The findings on error rates in recognizing words were interesting and pointed to a different model where the meaning representations and syntatic functions of words have an active role in sentence retention.     

On the theoretical error bound for estimating psychometric functions

The theoretical limits to the amount of error, or the Cramer-Rao bounds, were derived for estimating psychometric functions. These theoretical error bounds were compared with the variability of psychometric functions estimated from human as well as computer-simulated observers. For the simulated observers, due to the limited efficiency of the sampling strategies, including the placement of the signals and the distribution of the trials, the variances of the estimated parameters are seven times the theoretical bound for threshold and 22 times that for slope. For the human observers, the variance is 18 times the theoretical bounds for threshold and 80 times that for slope. Therefore, a major portion of the variances (60% for threshold and 73% for slope) for the human observers is associated with factors other than sampling strategies. Further improvement of the accuracy for estimating psychometric functions will depend on not only optimizing the sampling strategy, but also better understanding the various sources of error related to the behavior of human observers.     

Experience sampling mood and its correlates at work

The experience sampling method is used to measure variance over time in events, moods, and behaviours in the work setting via palmtop computers in a sample of 41 employees. Theoretical propositions about event–mood–behaviour relations are derived from Affective Events Theory ( Weiss & Cropanzano, 1996 ) and tested using within‐ and between‐persons variance. The experiment found 56% of the variance in hedonic tone of mood was within‐ rather than between‐persons. Hedonic tone was significantly related to both positive and negative work events in expected directions. The relationship between negative events and mood was approximately five times stronger than that between positive events and mood, even though positive events were reported three to five times more frequently than negative events. Hedonic tone was positively related to engaging in work withdrawal and negatively related to engaging in work tasks. Implications of these findings as well as the use of experience sampling for the study of dynamic workplace variables are discussed.     

Reducing Measurement Error in Student Achievement Estimation

The achievement level is a variable measured with error, that can be estimated by means of the Rasch model. Teacher grades also measure the achievement level but they are expressed on a different scale. This paper proposes a method for combining these two scores to obtain a synthetic measure of the achievement level based on the theory developed for regression with covariate measurement error. In particular, the focus is on ordinal scaled grades, using the SIMEX method for measurement error correction. The result is a measure comparable across subjects with smaller measurement error variance. An empirical application illustrates the method.     

Competencies,Personality Traits,and Organizational Rewards of Middle Managers: A Motive-Based Approach

Previous literature suggests that performance ratings are saturated with rater-related idiosyncratic variance. Given that modern psychometric theories relegate this source of variance to measurement error, it has not been the subject of much previous research. Of importance, identifying and estimating the variance components underlying idiosyncratic rater variance will inform our understanding of the nature of this variance. In a sample of managerial performance ratings we report on components of variance and find that the idiosyncratic rater variance component is about one third rater main effects variance, one third Rater × Ratee interaction effects variance, and one third upper-bound Rater × Ratee × Dimension interaction effects variance. Further, results indicate that variance components are moderated by the acquaintanceship time between the rater and the ratee.