THE EFFECTS OF RANGE RESTRICTION ON ESTIMATES OF CRITERION INTERRATER RELIABILITY: IMPLICATIONS FOR VALIDATION RESEARCH

This paper explores the implications of the fact that range restriction affects reliability coefficients as well as validity coefficients. Of concern was the possibility that, if range restriction produced an underestimate of criterion reliability, the use of such an estimate in correcting observed validity coefficients for attenuation could result in overestimating population validity, lb address this issue, 3 scenarios were considered: truncation on the predictor, truncation on an unobserved variable, and truncation on the criterion. The 3 scenarios were found to have substantial differences in their consequences for reliability and validity estimation. Although range restriction did result in an underestimate of criterion reliability, it is generally not the case that these underestimates will result in overestimates of validity if researchers make effective use of available information about the presence and degree of range restriction.     

FACTORS AFFECTING VALIDITY OF A REGIONALLY ADMINISTERED ASSESSMENT CENTER

Validity coefficients for a single assessment center implemented in multiple locations are presented. Correction for range restriction, criterion unreliability, and sampling error did not account for a large portion of the variability in these validity coefficients. The type of assessor used, the center's administrative arrangement, and prior assessor-assessee contact moderated these validities. It is suggested that the quality of the implementation of a selection procedure when there is local latitude in its implementation is an important factor in determining the procedure's effectiveness.     

A note on the computation of biserialr in item validation

A method of computing biserial coefficients of correlation through the use of punch card tabulating equipment is presented. Each item is assigned a separate column and successes are punched 1. By arranging the cards on the criterion variable and obtaining progressive sums on several columns simultaneously, it is possible to obtain data for several correlations in one run of the cards through the machine.     

VALCOR: A program for estimating standard error, confidence intervals, and probability of corrected validity

VALCOR is a Turbo-Basic program that corrects the observed (uncorrected) validity coefficients for criterion and predictor unreliability and range restriction in the predictor. Furthermore, using the formulas for the standard error of functions of correlations derived by Bobko and Rieck (1980), the program provides an estimation of the standard error, the confidence intervals, and the probability of the corrected validity coefficients. In this way, the probability and the boundaries of the corrected validity coefficients may be reported together with the probability of the uncorrected validity coefficients. The results are presented on the computer screen and may be saved in an external file.     

The relation between results obtainable with raw and corrected correlation coefficients in multiple factor analysis

This paper presents three theorems concerning the relation between results with obtained and corrected correlation coefficients in the Thurstone method of multiple factor analysis. (1) The rank of the correlational matrix, and thus the number of factors involved, is unaffected by correcting the obtained coefficients for attenuation. (2) The communality of a variable when the obtained coefficients have been corrected for attenuation is equal to the communality with obtained coefficients divided by the reliability coefficient of the variable. (3) The relationship is shown between the factorial matrix of a correlational matrix of raw correlation coefficients and the factorial matrix of a correlational matrix of corrected correlation coefficients, and a method of obtaining either of these factorial matrices from the other is indicated.     

Maximizing test battery prediction when the weights are required to be non-negative

A procedure is developed for computing optimum regression weights under the restriction that they be non-negative. The weights maximize, subject to the restrictions, the multiple correlation between several predictors and a criterion. A numerical example is provided.     

Simplified formulas for item selection and construction

The formula for the Pearson correlation coefficient of a dichotomous variable with a multiple-categoried variable is simplified for computational purposes by effecting in the multiple-categoried variable two types of arbitrary distributions: (1) rectangular and (2) proportional to binomial expansion coefficients. The formulas which result are convenient for the selection of test items and are applicable to the objective estimation of the comparative merits of the alternatives in multiple-choice test items. It is shown that the authoritative answer should have a high positive criterion coefficient, while the omissions and several wrong-answer alternatives should each have low (algebraic) negative criterion coefficients.     

Moving Forward Indirectly: Reanalyzing the validity of employment interviews with indirect range restriction methodology

This study provides updated estimates of the criterion‐related validity of employment interviews, incorporating indirect range restriction methodology. Using a final dataset of 92 coefficients (N = 7,389), we found corrected estimates by structural level of .20 (Level 1), .46 (Level 2), .71 (Level 3), and .70 (Level 4). The latter values are noticeably higher than in previous interview meta‐analyses where the assumption was made that all restriction was direct. These results highlight the importance of considering indirect range restriction in selection. However, we found a number of studies involving both indirect and direct restriction, which calls into question the viability of assuming all restriction is now indirect. We found preliminary empirical support for correction of one of these multiple restriction patterns, indirect then direct.     

The evaluation of multiple and partial correlation coefficients from the factorial matrix

This paper shows how to compute multiple correlation coefficients, partial correlation coefficients, and regression coefficients from the factorial matrix. Special emphasis is given to computation technique and to approximation formulas. The method is extremely flexible in application since it may be applied to any subset of the original set of observed variables. It is also extremely useful when many of these coefficients are desired.     

The simultaneous prediction of any number of criteria by the use of a unique set of weights

When several criteria are available, it is ordinarily necessary (1) to select one of them asthe criterion, (2) to use several and thus arrive at several different sets of weights, or (3) to combine them into a single measure. A formula is derived for the determination of a unique set of weights. The use of these weights will produce the highest possible average coefficient of correlation between the various criteria and two (or more) weighted independent variables. If desired, the criteria may be assigned any predetermined weights. The weights then derived for the independent variables are such that the weighted average of the correlation coefficients between the various criteria and the independent variable composite will be a maximum. In the use of these formulas, no assumptions are necessary regarding the interrelationships existing among the criteria and it is not necessary to compute the intercorrelations among the criteria. A numerical example is included.     

Bootstrap standard error and confidence intervals for the correlation corrected for range restriction: a simulation study

The standard Pearson correlation coefficient is a biased estimator of the true population correlation, rho, when the predictor and the criterion are range restricted. To correct the bias, the correlation corrected for range restriction, rc, has been recommended, and a standard formula based on asymptotic results for estimating its standard error is also available. In the present study, the bootstrap standard-error estimate is proposed as an alternative. Monte Carlo simulation studies involving both normal and nonnormal data were conducted to examine the empirical performance of the proposed procedure under different levels of rho, selection ratio, sample size, and truncation types. Results indicated that, with normal data, the bootstrap standard-error estimate is more accurate than the traditional estimate, particularly with small sample size. With nonnormal data, performance of both estimates depends critically on the distribution type. Furthermore, the bootstrap bias-corrected and accelerated interval consistently provided the most accurate coverage probability for rho.     

Fisher transformations for correlations corrected for selection and missing data

The validity of a test is often estimated in a nonrandom sample of selected individuals. To accurately estimate the relation between the predictor and the criterion we correct this correlation for range restriction. Unfortunately, this corrected correlation cannot be transformed using Fisher'sZ transformation, and asymptotic tests of hypotheses based on small or moderate samples are not accurate. We developed a Fisherr toZ transformation for the corrected correlation for each of two conditions: (a) the criterion data were missing due to selection on the predictor (the missing data were MAR); and (b) the criterion was missing at random, not due to selection (the missing data were MCAR). The twoZ transformations were evaluated in a computer simulation. The transformations were accurate, and tests of hypotheses and confidence intervals based on the transformations were superior to those that were not based on the transformations.     

The Variability of Criterion‐Related Validity Estimates Among Interviewers and Interview Panels

The authors examined differences in criterion‐related validity estimates among ratings from individual interviewers and interview panels within a structured interview. Senior non‐commissioned officers (NCOs) in the U.S. Army (N=64) conducted panel interviews with 944 junior NCOs during a concurrent validation project. Analysis of the data revealed considerable variation in interviewer validity coefficients in relation to multiple performance criteria. Results also indicated the importance of adopting a multivariate perspective when evaluating interviewer validity differences in that the amount of variation in validity coefficients differed both by interview dimension and criterion. A similar pattern of findings emerged when analyses were performed on ratings averaged within interview panel. Nonetheless, when meta‐analysis was used to estimate the amount of true variance in interviewer‐ and panel‐level validity coefficients, most or all of the variance for some interview‐criterion combinations appeared to be due to statistical artifacts.     

A theoretical expectancy table calculator

A program for calculating theoretical expectancy tables is described. These tables are useful for communicating the meaning of correlation coefficients, since they display the relationship in a simple way. The program allows users to go beyond empirical expectancy tables, which are routinely calculated, by allowing the table to reflect corrections for effects such as unreliability and restriction of range.     

一种新的等值准则及其适用范围的探讨

受假设检验方法的启发,该文引出了一种基于项目反应理论的新等值方法——平方根等值准则。它具有一些特点：定义式中答对、答错概率同时出现而不能互相替代;极易从0—1评分模式的版本转换到多级评分版本;它可以看成是Haebara等值准则的加权形式。以等值系数估计值的误差大小为衡量标准,以Wilcoxon符号秩检验为依据,大量的Monte Carlo模拟结果显示了一种有趣的现象,即等值方法的运用范围既与项目参数估计精度有关,又与等值系数A的范围有关,但与另一个等值系数B的范围无关。当项目参数估计精度较高或中等而A取值在0.9～1.3之间,新方法往往比Stocking_Lord方法和Haebara方法的估计误差小且有显著性差异,当项目参数估计精度较低时,而A从1.0～2.0时新方法都有优越性。     

Examining the reliability of interval level data using root mean square differences and concordance correlation coefficients

This article introduces new statistics for evaluating score consistency. Psychologists usually use correlations to measure the degree of linear relationship between 2 sets of scores, ignoring differences in means and standard deviations. In medicine, biology, chemistry, and physics, a more stringent criterion is often used: the extent to which scores are identically equal. For each test taker (or other unit of measurement), the difference between the 2 scores is calculated. The root mean square difference (RMSD) represents the average change from 1 set of scores to the other, and the concordance correlation coefficient (CCC) rescales this coefficient to have a maximum value of 1. This article shows the relationship of the RMSD and CCC to the intraclass correlation coefficients, product-moment correlation, and standard error of measurement. Finally, this article adapts the RMSD and the CCC for linear, consistency, and absolute definitions of agreement.     

A comparison of preference‐assessment methods

In Study 1, we evaluated preference stability across 4 preference‐assessment methods for 6 individuals, 5 of whom had autism spectrum disorder and 1 of whom had traumatic brain injury. We also measured participants’ problem behavior as a corollary measure during all assessment methods. The highest mean correlation coefficients and Kendall rank coefficients of concordance across administrations were observed for the paired‐stimulus and multiple‐stimulus‐without‐replacement methods. Lower correspondence across administrations was observed for the free‐operant and response‐restriction methods. Although differentially higher levels of problem behavior did not occur with a single method, lower levels were consistently observed with the free‐operant method. During Study 2, we evaluated the implications of lower coefficients on reinforcer efficacy by comparing an initially identified and an immediately identified high‐preference stimulus in a reinforcer assessment. Initially identified and immediately identified high‐preference stimuli were equally effective reinforcers, suggesting that fluctuations in preference do not necessarily affect reinforcer efficacy in practice.     

Cautionary note on the use of short state-anxiety measures

The report noted that the use of high item-remainder coefficients as the sole criterion in the selection of items for short state-anxiety scales from full scales can lead to inaccurate interpretation of results. It was demonstrated that the use of a four-item state-anxiety scale from the State-Trait Anxiety Inventory A-State scale based on high item-remainder coefficients changed relationships in a previously reported study which used full-scale scores. It was proposed that such changes in relationships were the result of selecting four items which happen to load on only one of three dimensions of the A-State scale. The criterion of high item-remainder coefficents alone may be inadequate in the formation of short state-anxiety scales.     

Current misuses of multiple regression for investigating bivariate hypotheses: an example from the organizational domain

By definition, multiple regression (MR) considers more than one predictor variable, and each variable’s beta will depend on both its correlation with the criterion and its correlation with the other predictor(s). Despite ad nauseam coverage of this characteristic in organizational psychology and statistical texts, researchers’ applications of MR in bivariate hypothesis testing has been the subject of recent and renewed interest. Accordingly, we conducted a targeted survey of the literature by coding articles, covering a five-year span from two top-tier organizational journals, that employed MR for testing bivariate relations. The results suggest that MR coefficients, rather than correlation coefficients, were most common for testing hypotheses of bivariate relations, yet supporting theoretical rationales were rarely offered. Regarding the potential impact on scientific advancement, in almost half of the articles reviewed (44 %), at least one conclusion of each study (i.e., that the hypothesis was or was not supported) would have been different, depending on the author’s use of correlation or beta to test the bivariate hypothesis. It follows that inappropriate decisions to interpret the correlation versus the beta will affect the accumulation of consistent and replicable scientific evidence. We conclude with recommendations for improving bivariate hypothesis testing.