MULTIPLE ASSESSMENT OF MANAGERIAL EFFECTIVENESS: INTERRATER AGREEMENT AND CONSENSUS IN EFFECTIVENESS MODELS

Research has consistently identified poor interrater agreement among multiple assessments of managerial performance. Three alternative sources of dissensus in the effectiveness ratings were examined: rating errors, selective perceptions, and variations in criteria type or weight. As the available empirical evidence and theoretical analysis show, all three causes provide plausible reasons—though in varying degrees—for the low agreement coefficient. However, an empirical study designed to test three specific hypotheses on criterion type and criterion weights found consensus in the effectiveness models of superiors, subordinates, and peers. Consensus among different raters was high on both the role behaviors and on the personal traits of the managers as criteria for effectiveness. While these findings supported Biddle's role theory (1979), disagreement on the relative weights of these criteria was evident. These observations underscore the need for further conceptualization on the preference functions of raters as a primary source of the low convergent validity coefficients among multiple raters. Further research is also desirable on contextual and cognitive factors that may lead to shifts in criterion type and criterion weight, as well as on actual rating error tendencies among different raters.     

Identifying and prioritizing factors influencing the selection of the top suppliers of e‐procurement using FDEMATEL and FANP

The aim of this study is to identify, rank, and determine the importance of factors influencing the selection of the top supplier of electronic procurement in organizations using a hybrid approach. This is a case study of Irancell Telecommunication Service Company in which fuzzy decision‐making trial and evaluation (FDEMATEL) and fuzzy analytical network process (FANP) were used. FDEMATEL was employed to determine the interaction and interrelationships among the identified criteria and subcriteria, while utilizing FANP to calculate the weights of the criteria and their relevant subcriteria to determine their priorities. Therefore, the research literature was first reviewed to identify the factors influencing the selection of suppliers. For this purpose, the experts on communication networks were interviewed to select 16 final factors as the most important subcriteria that were then categorized as 5 groups (technology, environmental, services, organizational, and commercial) on which the study was based. After extracting the criteria for ranking the suppliers, codifying evaluation questionnaire, and converting the opinions in the Likert scale, 16 criteria were finally chosen with scores above the mean. FDEMATEL was used to determine the weights of subcriteria. In the final step, FANP was employed for prioritization. Results indicate that the organizational criterion is the most influential factor; however, the criterion services is the most permeable factor. The greatest weights were obtained from production capacity and provisions and geographical situation which were weighted 0.097 and 0.039, respectively. They are also regarded as the most and least important criteria for the selection of the top supplier of e‐procurement by the experts.     

The simultaneous prediction of any number of criteria by the use of a unique set of weights

When several criteria are available, it is ordinarily necessary (1) to select one of them asthe criterion, (2) to use several and thus arrive at several different sets of weights, or (3) to combine them into a single measure. A formula is derived for the determination of a unique set of weights. The use of these weights will produce the highest possible average coefficient of correlation between the various criteria and two (or more) weighted independent variables. If desired, the criteria may be assigned any predetermined weights. The weights then derived for the independent variables are such that the weighted average of the correlation coefficients between the various criteria and the independent variable composite will be a maximum. In the use of these formulas, no assumptions are necessary regarding the interrelationships existing among the criteria and it is not necessary to compute the intercorrelations among the criteria. A numerical example is included.     

Bayesian decision theory,subjective and objective probabilities,and acceptance of empirical hypotheses

It is argued that we need a richer version of Bayesian decision theory, admitting both subjective and objective probabilities and providing rational criteria for choice of our prior probabilities. We also need a theory of tentative acceptance of empirical hypotheses. There is a discussion of subjective and of objective probabilities and of the relationship between them, as well as a discussion of the criteria used in choosing our prior probabilities, such as the principles of indifference and of maximum entropy, and the simplicity ranking of alternative hypotheses.     

金钱激励在社会排斥影响不确定性决策中的调节作用

研究从金钱补偿的视角考察社会排斥对不确定性决策的影响以及金钱激励的调节作用。实验发现,有金钱激励时,被排斥者在爱荷华赌博任务(Iowa gambling task,IGT)中的行为分数(组块2和3的选牌净分数和总的盈利分数)显著高于社会接纳者,表现为风险规避; 在无金钱激励时,被排斥者在IGT任务中的行为分数(组块4和5的选牌净分数和总的盈利分数)显著低于被接纳者,表现为风险寻求。实验结果表明社会排斥对不确定性决策的影响受到金钱激励的调制。     

Candidates' Ability to Identify Criteria in Nontransparent Selection Procedures: Evidence from an assessment center and a structured interview

In selection procedures like assessment centers (ACs) and structured interviews, candidates are often not informed about the targeted criteria. Previous studies have shown that candidates' ability to identify these criteria (ATIC) is related to their performance in the respective selection procedure. However, past research has studied ATIC in only one selection procedure at a time, even though it has been assumed that ATIC is consistent across situations, which is a prerequisite for ATIC to contribute to selection procedures' criterion‐related validity. In this study, 95 candidates participated in an AC and a structured interview. ATIC scores showed cross‐situational consistency across the two procedures and accounted for part of the relationship between performance in the selection procedures. Furthermore, ATIC scores in one procedure predicted performance in the other procedure even after controlling for cognitive ability. Implications and directions for future research are discussed.     

Predicting Rehabilitation Success with the MMPI and Demographic Data

The purpose of this investigation was to assess the value of selected demographic and personality variables in the prediction of rehabilitation success of handicapped individuals. Demographic data and MMPI scores of 79 clients who had been served by a state Division of Vocational Rehabilitation (DVR) were used as predictor variables in a multiple-regression analysis. An attempt was made to predict three criteria: occupational level, upward mobility, and closure status. The multiple correlations for each of the three regression equations approached significance at the .05 level. Correlations significant at the .05 level, were found between age at onset, age at time of application, type of disability, and education, for one or more of the criteria. Significant beta weights were obtained for age at time of application, IQ, and the Mf, K, and Pd scales from the MMPI.     

Longitudinal Predictive Validity of a Learning Potential Test

The aim of this article is to provide empirical psychometric evidence of the (longitudinal) predictive validity of a learning potential measure—the Learning Potential Computerised Adaptive Test (LPCAT)—in comparison with standard static tests with school aggregate results as the criterion measure. Participants were 79 boys (mean age 12.44, SD = 0.44) and 72 girls (mean age 11.18, SD = 0.42) attending two private schools. Correlation and regression analyses were used to evaluate the predictive validity of the learning potential and standard test scores for school aggregate academic results as criterion measure. Results indicate that learning potential scores were statistically significant predictors of aggregate academic results and provided results that were comparable to those of the standard test results—providing empirical support for the use of learning potential tests in mainstream educational settings.     

Predicted differences and differences between predictions

WhenK tests are given toN individuals, and for each individual there are two criterion measures, then (1) the multiple regression weight to be applied to the standard score for each test to predict the criterion-difference score equals the difference of the weights for predicting each criterion separately; (2) the difference between the predicted scores equals the predicted difference (each test being assigned the appropriate multiple regression weight); (3) the square of the multiple correlation between predicted and actual criterion-difference scores equals the sum of squares of the multiple correlations of the battery with each criterion less the product of these correlations and the correlation between predicted scores all divided by twice the quantity one minus the criterion intercorrelation; and (4) the variance of errors of estimating the criterion-difference score equals the sum of the variances of errors of estimating each criterion score minus twice the criterion intercorrelation, plus twice the correlation between predicted scores multiplied by the product of the square root of one minus the variance of errors of estimating one criterion and the corresponding square root for the second criterion.The author wishes to express his appreciation for the suggestions and guidance given by Dr. Harold Gulliksen in the preparation of this article. He also wishes to acknowledge the helpful comments of Dr. Paul Horst and Dr. Ledyard Tucker on certain phases of the development.     

Multivariate relative importance: extending relative weight analysis to multivariate criterion spaces

For years, organizational scholars have sought effective ways to evaluate the importance of predictors included in a regression analysis. Recent techniques, such as general dominance weights and relative weights, have shown great promise for guiding evaluations of predictor importance. Nevertheless, questions remain regarding how one should investigate relative importance in the presence of a multidimensional criterion variable. The purpose of this article is to extend understanding of relative importance statistics to multivariate designs. The authors review the concept of relative importance and discuss a new procedure for calculating estimates of importance in the presence of multiple correlated criteria. Finally, a published correlation matrix is reanalyzed and a Monte Carlo simulation conducted to compare the new procedure with another technique for estimating importance. Unlike canonical solutions, which are often uninterpretable, the proposed multivariate relative weights provide an intuitive index regarding the relationship between predictors and criteria. Implications for organizational researchers are discussed.     

Accepting the null hypothesis

This article concerns acceptance of the null hypothesis that one variable has no effect on another. Despite frequent opinions to the contrary, this null hypothesis can be correct in some situations. Appropriate criteria for accepting the null hypothesis are (1) that the null hypothesis is possible; (2) that the results are consistent with the null hypothesis; and (3) that the experiment was a good effort to find an effect. These criteria are consistent with the meta-rules for psychology. The good-effort criterion is subjective, which is somewhat undesirable, but the alternative—never accepting the null hypothesis—is neither desirable nor practical.     

THE CONCEPT OF DYNAMIC CRITERIA: A CRITICAL REANALYSIS

Discussions of "the criterion problem" stress the assumedly frequent incidence of dynamic criteria. However, different concepts of dynamic criteria have not been distinguished. Three views of dynamic criteria are clarified in this paper. These are dynamic criteria conceptualized as (a) changes in group average performance over time, (b) changes in validity over time, and (c) changes in the rank-ordering of scores on the criterion over time. The evidence cited for each concept of dynamic criteria is critically analyzed and submitted to significance tests. The results of analyses of 735 r 's and 532 paired r 's across time from studies pertinent to dynamic criteria are reported. It is concluded that dynamic criteria are rare phenomena, with the significant changes found in key studies explainable by methodological artifacts. Though many sources suggest that dynamic criteria have dire consequences for the overall selection process, we argue that the phenomena have been over-emphasized in the literature, and that practitioners' concern should focus instead on removing sources of criterion unreliability.     

Attitudes Toward Prostitution and Acceptance of Rape Myths1

Rape myths and prostitution myths are a component of culturally supported attitudes that normalize violence against women. Prostitution myths justify the existence of prostitution, promote misinformation about prostitution, and contribute to a social climate that exploits and harms not only prostituted women, but all women. This study investigated the relationship between prostitution myth acceptance and rape myth acceptance in a sample of university undergraduates. Rape myth acceptance was positively correlated with prostitution myth acceptance among 783 university undergraduates from California, Iowa, Oregon, and Texas. College men were significantly more accepting of prostitution myths than were college women. Results suggest that acceptance of prostitution myths are a component of attitudes that justify violence against women.     

Peaks and Valleys: Predicting interests in leadership and managerial positions from personality profiles

This study investigates the relationship between personality and leadership and managerial interests at different levels of the vocational interest taxonomy. Personality scale scores from four different inventories were used to predict vocational interests of 574 adults. Influencing/enterprising interests, leadership and supervisory interests, and job‐specific managerial interests (e.g., CEO, Media Executive, Human Resources Director) served as criterion measures. A multiple regression‐based pattern recognition procedure recently devised by Davison and Davenport was applied to identify configurations of personality scores relating to these interest criteria. The personality profile pattern predictive of influencing and leadership interests was stable across different managerial domains. Results indicate that personality profile patterns drive the predictive power of personality scores, and that they explain a larger proportion of the variance in influencing and leadership interests compared with individuals' absolute trait levels.     

Item selection counts: a comparison of empirical key and rational scale validities in theory-based and non-theory-based item pools

Little explicit attention has been given to the impact of item pools on the validities and cross-validities of different background data scoring approaches. This study tests the idea that pools of items theoretically related to the performance of interest will outperform pools of items with no hypothesized relationship with the criterion. Validities and cross-validities of rational scales and empirical keys created from theory- and non-theory-based item pools were compared for 3 criteria. When size of the item pools was held constant, theory-based empirical keys (correlational and vertical percent) and rational scales showed larger validities and cross-validities than non-theory-based empirical keys (correlational and vertical percent) and showed minimal shrinkage in cross-validities. Even when item pool for the non-theory-based keys was expanded to include all items in the instrument, the theory-based keys showed comparable or slightly better validities and cross-validities for 2 of the 3 criteria, including college GPA, which was separated from the predictors by 4 years.     

Evaluating the Iowa Gambling Task as a direct assessment of impulsivity with low-income children

This study examined performance on the Iowa Gambling Task (IGT; Bechara, Damasio, Damasio, & Anderson, 1994) as a measure of low-income school-aged children’s affective decision-making and considered its utility as a direct indicator of impulsivity. One hundred and ninety-three 8–11 year olds performed a computerized version of the Iowa Gambling Task, a validated measure of decision-making. Multi-level modeling was used to examine children’s performance over the course of the task, with age, gender, and teachers’ ratings of child impulsivity (BIS-11; Patton, Stanford, & Barratt, 1995) used to predict children’s Iowa Gambling performance. Higher impulsivity scores predicted a decrease in slope of Iowa Gambling performance, indicating students rated higher on impulsivity chose more disadvantageously across the task blocks. Results support evidence of the validity of the Iowa Gambling Task as a measure of impulsivity in low-income minority children.     

Vocational Counselor Success Rate as a Criterion for Client Assignment

Although researchers have suspected that rehabilitation counselors may often experience different success rates with different kinds of clients, little effort has been made to determine the best methods of assigning clients to counselors. Our study represents an attempt to provide insight into the desirability of assigning clients to counselors with a high proportion of successful closures with members of the clients' disability group. The criterion of desirability was the Cooper-Rubin suitability-of-placement index. Results indicated no positive relationship between counselors' proportion of successful closures and their average Cooper-Rubin index scores.