Bias and error in multiple-choice tests

A formula for estimating real scores on a multiple-choice test from a knowledge of raw scores is derived. This formula does not involve the assumption of a binomial distribution of real scores as does the Calandra formula. Other important formulas derived show: the variance of real scores in terms of the variance of raw scores and the correlation between real scores and raw scores. If the variance of real scores (or of raw scores also) is binomial, the regression of real scores on raw scores is linear; but, otherwise the regression is curvilinear. Yet the linear estimating formula is a close approximation to the curvilinear relationship. Factors affecting the regression of real scores on raw scores and the correlation coefficient are: (1) the number of choices per question; (2) the number of questions answered; (3) the ratio of the average group raw score to the variance of raw scores.     

Bias and error in human judgment

Currently prevalent views of human inference are contrasted with an integrated theory of the epistemic process. The prevailing views are characterized by the following orienting assumptions: (1) There exist reliable criteria of inferential validity based on objectively veridical or optimal modes of information processing. (2) Motivational and cognitive factors bias inferences away from these criteria and thus enhance the likelihood of judgmental error. (3) The layperson's epistemic process is pluralistic; it consists of a diverse repertory of information-processing strategies (heuristics, schemas) selectively invoked under various circumstances. By contrast, the present analysis yields the following conclusions: (1) There exist no secure criteria of validity. (2) Psychological factors that bias inferences away from any currently accepted criteria need not enhance the likelihood of error. (3) The inference process may be considered unitary rather than pluralistic. The various strategies and biases discussed in the literature typically confound universal epistemic process with specific examples (or contents) of such processes. Empirical support for the present analysis is presented, including evidence refuting proposals that specific contents of inference are of universal applicability; evidence suggesting that people do not, because of a reliance on subnormative heuristics, underutilize nonnative statistical information—rather, people seem unlikely to utilize any information if it is nonsalient or (subjectively) irrelevant; and evidence demonstrating that the tendency of beliefs to persevere despite discrediting information can be heightened or lowered by introducting appropriate motivational orientations.     

Researchers misunderstand confidence intervals and standard error bars

Little is known about researchers' understanding of confidence intervals (CIs) and standard error (SE) bars. Authors of journal articles in psychology, behavioral neuroscience, and medicine were invited to visit a Web site where they adjusted a figure until they judged 2 means, with error bars, to be just statistically significantly different (p < .05). Results from 473 respondents suggest that many leading researchers have severe misconceptions about how error bars relate to statistical significance, do not adequately distinguish CIs and SE bars, and do not appreciate the importance of whether the 2 means are independent or come from a repeated measures design. Better guidelines for researchers and less ambiguous graphical conventions are needed before the advantages of CIs for research communication can be realized.     

Variation of the standard error of measurement

As usually interpreted, the standard error of measurement is assumed to be constant throughout the test-score range. In this investigation the standard error of measurement was assumed to be not higher than a second-degree function of the test score. By conceiving a test score to be made up of the scores on two parallel tests, an equation was derived for predicting the standard error of measurement from the test score. In the derivation the corresponding first four moments of the score distributions for the parallel tests were assumed to be identical, and certain errors of estimate involved in predicting the second test score from the first were assumed to be uncorrelated with powers of the score on the first test. An empirical verification was carried out, using nine synthetic tests and a 1000-case sample, and showed good agreement between predicted and observed results. The findings indicated that the standard error of measurement was constant only for a symmetrical, mesokurtic distribution of scores.This study was carried out while the author was a National Research Council Predoctoral Fellow in psychology at Princeton University.The author wishes to express his appreciation for the guidance given by his thesis adviser, Professor Harold Gulliksen. He wishes also to acknowledge his gratitude to the Educational Testing Service for extensive assistance in the empirical phase of the study, and to Dr. Ledyard Tucker for suggesting efficient methods of handling special computational problems.     

Bootstrap standard error and confidence intervals for the correlations corrected for indirect range restriction

The standard Pearson correlation coefficient, r, is a biased estimator of the population correlation coefficient, ρ(XY) , when predictor X and criterion Y are indirectly range-restricted by a third variable Z (or S). Two correction algorithms, Thorndike's (1949) Case III, and Schmidt, Oh, and Le's (2006) Case IV, have been proposed to correct for the bias. However, to our knowledge, the two algorithms did not provide a procedure to estimate the associated standard error and confidence intervals. This paper suggests using the bootstrap procedure as an alternative. Two Monte Carlo simulations were conducted to systematically evaluate the empirical performance of the proposed bootstrap procedure. The results indicated that the bootstrap standard error and confidence intervals were generally accurate across simulation conditions (e.g., selection ratio, sample size). The proposed bootstrap procedure can provide a useful alternative for the estimation of the standard error and confidence intervals for the correlation corrected for indirect range restriction.     

Giving and receiving social support at work: The roles of personality and reciprocity

Social support is an important variable in occupational stress research and theory, yet little is know about the factors that influence the amount of social support one gives, and receives at work. We examined personality (extraversion, neuroticism, and agreeableness) and reciprocity as potential antecedents to giving and receiving support from co-workers. Data collected from 108 participants employed in a number of organizations indicated that giving social support was positively associated with the receipt of social support. Both extraversion and agreeableness predicted giving and receiving non-job support and positive work-related support. The relationship between personality and social support received was, in many cases, mediated by social support given.     

Bias and error rates for premorbid IQ estimators: comment on Veiel and Koopman (2001)

H. O. F. Veiel and R. F. Koopman (2001) advance statistical and legal theses. They correctly point out that the usual regression formula for estimating a pre-event IQ underestimates high IQs and overestimates low IQs (due to regression to the mean). They call this a conditional bias and show it can be sizeable. The author takes issue with their claim that a new estimator they propose should be used in place of the usual formulas, because it negates this statistical bias. Their argument against the usual estimator conflates statistical bias and legal bias. Their discussion in favor of their new estimator mentions, but does not derive a general formula for, a gross loss of precision entailed by use of the new estimator. The author quantifies this loss of precision and, using Veiel and Koopman's numerical example, shows that their estimator quadruples error.     

Parent-adolescent reciprocity in negative affect and its relation to early adult social development.

The purposes of this 9-year, prospective longitudinal study were (a) to investigate hypothesized reciprocal growth in negative emotions between parents and adolescents and (b) to examine the influence of this reciprocal process on the development of social relationships during early adulthood. The results showed that both parents' and adolescents' initial levels of negative emotion toward each other predicted the rate of growth and rate of change in growth of expressed negative affect. In addition, the analyses indicated that an adolescent's enmeshment in reciprocal negativity in the family of origin carried over into early adult social relationships. The findings demonstrate the reciprocal nature of negative affect in interactions between parents and adolescents and suggest that family experience with this interactional style may have an adverse influence on the development of early adult social relationships.     

On the standard error of the modified biserial correlation

A paradoxical implication of Kraemer's expression for the large-sample standard error of Brogden's form of the biserial correlation is identified, and a new expression is given which does not imply the paradox. However, numerical evidence is presented which calls into question the correctness of the expression.     

Tables of the standard error of tetrachoric correlation coefficient

Tables are given forσ _r √N for the tetrachoric correlation coefficient for the following values of the correlation in the population: .00, ±.10, ±.20, ..., ±.80, ±.90, ±.95.     

Testing reciprocity in social interactions: A comparison between the directional consistency and skew-symmetry statistics

In the present article, we focus on two indices that quantify directionality and skew-symmetrical patterns in social interactions as measures of social reciprocity: the directional consistency (DC) and skew-symmetry indices. Although both indices enable researchers to describe social groups, most studies require statistical inferential tests. The main aims of the present study are first, to propose an overall statistical technique for testing null hypotheses regarding social reciprocity in behavioral studies, using the DC and skew-symmetry statistics (Phi) at group level; and second, to compare both statistics in order to allow researchers to choose the optimal measure depending on the conditions. In order to allow researchers to make statistical decisions, statistical significance for both statistics has been estimated by means of a Monte Carlo simulation. Furthermore, this study will enable researchers to choose the optimal observational conditions for carrying out their research, since the power of the statistical tests has been estimated.     

A worksheet for tetrachoric r and standard error of tetrachoric r using Hayes diagrams and tables

A worksheet simplifying the calculation of tetrachoric correlation coefficients and their standard errors is presented for use with Hayes' percentage difference method.We wish to thank Samuel P. Hayes, Jr. for his review and helpful suggestions leading to a simplification of the worksheet.     

Bootstrap standard error and confidence intervals for the correlation corrected for range restriction: a simulation study

The standard Pearson correlation coefficient is a biased estimator of the true population correlation, rho, when the predictor and the criterion are range restricted. To correct the bias, the correlation corrected for range restriction, rc, has been recommended, and a standard formula based on asymptotic results for estimating its standard error is also available. In the present study, the bootstrap standard-error estimate is proposed as an alternative. Monte Carlo simulation studies involving both normal and nonnormal data were conducted to examine the empirical performance of the proposed procedure under different levels of rho, selection ratio, sample size, and truncation types. Results indicated that, with normal data, the bootstrap standard-error estimate is more accurate than the traditional estimate, particularly with small sample size. With nonnormal data, performance of both estimates depends critically on the distribution type. Furthermore, the bootstrap bias-corrected and accelerated interval consistently provided the most accurate coverage probability for rho.     

Dissociations between Covariation Bias and Expectancy Bias for Fear-relevant Stimuli

The illusory correlation paradigm of Tomarken, Mineka, and Cook (1989) was used to examine bias in threat appraisal for biological (snake, spider) and technological (gun, knife) fear-relevant stimuli. Subjects showed bias in terms of higher on-line shock expectancy ratings and skin conductance responses, and higher post-experimental judgements of shock covariation, for fear-relevant stimuli. However, there were multiple dissociations between the measures: (1) expectancy bias was observed for both biological and technological stimuli, whereas covariation bias was restricted to biological stimuli; (2) prior fear of the target stimulus had no effect on expectancy bias, but covariation bias was restricted to high fear subjects; and (3) covariation bias was observed at the end of the experiment, by which time expectancy bias had disappeared. These results suggest that covariation bias is not simply a continuation of a pre-experimental expectancy bias.     

In medio stat virtus: theoretical and methodological extremes regarding reciprocity will not explain complex social behaviors

Guala contests the validity of strong reciprocity as a key element in shaping social behavior by contrasting evidence from experimental games to that of natural and historic data. He suggests that in order to understand the evolution of social behavior researchers should focus on natural data and weak reciprocity. We disagree with Guala's proposal to shift the focus of the study from one extreme of the spectrum (strong reciprocity) to the other extreme (weak reciprocity). We argue that the study of the evolution of social behavior must be comparative in nature, and we point out experimental evidence that shows that social behavior is not cooperation determined by a set of fixed factors. We argue for a model that sees social behavior as a dynamic interaction of genetic and environmental factors.     

Social roles and social perception: individual differences in attribution and error

In a replication of the social roles experiment by Ross, Amabile, and Steinmetz (1977), 103 male and female 14-year-olds viewed a brief videotape that showed a randomly assigned "questioner" asking general knowledge questions of a "contestant," who answered most of them incorrectly. Subjects subsequently manifested the role-based attribution error of attributing significantly more knowledge and ability to the arbitrarily designated questioner than to the arbitrarily designated contestant, and this effect was stronger among girls than boys. Personality data were available on the subjects for when they were in nursery school and again at age 14 in the form of independent Q-sort ratings. Personality correlates of the role effect were stronger among boys but manifested a basically similar pattern among girls and suggested that those most prone to make this attribution error, far from being disadvantaged in social judgment, tended to be more socially engaged and competent as well as emotionally well adjusted. The role effect was also positively associated with self-esteem. Personality correlates of attributional generosity and the tendency to attribute high ability to stimulus persons were associated with generally positive interpersonal qualities and attitudes. Results were interpreted in terms of usually appropriate heuristic social competencies that, in special circumstances, may sometimes underlie attribution errors. The general usefulness of examining individual differences in research on social cognition was noted.