A Meta-Analysis of the Relative Effects of Tabular and Graphic Display Formats on Decision-Making Performance

The results of 21 between-subjects experiments assessing the relative effects of tabular and graphic formats of data presentation on decision-making accu- racy were quantitatively reviewed using the meta-analysis procedure developed by Hunter, Schmidt, and Jackson (1982). Sampling error and differential range restriction accounted for variability in the average differences between tabular and graphic presentations, and there was no difference in the effects of the two presentation formats. These results counter recent assertions that task complexity moderates the effect of presentation format. The findings should also discourage further investigation of the effects of tabular and graphic formats using the conventional methodology.     

Is the meta-analysis of correlation coefficients accurate when population correlations vary?

One conceptualization of meta-analysis is that studies within the meta-analysis are sampled from populations with mean effect sizes that vary (random-effects models). The consequences of not applying such models and the comparison of different methods have been hotly debated. A Monte Carlo study compared the efficacy of Hedges and Vevea's random-effects methods of meta-analysis with Hunter and Schmidt's, over a wide range of conditions, as the variability in population correlations increases. (a) The Hunter-Schmidt method produced estimates of the average correlation with the least error, although estimates from both methods were very accurate; (b) confidence intervals from Hunter and Schmidt's method were always slightly too narrow but became more accurate than those from Hedges and Vevea's method as the number of studies included in the meta-analysis, the size of the true correlation, and the variability of correlations increased; and (c) the study weights did not explain the differences between the methods.     

ESTIMATION OF EMPLOYMENT VALIDITIES BY LESS EXPERIENCED JUDGES

In a previous study, Schmidt, Hunter, Croll and McKenzie (1983) demonstrated that estimates of the validity of cognitive tests made by highly trained and experienced judges are more accurate than empirical estimates obtained from small-sample validity studies. The present study examined whether less experienced judges could also produce accurate estimates. Twenty-eight recent Ph.D.'s in I/O Psychology estimated observed validities for the same 54 job-test combinations used by Schmidt et al. (1983). The estimates of these judges contained about twice as much random error as the experts' estimates. Systematic error of the less experienced judges was also greater than that of the experts (.0732 vs .019). The systematic errors of the two sets of judges were in opposite directions: less experienced judges overestimated validities, on average, while experts underestimated them. The results show that the estimates of less experienced judges contain less information than those of experts, but also that averages of estimates of several less experienced judges are as accurate as those obtained from small-sample empirical studies.     

Blocking and pseudoblocking: the reply of Rattus norvegicus to Apis mellifera

Blaser, Couvillon, and Bitterman (2006) presented data obtained with honeybees that in principle challenged all traditional interpretations of blocking. They administered A + followed by either A + or + alone (where + indicates an unconditioned stimulus) and then tested on X. They observed less responding to X when they administered A + than when + alone was administered, a phenomenon they called "pseudoblocking". Here we examined pseudoblocking in a rat fear-conditioning preparation. In Experiment 1, using a control procedure that was similar to our usual blocking control, we obtained conventional blocking but failed to observe pseudoblocking in our analogue to Blaser et al.'s procedure. In Experiment 2, we used Blaser et al.'s control procedure and again failed to observe the pseudoblocking effect with rats when we used the experimental context as an analogue to the honeybee feeder used by Blaser et al. After reviewing their protocol and previously published studies from their laboratory, we hypothesized that the feeder that they treated as a training context probably served as a punctate cue. We also tested this possibility in Experiment 2, using a punctate cue as a surrogate feeder, and were now able to reproduce their pseudoblocking phenomena. Our results are consistent with a simple overshadowing account of pseudoblocking, within the framework of existing theories of associative learning, which is not applicable to the conventional blocking paradigm. Thus, blocking remains a real phenomenon that must be addressed by models of associative learning.     

The validity and appropriateness of methods, analyses, and conclusions in Rind et al. (1998): A rebuttal of victimological critique from Ondersma et al. (2001) and Dallam et al. (2001).

The authors respond to 2 victimological critiques of their 1998 meta-analysis on child sexual abuse (CSA). S. J. Dallam et al. (2001) claimed that B. Rind, P. Tromovitch, and R. Bauserman (1998) committed numerous methodological and statistical errors, and often miscoded and misinterpreted data. The authors show all these claims to be invalid. To the contrary, they demonstrate frequent bias in Dallam et al.'s criticisms. S. J. Ondersma et al. (2001) claimed that Rind et al.'s study is part of a backlash against psychotherapists, that its suggestions regarding CSA definitions were extrascientific, and that the moral standard is needed to understand CSA scientifically. The authors show their suggestions to have been scientific and argue that it is Ondersma et al.'s issue-framing and moral standard that are extrascientific. This reply supports the original methods, analyses, recommendations, and conclusions of Rind et al.     

Assessing the belief bias effect with ROCs: reply to Dube, Rotello, and Heit (2010)

Dube, Rotello, and Heit (2010) argued (a) that the so-called receiver operating characteristic is nonlinear for data on belief bias in syllogistic reasoning; (b) that their data are inconsistent with Klauer, Musch, and Naumer's (see record 2000-02818-008) model of belief bias; (c) that their data are inconsistent with any of the existing accounts of belief bias and only consistent with a theory provided by signal detection theory; and (d) that in fact, belief bias is a response bias effect. In this reply, we present reanalyses of Dube et al.'s data and of old data suggesting (a) that the receiver operating characteristic is linear for binary "valid" versus "invalid" responses, as employed by the bulk of research in this field; (b) that Klauer et al.'s model describes the old data significantly better than does Dube et al.'s model and that it describes Dube et al.'s data somewhat better than does Dube et al.'s model; (c) that Dube et al.'s data are consistent with the account of belief bias by misinterpreted necessity, whereas Dube et al.'s signal detection model does not fit their data; and (d) that belief bias is more than a response bias effect.     

Decision rules for selecting effect sizes in meta-analysis: a review and reanalysis of psychotherapy outcome studies

This study deals with some of the judgmental factors involved in selecting effect sizes from within the studies that enter a meta-analysis. Particular attention is paid to the conceptual redundancy rule that Smith, Glass, and Miller (1980) used in their study of the effectiveness of psychotherapy for deciding which effect sizes should and should not be counted in determining an overall effect size. Data from a random sample of 25 studies from Smith et al.'s (1980) population of psychotherapy outcome studies were first recoded and then reanalyzed meta-analytically. Using the conceptual redundancy rule, three coders independently coded effect sizes and identified more than twice as many of them per study as did Smith et al. Moreover, the treatment effect estimates associated with this larger sample of effects ranged between .30 and .50, about half the size claimed by Smith et al. Analyses of other rules for selecting effect sizes showed that average effect estimates also varied with these rules. Such results indicate that the average effect estimates derived from meta-analyses may depend heavily on judgmental factors that enter into how effect sizes are selected within each of the individual studies considered relevant to a meta-analysis.     

Implications of direct and indirect range restriction for meta-analysis methods and findings

Range restriction in most data sets is indirect, but the meta-analysis methods used to date have applied the correction for direct range restriction to data in which range restriction is indirect. The authors show that this results in substantial undercorrections for the effects of range restriction, and they present meta-analysis methods for making accurate corrections when range restriction is indirect. Applying these methods to a well-known large-sample empirical database, the authors estimate that previous meta-analyses have underestimated the correlation between general mental ability and job performance by about 25%, indicating that this is potentially an important methodological issue in meta-analysis in general.     

Neuroscience findings are consistent with appraisal theories of emotion; but does the brain "respect" constructionism?

I reject Lindquist et al.'s implicit claim that all emotion theories other than constructionist ones subscribe to a "brain locationist" approach. The neural mechanisms underlying relevance detection, reward, attention, conceptualization, or language use are consistent with many theories of emotion, in particular componential appraisal theories. I also question the authors' claim that the meta-analysis they report provides support for the specific assumptions of constructionist theories.     

Need for more evolutionary and developmental perspective on basic emotional mechanisms

Lindquist et al.'s meta-analysis focuses on adult humans; the authors' emotion model might be strengthened by considering research on infants and animals, highlighting the importance of the limbic system. Reliance on the James-Lange theory is questionable; emotions typically occur instantaneously, with dubious dependence on bodily feedback for affect. Stronger evidence for localization might be obtained using more precise emotion terms and alterative localization methods.     

A META-ANALYSIS OF CONDITIONAL REASONING TESTS OF AGGRESSION

James et al. (2005) reported an estimate of criterion-related validity (corrected only for dichotomization of criteria) of r = .44 across 11 conditional reasoning test of aggression (CRT-Aggression) validity studies. This meta-analysis incorporated a total sample size more than twice that of James et al. Our comparable validity estimate for CRT-Aggression scales predicting counterproductive work behaviors was r = .16. Validity for the current, commercially marketed test version (CRT-A) was lower (r = .10). These validity estimates increased somewhat (into the .24–.26 range) if studies using dichotomous criteria with low base rates were excluded from the meta-analysis. CRT-Aggression scales were correlated r = .14 with measures of job performance. As we differed with James et al. in some of our coding decisions, we reran all analyses using James et al.'s coding decisions and arrived at extremely similar results.     

Cohen's d Corrected for Case IV Range Restriction: A More Accurate Procedure for Evaluating Subgroup Differences in Organizational Research

Organizational and staffing researchers are often interested in evaluating whether subgroup differences exist (e.g., between Caucasian and African‐American individuals) on predictors of job performance. To investigate subgroup differences, researchers often will collect data from current employees to make inferences about subgroup differences among job applicants. However, the magnitude of subgroup differences (i.e., Cohen's d) within incumbent samples may be different (i.e., smaller) than the magnitude of subgroup differences in applicant samples because selection of applicants typically reduces the variance of scores on the predictors (i.e., because lower scoring applicants are not selected). If researchers seek to generalize a d value in an incumbent sample to the applicant population, they may use Bobko, Roth, and Bobko's (correcting the effect size of d for range restriction and unreliability, 2001) Case II or III correction. By extension, Hunter, Schmidt, and Le (implications of direct and indirect range restriction for meta‐analysis methods and findings, 2006) have proposed a Case IV correction, which is more realistic than Bobko et al.'s approach. Therefore, this paper develops a Case IV correction for d (i.e., d_c4). The simulation results showed that the d_c4 was generally accurate across 6,000 simulation conditions. Moreover, 2 published datasets were reanalyzed to show the influence of the Case IV correction on d. In addition, implications and future directions of the d_c4 are discussed.     

Integrity tests predict counterproductive work behaviors and job performance well: comment on Van Iddekinge, Roth, Raymark, and Odle-Dusseau (2012)

Examination of the Van Iddekinge, Roth, Raymark, and Odle-Dusseau (2012) meta-analysis reveals a number of problems. They meta-analyzed a partial database of integrity test validities. An examination of their coded database revealed that measures coded as integrity tests and meta-analyzed as such often included scales that are not in fact integrity tests. In addition, there were important deficiencies in their analytic approach relating to application of range restriction corrections and identification of moderators. We found the absence of fully hierarchical moderator analyses to be a serious weakness. We also explain why empirical comparisons between test publishers versus non-publishers cannot unambiguously lead to inferences of bias, as alternate explanations are possible, even likely. In light of the problems identified, it appears that the conclusions about integrity test validity drawn by Van Iddekinge et al. cannot be considered accurate or reliable.     

Item Variances and Median Splits: Some Encouraging and Reassuring Findings

ABSTRACT The article by Bissonnette, Ickes, Bernstein, and Knowles (1990) presented in this issue contains a series of simulations which vary widely in the degree to which they resemble the parameters that have been found in actual data. Their conclusions feature warnings based on simulations containing large confounds between trait extremity and item variance. In real data, however, the corresponding confounds typically have been small. When Bissonnette et al. used appropriately small confounds in their simulations, their results showed that median split techniques failed to inflate Type I error rates. The constructive implications of Bissonnette et al.'s simulations indicate that moderator-variable researchers should always perform a simple check for range restriction and for variance-extremity confounds, and that there is no danger of spurious conclusions as long as these checks conform to what has typically been found with real data.     

META-ANALYSIS OF PERSONALITY-JOB PERFORMANCE RELATIONS: A REPLY TO ONES, MOUNT, BARRICK, AND HUNTER (1994)

Tett, Jackson, and Rothstein's (1991) meta-analysis identified higher average correlations for personality in predicting job proficiency than did Barrick and Mount's (1991). Ones et al. suggest that discrepancies between the two studies involving the Big Five personality dimensions are due to certain procedural differences. In this reply, we show that their arguments do not adequately explain the noted discrepancies. We also show that, because personality traits correlate significantly with job performance both positively and negatively beyond chance levels, use of absolute values, contrary to Ones et al., is important in meta-analyses involving personality. Addressing all of Ones et al.3 statistical concerns, re-analysis of Tett et al.k main data set results in slightly lower mean validities (e.g., .24 vs.29 for fully corrected values based on confirmatory estimates), and renders non-significant the job analysis/no job analysis distinction found to be significant in the original study. Tett et al.'s main conclusions, however, remain unchanged. We suggest that Barrick and Mount's lower mean validities may be due to their averaging signed correlations, pooling exploratory and confirmatory findings, and to the use of different inclusion criteria for selecting source studies.     

Revisiting Case IV: A reassessment of bias and standard errors of Case IV under range restriction

In 2004, Hunter and Schmidt proposed a correction (called Case IV) that seeks to estimate disattenuated correlations when selection is made on an unmeasured variable. Although Case IV is an important theoretical development in the range restriction literature, it makes an untestable assumption, namely that the partial correlation between the unobserved selection variable and the performance measure is zero. We show in this paper why this assumption may be difficult to meet and why previous simulations have failed to detect the full extent of bias. We use meta‐analytic literature to investigate the plausible range of bias. We also show how Case IV performs in terms of standard errors. Finally, we give practical recommendations about how the contributions of Hunter and Schmidt (2004) can be extended without making such stringent assumptions.     

Human cognition and a pile of sand: a discussion on serial correlations and self-organized criticality

Recently, G. C. Van Orden, J. G. Holden, and M. T. Turvey (2003) proposed to abandon the conventional framework of cognitive psychology in favor of the framework of nonlinear dynamical systems theory. Van Orden et al. presented evidence that "purposive behavior originates in self-organized criticality" (p. 333). Here, the authors show that Van Orden et al.'s analyses do not test their hypotheses. Further, the authors argue that a confirmation of Van Orden et al.'s hypotheses would not have constituted firm evidence in support of their framework. Finally, the absence of a specific model for how self-organized criticality produces the observed behavior makes it very difficult to derive testable predictions. The authors conclude that the proposed paradigm shift is presently unwarranted.     

On the science of Rorschach research

Wood et al.'s (1999b) article contained several general points that are quite sound. Conducting research with an extreme groups design does produce effect sizes that are larger than those observed in an unselected population. Appropriate control groups are important for any study that wishes to shed light on the characteristics of a targeted experimental group and experimental validity is enhanced when researchers collect data from both groups simultaneously. Diagnostic efficiency statistics--or any summary measures of test validity--should be trusted more when they are drawn from multiple studies conducted by different investigators across numerous settings rather than from a single investigator's work. There should be no question that these points are correct. However, I have pointed out numerous problems with specific aspects of Wood et al.'s (1999b) article. Wood et al. gave improper citations that claimed researchers found or said things that they did not. Wood et al. indicated my data set did not support the incremental validity of the Rorschach over the MMPI-2 when, in fact, my study never reported such an analysis and my data actually reveal that the opposite conclusion is warranted. Wood et al. asserted there was only one proper way to conduct incremental validity analyses even though experts have described how their recommended procedure can lead to significant complications. Wood et al. cited a section of Cohen and Cohen (1983) to bolster their claim that hierarchical and step-wise regression procedures were incompatible and to criticize Burns and Viglione's (1996) regression analysis. However, that section of Cohen and Cohen's text actually contradicted Wood et al.'s argument. Wood et al. tried to convince readers that Burns and Viglione used improper alpha levels and drew improper conclusions from their regression data although Burns and Viglione had followed the research evidence on this topic and the expert recommendations provided in Hosmer and Lemeshow's (1989) classic text. Wood et al. oversimplified issues associated with extreme group research designs and erroneously suggested that diagnostic studies were immune from interpretive confounds that can be associated with this type of design. Wood et al. ignored or dismissed the valid reasons why Burns and Viglione used an extreme groups design, and they never mentioned how Burns and Viglione used a homogeneous sample that actually was likely to find smaller than normal effect sizes. Wood et al. also overlooked the fact that Burns and Viglione identified their results as applying to female nonpatients; they never suggested their findings would characterize those obtained from a clinical sample. Wood et al. criticized composite measures although some of the most important and classic findings in the history of research on personality recommend composite measures as a way to minimize error and maximize validity. Wood et al. also were mistaken about the elements that constitute an optimal composite measure. Wood et al. apparently ignored the factor-analytic evidence that demonstrated how Burns and Viglione created a reasonable composite scale, and Wood et al. similarly ignored the clear evidence that supported the content and criterion related validity of the EMRF. With respect to the HEV, Wood et al. created a z-score formula that used the wrong means and standard deviations. They continued to use this formula despite being informed that it was incorrect. Subsequently, Wood et al. told readers that their faulty z-score formula was "incompatible" with the proper weighted formula and asserted that the two formulas "do not yield identical results" and "do not yield HEV scores that are identical or even very close." These published claims were made even though Wood et al. had seen the results from eight large samples, all of which demonstrated that their wrong formula had correlations greater than .998 with the correct formula. At worst, it seems that Wood et al. (199     

Meta-analysis of the brief psychiatric rating scale factor structure

A meta-analysis (N=17,620; k=26) of factor analyses of the Brief Psychiatric Rating Scale (BPRS) was conducted. Analysis of the 12 items from Overall et al.'s (J. E. Overall, L. E. Hollister, & P. Pichot, 1974) 4 subscales found support for his 4 subscales. Analysis of all 18 BPRS items found 4 components similar to those of Overall et al. In a 5-component solution, a 5th activation component emerged but was best supported among samples of schizophrenic patients. The first 4 components appear to form the core of the BPRS factor structure. Results of the meta-analysis suggest 5 subscales (with items in parentheses): Affect (anxiety, guilt, depression, somatic); Positive Symptoms (thought content, conceptual disorganization, hallucinatory behavior, grandiosity); Negative Symptoms (blunted affect, emotional withdrawal, motor retardation); Resistance (hostility, uncooperativeness, suspiciousness); and Activation (excitement, tension, mannerisms-posturing).     

INCREASING THE ACCURACY OF CORRECTIONS FOR RANGE RESTRICTION: IMPLICATIONS FOR SELECTION PROCEDURE VALIDITIES AND OTHER RESEARCH RESULTS

The common practice in meta-analyses and in individual studies of correcting for direct range restriction even though range restriction is actually indirect has long been known to lead to undercorrection, but this error has been assumed to be small. Using validity generalization data sets for 4 jobs, this study calibrated this error by comparing meta-analysis results based on corrections for direct range restriction with the more accurate results from a recently developed method of correcting for indirect range restriction. It was found that, on average, correction for direct range restriction resulted in substantial underestimation of operational validities for both job performance measures (21%) and training performance measures (28%). In addition, 90% credibility values were on average underestimated by 38%–40%. In addition to the implications for personnel selection, these findings suggest that similar underestimation of important relationships has occurred in other areas of research, with potential implications for theory development.