Standard Errors of Multivariate Range-Corrected Validities

The multivariate rather than the univariate range correction is used for estimating unrestricted applicant population validities in many military test validity studies but not uniformly. A Monte Carlo approach compared the standard errors of range-corrected validities under various experimental conditions adhering to the assumptions underlying correction accuracy. The multivariate corrected validities had smaller standard errors than both the univariate-corrected validities and the unrestricted validities. We conclude that using the univariate correction could fail to reveal the most valid selection instrument and that the multivariate correction should be used when scores for relevant predictors are available for the unrestricted population.     

IMPLICATIONS OF THE MULTIDIMENSIONAL NATURE OF JOB PERFORMANCE FOR THE VALIDITY OF SELECTION TESTS: MULTIVARIATE FRAMEWORKS FOR STUDYING TEST VALIDITY

Although most studies of criterion-related validity focus on univariate relationships, the complex and multidimensional nature of the performance construct and the widespread use of multiple selection devices argue in favor of multivariate frameworks for evaluating validity. Using a Monte Carlo simulation we estimated the validity of general cognitive ability tests and personality tests in predicting "job performance," where performance is conceptualized as a composite of multiple performance measures (i.e., individual job task performance and organizational citizenship behaviors). The validity of a selection battery varies substantially as a function of the relative weight given to both predictors and criteria; the 95% confidence interval for validities ranged from .20 to .78. The effective weights given to performance dimensions accounted for 34% of the variance in selection battery validities; depending on precisely how "performance" is defined, the same test battery can have relatively high or relatively low levels of validity. Our model suggests that the way an organization defines job performance is a source of true and important variability in validities, and that the validity of selection tests for predicting complex performance criteria may show considerably less generalizability that current metaanalysis of univariate validities would suggest.     

A reevaluation of the absenteeism-job satisfaction relationship

Previous reviews of the relationship of employee absenteeism to job satisfaction have largely neglected the size of the relationships reported and the artifacts that can affect statistical tests of significance. This paper applies the F. L. Schmidt-J. E. Hunter (1977, Journal of Applied Psychology, 62, 529-540) model of validity generalization in assessing the nature and strength of the relationship of absence to attitudes. Issues concerning the reliability and validity of absence measures are addressed, correlations between absence and job satisfaction are compiled and summarized, and an agenda for future research is set out. Considering the reliability estimates reported for the Frequency, Attitudinal, and Time Lost indices, the Time Lost Index was found to be the most reliable (r xx = .66, SD = .28). Factor analyses of intercorrelations among absence measures provided tentative support for a voluntary-involuntary absenteeism distinction. Combining all measures of satisfaction and all measures of absences, the mean correlation between absence and attitudes is -.09 (SD = .13). In addition to more comprehensive theory-guided multivariate research, future studies should aim toward a reconceptualization of absenteeism as a construct to take into consideration the perceptions of the workers themselves.     

The MMPI–2 Restructured Clinical (RC) Scales and Redundancy: Response to Tellegen,Ben-Porath,and Sellbom

In this article, we examine 5 criticisms of Tellegen, Ben-Porath, and Sellbom (2009/this issue) about our study demonstrating the redundant relationships of the Restructured Clinical (RC) scales with extant MMPI–2 scales. We discuss differences in univariate versus multivariate comparisons of the RC scales and our “proxy” scales using their data. We show that (a) both the RC and extant proxy scales identified in our analyses account for most of the variance in the Clinical scales; (b) the proxy scales are redundant with the RC scales; (c) the proxy scales matched the 6 RC scales in accounting for variance in the Clinical scales exactly in three cases, differed by ≤.02 in 2 cases, and reached a maximum of .11 in one case; (d) the item overlap between RC1 and HEA is not at issue but rather their correlation with Scale 1; and (e) the evidence for the construct validity of the RC scales is weak using findings on the incremental validity of RC4 as illustrative.     

Compensatory and non‐compensatory multidimensional randomized item response models

Randomized response (RR) models are often used for analysing univariate randomized response data and measuring population prevalence of sensitive behaviours. There is much empirical support for the belief that RR methods improve the cooperation of the respondents. Recently, RR models have been extended to measure individual unidimensional behaviour. An extension of this modelling framework is proposed to measure compensatory or non‐compensatory multiple sensitive factors underlying the randomized item response process. A confirmatory multidimensional randomized item response theory model (MRIRT) is proposed for the analysis of multivariate RR data by modelling the response process and specifying structural relationships between sensitive behaviours and background information. A Markov chain Monte Carlo algorithm is developed to estimate simultaneously the parameters of the MRIRT model. The model extension enables the computation of individual true item response probabilities, estimates of individuals’ sensitive behaviour on different domains, and their relationships with background variables. An MRIRT analysis is presented of data from a college alcohol problem scale, measuring alcohol‐related socio‐emotional and community problems, and alcohol expectancy questionnaire, measuring alcohol‐related sexual enhancement expectancies. Students were interviewed via direct or RR questioning. Scores of alcohol‐related problems and expectancies are significantly higher for the group of students questioned using the RR technique. Alcohol‐related problems and sexual enhancement expectancies are positively moderately correlated and vary differently across gender and universities.     

Contextual Performance and the Job Satisfaction–Dissatisfaction Distinction: Examining Artifacts and Utility

The relationship between contextual performance and job satisfaction is reexamined by considering the distinction between scores on measures of satisfaction and measures of dissatisfaction. Data from four samples and two measures of job satisfaction suggest that scores on measures of both satisfaction and dissatisfaction provide significant incremental validity in predicting contextual performance, that the satisfaction–dissatisfaction distinction is valid at the facet level, and that the satisfaction–dissatisfaction distinction cannot be accounted for by the artifactual explanations previously proposed in the literature.     

Dispositional resistance to change: measurement equivalence and the link to personal values across 17 nations

The concept of dispositional resistance to change has been introduced in a series of exploratory and confirmatory analyses through which the validity of the Resistance to Change (RTC) Scale has been established (S. Oreg, 2003). However, the vast majority of participants with whom the scale was validated were from the United States. The purpose of the present work was to examine the meaningfulness of the construct and the validity of the scale across nations. Measurement equivalence analyses of data from 17 countries, representing 13 languages and 4 continents, confirmed the cross-national validity of the scale. Equivalent patterns of relationships between personal values and RTC across samples extend the nomological net of the construct and provide further evidence that dispositional resistance to change holds equivalent meanings across nations.     

Distinguishing multi-voxel patterns and mean activation: Why, how, and what does it tell us?

The introduction of multi-voxel pattern analysis (MVPA) to the functional magnetic resonance imaging (fMRI) community has brought a deeper appreciation for the diverse forms of information that can be present within fMRI activity. The conclusions drawn from MVPA investigations are frequently influenced by both the ability to decode information from multi-voxel patterns and mean activation levels. In practice, MVPA studies vary widely in why and how they test for differing overall response levels. In this article, I examine the place of univariate information in MVPA investigations. I first discuss the variety of interpretations given to finding univariate response differences. I go on to discuss some of the analysis approaches used to investigate and compare univariate and multivariate sources of information, which can illuminate their respective contributions. It will be important for the MVPA and general fMRI community to continue to discuss and debate these important issues.     

Multivariate Relationships Between Motor and Personality Measures in Culturally Deprived High School Pupils

The relationship between personality, as measured by the IPAT-HSPQ, and certain motor performance variables was studied by utilizing the multivariate technique of canonical correlation. Ss for the study were 91 culturally deprived high school students who were achieving well below their expected capacity. A significant canonical relationship was not found between the domains of personality and motor performance. Also, no significant relationships between motor sub-domains and personality were found. It was thus concluded that for the population under study, the domains of motor performance and personality were independent Some discussion of the multivariate technique of canonical correlation was offered and compared with univariate procedures.     

The Relationships Between First- and Second-Order Measures of Cultivation

In the research literature on the cultivation hypothesis, there are two types of measures: first order and second order. First-order measures require respondents to provide estimates of occurrences. Second-order measures assess respondents’ attitudes. This study seeks to determine whether this measurement distinction results in operationalizations that are independent or related to one another. It was found that the two are related to a weak degree at a general level, but stronger relationships are found on a contingent analysis. Subgroups defined by relevant situational variables and certain general variables (such as IQ and especially age) reveal significant differential patterns of relationships. Also, evidence was found of an asymmetric pattern on several topics where first-order estimates usually form a basis for second-order beliefs.     

Assessing readiness for change in adolescents with eating disorders

The Readiness and Motivation Interview (RMI) is a semistructured interview measure of readiness and motivation to change that can be used for all eating disorder diagnoses. The RMI has demonstrated excellent psychometric properties and has both clinical and predictive utility in adult samples. This study examined the psychometric properties of the RMI in a younger population, namely, 12- to 18-year-old girls with eating disorders. Study participants (N = 65) completed the RMI and measures of convergent, discriminant, and criterion validity. Adolescents with eating disorders were able to conceptualize and articulate their readiness for change and to report the extent to which change efforts were for themselves versus for others. RMI readiness profiles across eating disorder symptom domains in adolescents were comparable to those in adults, with higher reported readiness to change binge eating than to change dietary restriction or compensatory strategies. Differences in internal consistency between adult and adolescent samples are discussed. Interviewing adolescents early in treatment about readiness may assist clinicians in forming an alliance with this difficult-to-engage population, while also providing valuable information for treatment planning.     

Estimating Group-Level Relationships: General Recommendations and Considerations for the Use of Intraclass Correlation Coefficients

Purpose

In this investigation, we argue for why and how available intraclass correlation coefficients and other types of reliability estimates can be employed as sample-based reliability estimates within primary and meta-analytic studies when relationships between group-level phenomena are of interest.

Design/Methodology/Approach

Group-level correlations and reliability estimates were obtained from 46 studies examining organizational climate–performance relationships. We illustrate how the group-level reliability estimates can be used to correct correlations for predictor and criterion unreliability. Procedures are presented for computing the sampling variances of individually corrected correlations that account for sampling error in the group-level reliability estimates.

Findings

Support was found for the conservative nature of meta-analytic parameter estimates when group-level reliability information is sample-based as opposed to assumed population values. In addition, our analyses indicated that conclusions about substantive relationships between group-level variables can change based on availability of sample-based reliabilities within both primary and meta-analytic studies.

Implications

Results from this study suggest that researchers should rely on sample-based meta-analytic procedures when examining the generalizability of group-level relationships. This study also demonstrates the importance of using all available reliability information and accounting for sampling error in the reliability estimates when conducting meta-analyses at the group level of analysis.

Originality/Value

This study breaks ground by systematically examining the use of intraclass correlation coefficients as reliability estimates within group-level meta-analytic studies. Furthermore, illustrative analyses provide guidance to primary and meta-analytic researchers in regard to how to correct group-level correlations for unreliability in the predictor, criterion, or both whenever and in whatever proportions the artifact information is available.

     

USING RANDOM RATHER THAN FIXED EFFECTS MODELS IN META-ANALYSIS: IMPLICATIONS FOR SITUATIONAL SPECIFICITY AND VALIDITY GENERALIZATION

Combining statistical information across studies (i.e., meta-analysis) is a standard research tool in applied psychology. The most common meta-analytic approach in applied psychology, the fixed effects approach, assumes that individual studies are homogeneous and are sampled from the same population. This model assumes that sampling error alone explains the majority of observed differences in study effect sizes and its use has lead some to challenge the notion of situational specificity in favor of validity generalization. We critique the fixed effects methodology and propose an advancement–the random effects model (RE) which provides estimates of how between-study differences influence the relationships under study. RE models assume that studies are heterogeneous since they are often conducted by different investigators under different settings. Parameter estimates of both models are compared and evidence in favor of the random effects approach is presented. We argue against use of the fixed effects model because it may lead to misleading conclusions about situational specificity.     

A Longitudinal Investigation of Object Relations: Child-Rearing Antecedents,Stability in Adulthood,and Construct Validation

Object relations theories posit that a child's relationship with caregivers has implications for subsequent relationships. Object relations refer to mental representations of relationships. Longitudinal data and Westen's (1993) methodology were employed to investigate questions related to (a) child-rearing antecedents of adult object relations in a nonclinical population, (b) temporal stability of adult object relations, and (c) construct validation. A multivariate effect of child-rearing variables on adult object relations was found, and significant univariate effects were nearly all in the predicted direction. Temporal stability (across 10 years) of adult object relations received tentative support. Object relations correlated positively with interpersonal functioning and physical health, but these correlations tended to be nonsignificant. A positive relationship between object relations and psychological health was not found.     

Some symmetric,invariant measures of multivariate association

A distinction is drawn between redundancy measurement and the measurement of multivariate association for two sets of variables. Several measures of multivariate association between two sets of variables are examined. It is shown that all of these measures are generalizations of the (univariate) squared-multiple correlation; all are functions of the canonical correlations, and all are invariant under linear transformations of the original sets of variables. It is further shown that the measures can be considered to be symmetric and are strictly ordered for any two sets of observed variables. It is suggested that measures of multivariate relationship may be used to generalize the concept of test reliability to the case of vector random variables.     

Contemporary attitudes and their ideological representation in Flanders (Belgium), Poland,and the Ukraine

The present research tests the validity of a two‐dimensional cultural and economic right‐wing model of ideology, as well as the relationships between these dimensions and attitudes toward recent political issues. Opinions about environmentalism, war on terror, and European Union enlargement were selected as representative contemporary attitudes. The present research questions were investigated in a Flemish (N = 176), Polish (N = 93), and Ukrainian (N = 93) sample. The results revealed that two dimensions underlie the ideological spectrum in all samples, one referring to cultural conservatism and right‐wing authoritarianism (RWA), and one referring to economic conservatism and social dominance orientation (SDO). Although the validity of differentiating between these ideological dimensions has been amply illustrated in studies on Western samples, the replication of these dimensions in former communist countries add to their validity and generality, because these countries have a specific political culture and history. However, important cross‐cultural differences with respect to the representation of the contemporary attitudes in this two‐dimensional ideological space were obtained as well. Whereas the war on terror attitude was meaningfully related to the cultural right‐wing dimension in all samples, the relationships for the environmentalism and EU enlargement attitudes were sample dependent. These cross‐cultural differences were explained in terms of saliency of the issue (nonsignificant versus significant relationships in the case of environmentalism) and pragmatic concerns in terms of profits and costs (in the case of EU enlargement). In the discussion it is also argued that the closer correspondence between the ideological representation of these attitudes in Polish and Western society may be caused by the fact that communism was less successful in penetrating political mentality in Poland than in the Ukraine.     

Refining individualized consideration: Distinguishing developmental leadershipand supportive leadership

This study explores the theoretical and empirical distinction between developmental leadership and supportive leadership, which are currently encompassed in a single sub dimension of transformational leadership, individualized consideration. Items were selected to assess these constructs, and hypotheses regarding the differential effects of developmental and supportive leadership were proposed. Confirmatory factor analyses provided support for the proposed distinction between developmental and supportive leadership, although these leadership factors were very strongly associated. Structural equation modelling and multi‐level modelling results indicated that both developmental leadership and supportive leadership displayed unique relationships with theoretically selected outcome measures. Developmental leadership displayed significantly stronger relationships with job satisfaction, career certainty, affective commitment to the organization and role breadth self‐efficacy than did supportive leadership. Results provide initial evidence in support of the discriminant validity of these two types of leadership. Discussion focuses on the need to further examine the construct of developmental leadership.     

Where (who) are collectives in collectivism? Toward conceptual clarification of individualism and collectivism

In psychological research on cultural differences, the distinction between individualism and collectivism has received the lion's share of attention as a fundamental dimension of cultural variation. In recent years, however, these constructs have been criticized as being ill-defined and "a catchall" to represent all forms of cultural differences. The authors argue that there is a conceptual confusion about the meaning of ingroups that constitute the target of collectivism. Collectives are rarely referred to in existing measures to assess collectivism. Instead, networks of interpersonal relationships dominate the operational definition of "ingroups" in these measures. Results from a content analysis of existing scales support this observation. To clarify and expand the individualism-collectivism distinction, a theoretical framework is proposed that draws on M. B. Brewer and G. Gardner's (1996) conceptualization of individual, relational, and collective selves and their manifestation in self-representations, beliefs, and values. Analyses of data from past studies provide preliminary support for this conceptual model. The authors propose that this new theoretical framework will contribute conceptual clarity to interpretation of past research on individualism and collectivism and guide future research on these important constructs.     

The Ecological Validity of Neuropsychological Tests: A Review of the Literature on Everyday Cognitive Skills

Evaluating the ecological validity of neuropsychological tests has become an increasingly important topic over the past decade. In this paper, we provide a comprehensive review of the research on the ecological validity of neuropsychological tests, as it pertains to everyday cognitive skills. This review is presented in the context of several theoretical issues facing ecological validity research. Overall, the research suggests that many neuropsychological tests have a moderate level of ecological validity when predicting everyday cognitive functioning. The strongest relationships were noted when the outcome measure corresponded to the cognitive domain assessed by the neuropsychological tests. Several other factors that may moderate the degree of ecological validity established for neuropsychological tests are in need of further exploration. These factors include the effects of the population being tested, the approach utilized (verisimilitude vs. veridicality), the person completing the outcome measure (significant other vs. clinician), illness severity, and time from injury until evaluation. In addition, a standard measurement of outcome for each cognitive domain is greatly needed to allow for comparison across studies.