THE EFFECTS OF RANGE RESTRICTION ON ESTIMATES OF CRITERION INTERRATER RELIABILITY: IMPLICATIONS FOR VALIDATION RESEARCH

This paper explores the implications of the fact that range restriction affects reliability coefficients as well as validity coefficients. Of concern was the possibility that, if range restriction produced an underestimate of criterion reliability, the use of such an estimate in correcting observed validity coefficients for attenuation could result in overestimating population validity, lb address this issue, 3 scenarios were considered: truncation on the predictor, truncation on an unobserved variable, and truncation on the criterion. The 3 scenarios were found to have substantial differences in their consequences for reliability and validity estimation. Although range restriction did result in an underestimate of criterion reliability, it is generally not the case that these underestimates will result in overestimates of validity if researchers make effective use of available information about the presence and degree of range restriction.     

Reliability and validity of retrospective self-reports of the age of pubertal onset using twin, sibling, and college student data

This study presents a methodology, derived from behavioral genetic research, designed to evaluate the validity of retrospective self-report inventories. The application of this technique to college student, twin, and sibling data demonstrated the reliability and validity of a retrospective self-report questionnaire on the timing of pubertal onset. Support was provided by four main findings: (1) substantial test-retest reliability coefficients for individual items (r ranged from .73 to .97) were obtained for a sample of college students; (2) nearly all of the monozygotic twin (MZ) intraclass correlations exceeded those for dizygotic twins (DZ); (3) the magnitude of the test-retest and intraclass correlations for recall of the timing of pubertal onset followed theoretical expectations for a trait demonstrating significant genetic influence (i.e., r for the same individual tested twice greater than r for MZs greater than r for DZs greater than r for siblings); (4) the expected secular trend of a cross-generational decrease in the age of sexual maturation was demonstrated for retrospective self-reports of family members representing three different generations. Moreover, the size of the mean differences across generations mirrored those reported for nonretrospective assessments of the actual age of pubertal onset. Finally, the value of retrospective self-report surveys in psychological research is discussed, and the relative merits of these instruments as indices of individual differences in pubertal development, as opposed to indices of the actual age of sexual maturation, are emphasized.     

The Role of Range Restriction and Criterion Contamination in Assessing Differential Validity by Race/Ethnicity

Purpose

Berry et al.’s (J Appl Psychol 96:881–906, 2011) meta-analysis of cognitive ability test validity data across employment, college admissions, and military domains demonstrated that validity is lower for Black and Hispanic subgroups than for Asian and White subgroups. However, Berry et al. relied on observed test-criterion correlations and it is therefore not clear whether validity differences generalize beyond observed validities. The present study investigates the roles that range restriction and criterion contamination play in differential validity.

Design/Methodology/Approach

A large dataset (N > 140,000) containing SAT scores and college grades of Asian, Black, Hispanic, and White test takers was used. Within-race corrections for multivariate range restriction were applied. Differential validity analyses were carried out using freshman GPA versus individual course grades as criteria to control for the contaminating influence of individual differences between students in course choice.

Findings

Observed validities underestimated the magnitude of validity differences between subgroups relative to when range restriction and criterion contamination were controlled. Analyses also demonstrate that validity differences would translate to larger regression slope differences (i.e., differential prediction).

Implications

Subgroup differences in range restriction and/or individual differences in course choice cannot account for lower validity of the SAT for Black and Hispanic subgroups. Controlling for these factors increased subgroup validity differences. Future research must look to other explanations for subgroup validity differences.

Originality

The present study is the first differential validity study to simultaneously control for range restriction and individual differences in course choice, and answers a call to investigate potential causes of differential validity.     

Doctrinal Orthodoxy, Religious Orientation, and Anthropocentrism

Anthropocentrism is a construct that may be defined as a dualistic worldview or set of ideals in which the individual views humanity as superior to other species within the environment and to nature, as a whole. A measure of anthropocentrism has been introduced as a means to assess the varying degrees of anthropocentric beliefs among individuals. In the initial validation studies of this scale, no significant differences in anthropocentrism were found to exist as a function of religion. The purpose of this investigation was to further examine the relationship between religion and anthropocentrism. More specifically, this study focused on doctrinal orthodoxy and religious orientation (Intrinsic, Extrinsic, and Quest) as possible correlates of anthropocentrism. Within the obtained sample of undergraduates (N=144), significant correlations were found to exist between anthropocentrism and the religious variables of interest, thus expanding the nomological network of related constructs, adding support to the construct validity of the Anthropocentrism Scale.     

An Idiographic Examination of Day-to-Day Patterns of Substance Use Craving,Negative Affect,and Tobacco Use Among Young Adults in Recovery

Psychological constructs, such as negative affect and substance use cravings that closely predict relapse, show substantial intraindividual day-to-day variability. This intraindividual variability of relevant psychological states combined with the “one day at a time” nature of sustained abstinence warrant a day-to-day investigation of substance use recovery. This study examines day-to-day associations among substance use cravings, negative affect, and tobacco use among 30 college students in 12-step recovery from drug and alcohol addictions. To account for individual variability in day-to-day process, it applies an idiographic approach. The sample of 20 males and 10 females (mean age = 21) was drawn from members of a collegiate recovery community at a large university. Data were collected with end-of-day data collections taking place over an average of 26.7 days. First-order vector autoregression models were fit to each individual predicting daily levels of substance use cravings, negative affect, and tobacco use from the same 3 variables 1 day prior. Individual model results demonstrated substantial interindividual differences in intraindividual recovery process. Based on estimates from individual models, cluster analyses were used to group individuals into 2 homogeneous subgroups. Group comparisons demonstrate distinct patterns in the day-to-day associations among substance use cravings, negative affect, and tobacco use, suggesting the importance of idiographic approaches to recovery management and that the potential value of focusing on negative affect or tobacco use as prevention targets depends on idiosyncratic processes.     

The reliability and validity of the Greek version of the task-specific occupational self-efficacy scale

The present study examined the reliability and validity of the Greek version of the Task-Specific Occupational Self-Efficacy Scale in a sample of 170 high school students. Drawing on current social cognitive career theory, the validity of the TSOSS is supported by the expected gender differences on the TSOSS factors, and their high correlations with Self-Directed Search summary scores, and vocational choice goals. Additionally, there seems to be some evidence that self-efficacy beliefs, as measured by the TSOSS, may affect choice goals both directly and indirectly. The empirical structure of the instrument examined by a cluster analysis algorithm was perfectly confirmed, and its stability is supported by adequate test-retest correlations and high internal consistency coefficients. On the basis of these findings a preliminary evaluation of the TSOSS is made when used with a Greek sample. Implications for research and career guidance are discussed.     

The effect of individual differences in act prototypicality judgments on trait-behaviour correlations and cross-situational consistency

Individuals differ in their perceptions of actprototypicality. This study examined whether incorporating such individual differences enhances trait-behaviour correlations and provides stronger evidence for cross-situational consistency. Three hundred and fifty-three subjects rated the dominance prototypicality of 100 acts, indicated how often they performed each of these acts, and provided trait ratings of how dominant they were in general. There were substantial and reliable individual differences in prototypicality judgements over a 4–5 month period. A variety of weighting schemes were used to incorporate these individual differences, but none dramatically increased the trait-behaviour correlation. Similarly, incorporating individual differences did not increase the magnitude of cross-situational consistency correlations. However, incorporating individual differences did enhance the pattern of trait-behaviour and consistency correlations from less prototypical to highly prototypical acts. Differences in perceptions of act prototypicality thus do not affect the magnitude of the correlations that can be obtained, but they are useful in revealing theoretically meaningful patterns of relationships.     

Criterion-related validity data for the child behavior checklist—Teacher''s report form

Criterion-related validity data for the Child Behavior Checklist—Teacher's Report Form are presented. These data were collected from a sample of elementary school boys. A total behavior problem score and individual factor scores were calculated from the checklist data and were related to indices of academic achievement and overall adjustment in the classroom. Significant correlations were established between the two sets of scores. The results were interpreted as supporting the criterion-related validity of the teacher-judgment measure and its use as a screening or diagnostic device in the school.     

The Relationship Quality Interview: evidence of reliability, convergent and divergent validity, and incremental utility

Relationship satisfaction and adjustment have been the target outcome variables for almost all couple research and therapies. In contrast, far less attention has been paid to the assessment of relationship quality. The present study introduces the Relationship Quality Interview (RQI), a semistructured, behaviorally anchored individual interview. The RQI was designed to provide a more objective assessment of relationship quality as a dynamic, dyadic construct across 5 dimensions: (a) quality of emotional intimacy in the relationship, (b) quality of the couple's sexual relationship, (c) quality of support transactions in the relationship, (d) quality of the couple's ability to share power in the relationship, and (e) quality of conflict/problem-solving interactions in the relationship. Psychometric properties of RQI ratings were examined through scores obtained from self-report questionnaires and behavioral observation data collected cross-sectionally from a sample of 91 dating participants and longitudinally from a sample of 101 married couples. RQI ratings demonstrated strong reliability (internal consistency, interrater agreement, interpartner agreement, and correlations among scales), convergent validity (correlations between RQI scale ratings and questionnaire scores assessing similar domains of relationship quality), and divergent validity (correlations between RQI scale ratings and (a) behavioral observation codes assessing related constructs, (b) global relationship satisfaction scores, and (c) scores on individual difference measures of related constructs). Clinical implications of the RQI for improving couple assessment and interventions are discussed.     

Response style and the revised NEO personality inventory: validity scales and spousal ratings in a Chinese psychiatric sample

The effects of response style bias on profile scores from the family of NEO scales and the resultant influence of response style on the predictive capacity of these scales continues to be debated. In this study, a large sample of Chinese psychiatric patients were categorized into four response style groups based on their scores from recently developed "validity" scales for the revised NEO Personality Inventory (NEO PI-R). Mean differences and correlations between self-report and spousal ratings of these patients were examined for the NEO PI-R domain and facet scales. Excessive positive self-presentation bias resulted in mean differences between the self-report and spousal ratings for N and E. Correlations between self-report and spousal ratings were reduced in patients engaging in positive self-presentational bias compared to those who were not so categorized on three of the five NEO PI-R scales. However, these results were manifest only in a sub-sample of psychotic patients. Negative self-presentational bias did not affect mean differences or diminish the correlations between the self-report and spousal ratings.     

Assessing collective affect recognition via the Emotional Aperture Measure

Curiosity about collective affect is undergoing a revival in many fields. This literature, tracing back to Le Bon's seminal work on crowd psychology, has established the veracity of collective affect and demonstrated its influence on a wide range of group dynamics. More recently, an interest in the perception of collective affect has emerged, revealing a need for a methodological approach for assessing collective emotion recognition to complement measures of individual emotion recognition. This article addresses this need by introducing the Emotional Aperture Measure (EAM). Three studies provide evidence that collective affect recognition requires a processing style distinct from individual emotion recognition and establishes the validity and reliability of the EAM. A sample of working managers further shows how the EAM provides unique insights into how individuals interact with collectives. We discuss how the EAM can advance several lines of research on collective affect.     

Individual differences in reading aloud: A mega-study,item effects,and some models

Normal individual differences are rarely considered in the modelling of visual word recognition – with item response time effects and neuropsychological disorders being given more emphasis – but such individual differences can inform and test accounts of the processes of reading. We thus had 100 participants read aloud words selected to assess theoretically important item response time effects on an individual basis. Using two major models of reading aloud – DRC and CDP+ – we estimated numerical parameters to best model each individual’s response times to see if this would allow the models to capture the effects, individual differences in them and the correlations among these individual differences. It did not. We therefore created an alternative model, the DRC-FC, which successfully captured more of the correlations among individual differences, by modifying the locus of the frequency effect. Overall, our analyses indicate that (i) even after accounting for individual differences in general speed, several other individual difference in reading remain significant; and (ii) these individual differences provide critical tests of models of reading aloud. The database thus offers a set of important constraints for future modelling of visual word recognition, and is a step towards integrating such models with other knowledge about individual differences in reading.     

-BRIEF REPORT: ON THE DANGER INVOLVED IN THE USE OF TESTS WHICH MEASURE FACTORS

relatively pure factors. Measures of factors developed through the use of relationships among individual differences are not necessarily measures of factors is the same sense when they are used as dependent variables in experiments designed to determine the effect of an independent variable. Until a measure of a factor (developed from correlations among individual differences) has been empirically demonstrated to be a factor in an experimental sense (correlated mean changes), the construct validity of measures of factors must remain zero when the measures are used as dependent variables in experiments. Factor analytic procedures have produced tests which purport to measure     

The Environmental Reward Observation Scale (EROS): development, validity, and reliability

Researchers acknowledge a strong association between the frequency and duration of environmental reward and affective mood states, particularly in relation to the etiology, assessment, and treatment of depression. Given behavioral theories that outline environmental reward as a strong mediator of affect and the unavailability of an efficient, reliable, and valid self-report measure of environmental reward, we developed the Environmental Reward Observation Scale (EROS) and examined its psychometric properties. In Experiment 1, exploratory factor analysis supported a unidimensional 10-item measure with strong internal consistency and test-retest reliability. When administered to a replication sample, confirmatory factor analysis suggested an excellent fit to the 1-factor model and convergent/discriminant validity data supported the construct validity of the EROS. In Experiment 2, further support for the convergent validity of the EROS was obtained via moderate correlations with the Pleasant Events Schedule (PES; MacPhillamy & Lewinsohn, 1976). In Experiment 3, hierarchical regression supported the ecological validity of the EROS toward predicting daily diary reports of time spent in highly rewarding behaviors and activities. Above and beyond variance accounted for by depressive symptoms (BDI), the EROS was associated with significant incremental variance in accounting for time spent in both low and high reward behaviors. The EROS may represent a brief, reliable and valid measure of environmental reward that may improve the psychological assessment of negative mood states such as clinical depression.     

Divergent, criterion-related, and discriminant validities for the Kuwait University Anxiety Scale

Three analyses were carried out. Analysis I estimated the correlation between the total score of the Kuwait University Anxiety Scale and a self-rating scale of happiness with large sample of male (n = 1,312) and female (n = 1,272) Kuwaiti adolescents. Pearson product-moment correlations were -.43 and -.44 (p < .001) for boys and girls, respectively, suggesting divergent validity of the anxiety scale. Analysis II examined the correlations between scores on the Kuwait University Anxiety Scale and the Somatic Symptoms Inventory for a nonclinical sample of women (n = 30), and female outpatients with anxiety disorder (n = 30). Pearson product-moment correlations were, respectively, .85 and .53, (p < .001), which support the criterion-related validity of the Kuwait University Anxiety Scale with respect to the Somatic Symptoms Inventory. In Analysis III a sample of 60 male and female outpatients with anxiety disorder and 60 nonclinical participants were matched on age, education, and occupation. All responded individually to the Kuwait University Anxiety Scale. Significant differences among groups, especially noticeable for men, support the discriminant validity of the scale. So, the clinical as well as research use of the scale can be recommended. By and large, the divergent, discriminant, and criterion-related validities of the scale have been adequately supported so subsequent replication is expected.     

The psychometric properties of the Slovenian translation of the Francis Scale of Attitude toward Christianity: a study among Roman Catholic undergraduate students

In order to contribute to a growing international research programme concerned with the correlations, antecedents and consequences of individual differences in attitude toward religion, the Francis Scale of Attitude toward Christianity was translated into Slovenian. Data provided by a sample of 808 undergraduate students affiliated with the Roman Catholic Church supported the internal consistency reliability and construct validity of this instrument and commended it for further use in studies conducted in Slovenia.     

Incremental validity of new clinical assessment measures

The authors address conceptual and methodological foundations of incremental validity in the evaluation of newly developed clinical assessment measures. Incremental validity is defined as the degree to which a measure explains or predicts a phenomenon of interest, relative to other measures. Incremental validity can be evaluated on several dimensions, such as sensitivity to change, diagnostic efficacy, content validity, treatment design and outcome, and convergent validity. Indices of incremental validity can vary depending on the criterion measures, comparison measures, and individual differences in samples. The authors review the rationale for, principles, and methods of incremental validation, including the selection of comparison and criterion measures, and address data analytic strategies and the conditional nature of incremental validity evaluations in the selection of measures. Incremental validity contributes to, but is different from, cost-benefits, which reflect the cost of acquiring the data and the benefits from the data. The impact of an incremental validity index on whether a measure is selected will be moderated by the cost of acquiring the new data, the importance of the measured phenomenon, and the clinical utility of the new data.     

Aggregation, acquiescence, and the assessment of trait affectivity

General affect ratings are subject to problems that lessen their validity. Some critics therefore have argued that aggregated on-line ratings provide better measures of trait affectivity. However, aggregated mood ratings suffer from two significant problems. First, correlations among same-valenced affects are substantially higher, indicating poorer discriminant validity. Second, cross-valence correlations are much weaker, reflecting a virtual absence of bipolarity. We show how both of these effects are due to an inflated acquiescence component in aggregated data. We also demonstrate that (a) these effects are a direct result of the aggregation process itself and (b) controlling for acquiescence counteracts these effects. These results indicate that the validity of aggregated mood ratings is lessened by an augmented acquiescence component.     

A successive approximation method of maximizing test validity

The ratio of item validity to item-total correlation can be used to select items which will tend to yield the maximum correlation with a criterion. Items to be retained are identified by comparing the ratio for each item with the validity of the original test. Further improvement of the validity in the experimental sample can be obtained by adding items to or removing items from the selected nucleus, according to recomputed ratios involving the correlations of the items with the nucleus and evaluated by means of a revised cut-off point. With slight variations, the method may be used for interest and personality tests as well as for aptitude material. The principal advantage over previous methods is that for any cycle of the analysis an exact cut-off point is provided.