Individual Differences in Course Choice Result in Underestimation of the Validity of College Admissions Systems

ABSTRACT— We demonstrate that the validity of SAT scores and high school grade point averages (GPAs) as predictors of academic performance has been underestimated because of previous studies' reliance on flawed performance indicators (i.e., college GPA) that are contaminated by the effects of individual differences in course choice. We controlled for this contamination by predicting individual course grades, instead of GPAs, in a data set containing more than 5 million college grades for 167,816 students. Percentage of variance accounted for by SAT scores and high school GPAs was 30 to 40% lower when the criteria were freshman and cumulative GPAs than when the criteria were individual course grades. SAT scores and high school GPAs together accounted for between 44 and 62% of the variance in college grades. This study provides new estimates of the criterion-related validity of SAT scores and high school GPAs, and highlights the care that must be taken in choosing appropriate criteria in validity studies.     

The (Non)Impact of Differential Test Taker Engagement on Aggregated Scores

Disengaged test taking tends to be most prevalent with low-stakes tests. This has led to questions about the validity of aggregated scores from large-scale international assessments such as PISA and TIMSS, as previous research has found a meaningful correlation between the mean engagement and mean performance of countries. The current study, using data from the computer-based version of the PISA-Based Test for Schools, examined the distortive effects of differential engagement on aggregated school-level scores. The results showed that, although there was considerable differential engagement among schools, the school means were highly stable due to two factors. First, any distortive effects of disengagement in a school were diluted by a high proportion of the students exhibiting no non-effortful behavior. Second, and most interestingly, disengagement produced both positive and negative distortion of individual student scores, which tended to cancel out much of the net distortive effect on the school’s mean.     

中学生自我隐瞒倾向:因素结构与发展特点

本研究旨在探讨自我隐瞒量表在中学生样本中的适用性 ,以及中学生自我隐瞒倾向的特点。一个由 51 3名中学生组成的样本接受了自我隐瞒量表和中学生应付方式量表的测试。结果发现中文版自我隐瞒量表具有较好结构效度和较好的效标效度以及较好的信度 ,因此中文版自我隐瞒量表可在今后的有关研究中使用。研究还发现男生的自我隐瞒倾向同女生相比相对较高 ,而且无论男生和女生 ,中学生的自我隐瞒倾向显著地高于大学生。这些结果可能表明 ,处于心理“断乳期”的中学生的心理具有一定闭锁性 ,而男生的自我隐瞒倾向高于女生 ,则可能表明自我隐瞒跟男生的独立性较强有关     

The Role of Socioeconomic Status in SAT-Grade Relationships and in College Admissions Decisions

This article examines the role of socioeconomic status (SES) in the relationships among college admissions-test scores, secondary school grades, and subsequent academic performance. Scores on the SAT (a test widely used in the admissions process in the United States), secondary school grades, college grades, and SES measures from 143,606 students at 110 colleges and universities were examined, and results of these analyses were compared with results obtained using a 41-school data set including scores from the prior version of the SAT and using University of California data from prior research on the role of SES. In all the data sets, the SAT showed incremental validity over secondary school grades in predicting subsequent academic performance, and this incremental relationship was not substantially affected by controlling for SES. The SES of enrolled students was very similar to that of specific schools' applicant pools, which suggests that the barrier to college for low-SES students in the United States is a lower rate of entering the college admissions process, rather than exclusion on the part of colleges.     

Reliability and validity of the direct observation form of the child behavior checklist

This article reports reliability and validity data for the Direct Observation Form (DOF) of the Child Behavior Checklist. Observational data were collected on two samples of boys aged 6-11 in classroom settings. Interobserver agreement was high: r = .92 for behavior problem score and r = .83 for on-task score. Generalizability, as measured by the one-way intraclass correlation, was .86 and .71 for behavior problem score and on-task score, respectively. In terms of validity, DOF scores correlated significantly and in the expected directions with teacher-reported problem behavior, school performance, and adaptive functioning. In addition, boys who had been referred by their teachers due to problem behavior obtained significantly higher behavior problem scores and significantly lower on-task scores than a matched sample of normal boys observed in the same classrooms.     

Do Traditional Admissions Criteria Reflect Applicant Creativity?

College admissions decisions have traditionally focused on high school academic performance and standardized test scores. An ongoing debate is the validity of these measures for predicting success in college; part of this debate includes how success is defined. One potential way of defining college success is a student's creative accomplishments. We tested the hypothesis that traditional admissions criteria fail to capture adequately the creativity of applicants by asking 610 college applicants to complete several creativity tasks. These included divergent thinking, caption‐writing, an essay, and self‐report measures of creativity in numerous domains. Creativity scores were compared to data from the college application, including high school rank, standardized test scores, and admissions interview scores. Results showed that traditional admissions criteria were only weakly related to creativity. Indeed, students who report the highest creative self‐efficacy can be perceived as weaker applicants according to traditional criteria. Findings are discussed in light of the goals of higher education to increase diversity of the student body and the abilities of its students.     

Construction and validation of a japanese adaptation of the test anxiety inventory

This study describes the construction and validation of a Japanese adaptation of Spielberger's (1980) Test Anxiety Inventory (TAI) and presents evidence of the reliability and validity of this new instrument. The items for the Japanese TAI (TAI-J) were selected on the basis of content validity and itemremainder correlations, which were .40 or higher for both sexes. Alpha reliability coefficients for the TAI-J Total scores were .90 or higher for both high school and college students; test-retest stability over a three-week interval was .89. Mean TAI-J Total scores for Japanese high school females were significantly higher than those of Japanese college students, whose scores were slightly lower than those reported for American undergraduates. TAI-J Total scores correlated .72 with a Japanese trait anxiety measure. Significant negative correlations were found between TAI-J Total scores and measures of academic achievement. Implications of these results for the measurement of test anxiety in Japanese students are discussed from a cross-cultural perspective.     

Converging on quality: Examining multiple measures of teaching effectiveness

The present study explores the convergent and predictive validity for several widely used measures of teaching quality from the Measures of Effective Teaching Project (Bill and Melinda Gates Foundation, 2009-2011). Specifically, the Classroom Assessment Scoring System (CLASS; Pianta, Hamre, & Mintz, 2012), the Framework for Teaching (FFT; Danielson Group, 2013), and the Tripod Student Perceptions Scale (Tripod; Ferguson, 2008) were examined. Correlations among measures were assessed by developmental level and content area (elementary mathematics N = 70; elementary English language arts N = 101; middle school mathematics N = 291, middle school English language arts N = 280). Both average scores and score variability (i.e., coefficient of variation) for the CLASS, FFT, and Tripod were used to predict value-added models (VAM), a high-stakes measure of students' academic growth. For elementary mathematics and ELA, findings indicated the CLASS and FFT exhibited moderate convergent validity while divergent validity was found between the Tripod and the CLASS and FFT. Across content areas in middle school grades, the CLASS, FFT, and Tripod exhibited moderate to high-moderate convergent validity. Average student and observer scores were positively related to VAM scores, whereas variability in scores demonstrated negative relations to VAM scores. Implications of findings for teacher evaluation and professional development are discussed.     

Longitudinal Predictive Validity of a Learning Potential Test

The aim of this article is to provide empirical psychometric evidence of the (longitudinal) predictive validity of a learning potential measure—the Learning Potential Computerised Adaptive Test (LPCAT)—in comparison with standard static tests with school aggregate results as the criterion measure. Participants were 79 boys (mean age 12.44, SD = 0.44) and 72 girls (mean age 11.18, SD = 0.42) attending two private schools. Correlation and regression analyses were used to evaluate the predictive validity of the learning potential and standard test scores for school aggregate academic results as criterion measure. Results indicate that learning potential scores were statistically significant predictors of aggregate academic results and provided results that were comparable to those of the standard test results—providing empirical support for the use of learning potential tests in mainstream educational settings.     

Validity of the Rey Visual Design Learning Test in primary and secondary school children.

The Rey Visual Design Learning Test (Rey, 1964, cited in Spreen & Strauss, 1991, Wilhelm, 2004) assesses immediate memory span, new learning, delayed recall and recognition for nonverbal material. Two studies are presented that focused on the construct validity of the RVDLT in primary and secondary school children. In the first study, primary school children performed the RVDLT and the Biber Figural Learning Test, as well as the WISC-R Block design Test, Boston Naming Test, and the Trailmaking Test, to assess discriminant validity. In the second study, the age range was expanded and the subtest Visual Reproductions of the Wechsler Memory Scale with a Delayed recall phase was used to assess the construct validity. A test for visual-motor integration and a test for attention, concentration, and speed of information processing were also added to complete the test battery for assessing discriminant validity. Moderate to high correlations were found between scores on the RVDLT and the tests used to assess construct validity. The correlational pattern of RVDLT scores and the scores on the discriminant tests is discussed.     

Test of a psychological cost-benefit model for career choice

A psychological cost-benefit model for career choice was applied to the choice situation after high school graduation. Especially tested were the construct validity and predictive validity of the components of the model. Psychological cost, benefit, and profit scales, with regard to continued education, were constructed on the basis of questionnaire data from 421 high school seniors. The analyses showed a clear, positive relationship between psychological benefit-profit and level of aspiration with regard to continued education. This outcome was regarded as an indication of construct validity for the components of the model. Moreover, groups differing as to post high school choice differed markedly, and in the expected direction, as to psychological cost-benefit-profit. Thus, the model showed high predictive validity with respect to post high school choice, which was also supported by a probability analysis. The results were, in general, more pronounced for boys than for girls.     

Psychometric Properties of the Emotion Awareness Questionnaire for Children in a French-Speaking Population

In this study, we investigated the psychometric properties of the French version of the Emotion Awareness Questionnaire (EAQ30; Rieffe et al., 2008). The EAQ30 was administered to 707 French-speaking children aged 8 to 16 years old. The original 6-factor structure was replicated in our data. The internal consistency coefficients of the EAQ30 subscales were satisfactory. We found small significant differences for gender and age. Regarding convergent validity, we found positive correlations between EAQ30 scores and emotional intelligence and negative correlations between EAQ30 scores and alexithymia. There was preliminary evidence of discriminant validity, with EAQ30 scores being weakly related to school performance, and concurrent validity, with EAQ30 scores being negatively related to somatic complaints, depression, and anxiety. Finally, except for 1 dimension, EAQ30 scores were not susceptible to social desirability. Although some weaknesses of the scale remain to be addressed, these findings support the use of the EAQ30 for research and clinical purposes.     

Examining the validity of self‐reports on scales measuring students' strategic processing

Background. Self‐report inventories trying to measure strategic processing at a global level have been much used in both basic and applied research. However, the validity of global strategy scores is open to question because such inventories assess strategy perceptions outside the context of specific task performance. Aims. The primary aim was to examine the criterion‐related and construct validity of the global strategy data obtained with the Cross‐Curricular Competencies (CCC) scale. Additionally, we wanted to compare the validity of these data with the validity of data obtained with a task‐specific self‐report inventory focusing on the same types of strategies. Sample. The sample included 269 10th‐grade students from 12 different junior high schools. Methods. Global strategy use as assessed with the CCC was compared with task‐specific strategy use reported in three different reading situations. Moreover, relationships between scores on the CCC and scores on measures of text comprehension were examined and compared with relationships between scores on the task‐specific strategy measure and the same comprehension measures. Results. The comparison between the CCC strategy scores and the task‐specific strategy scores suggested only modest criterion‐related validity for the data obtained with the global strategy inventory. The CCC strategy scores were also not related to the text comprehension measures, indicating poor construct validity. In contrast, the task‐specific strategy scores were positively related to the comprehension measures, indicating good construct validity. Conclusion. Attempts to measure strategic processing at a global level seem to have limited validity and utility.     

Psychometric Properties of the Bullying Participant Behaviors Questionnaire (BPBQ) among Chinese Middle School Students

The Bullying Participant Behaviors Questionnaire (BPBQ) is an efficient self-report measure for investigating bullying participant role behaviors. The present study evaluated the psychometric properties of the BPBQ in a Chinese middle school sample. A total of 516 middle school students (47.7% girls; age range?=?12–14 years) were recruited from an urban middle school in China. Results revealed that a five-factor model fit the data best. Correlations between the BPBQ subscale scores and the external criterion variables, including empathy and sympathy, moral disengagement, and trait anger, provided evidence of criterion validity. Furthermore, the BPBQ had good alpha reliability and moderate to good test-retest reliability. In conclusion, the BPBQ is a promising assessment tool to measure bullying participant behaviors among Chinese middle school students.

     

Psychometric characteristics of a multidimensional measure to assess impairment: The Child and Adolescent Functional Assessment Scale

The Child and Adolescent Functional Assessment Scale (CAFAS) is a multidimensional measure of degree of impairment in functioning. Interrater reliability data are presented for lay raters, graduate students, and frontline staff. Reliability was high for the total score and behaviorally-oriented scales. Construct, concurrent, and discriminant validity were assessed with the sample of children and adolescents evaluated at the Fort Bragg Demonstration Evaluation Project. Youth and their caregivers were evaluated via interview and selfcompleted instruments at four time points. Significant correlations were found between the CAFAS and other related constructs. Concurrent validity was demonstrated by logistic regression analyses examining the relationship between CAFAS ratings and problematic behaviors endorsed on measures completed by parents, teachers, or the youth. Youth with higher CAFAS total scores were much more likely to have poor social relationships, difficulties in school, and problems with the law. Discriminant validity was assessed with a repeated measures analysis of variance with intensity of care at intake and time as factors. Youth who were inpatients or in residential treatment centers at intake had higher CAFAS scores than those who were outpatients. These findings provide strong evidence for the reliability and validity of the CAFAS.     

Relationship of Child Abuse Potential Inventory Scores to Parental Responses:

The purpose of the study was to assess the relationship of child Abuse Potential (CAP) scores to parental responses given to child stimuli in analogue parenting situations. To assess the construct validity of the CAT, it was hypothesized that parent responses to analogue child situations would be judges as more controlling, punishing, aroused, and negative as CAP scores increased. Sixteen mothers from a local child abuse support group participated. The majority of mothers had not completed high school, had a mean income of $12,188, with small families containing a mean of 2.25children ranging in age from 6.9 years to 9.4 years. The results indicated that as CAP scores increased, parent responses were judges to be more controlling, more punishing, more highly aroused, and more rejecting of the child. No significant relationship between effect and CAP scores was present. Multiple regression analyses revealed that CAP scores and risk factors predicted parent verbal responses. CAP scores alone were more effective predictors of parent verbal behaviors than risk factors traditionally used to predict abusive parent responses. This study represented an advance because (1) an adult abusive sample was used and (2) independent ratings of parent verbal responses were obtained. Future research would benefit from the use of a larger, more heterogeneous sample and incorporation of direct observational data on parent-child interaction.     

Basic psychological need frustration scale: Adaptation and validation to active commuting to school in Spanish children and adolescents

Based on self-determination theory (SDT), the aims of this study were to adapt the Psychological Need Thwarting Scale to active commuting to and from school, as well as to gather information about validity and reliability of the Basic Psychological Need Frustration Scale in Active Commuting to and from School (BPNFS-ACS). A total of 285 children and adolescents, aged 10–17 years, participated (49.47 % girls; M_age = 12.88, SD_age = 2.16). Need satisfaction and frustration, as well as motivation for ACS were measured. The results of the confirmatory factor analysis supported the 12-item three-factor correlated model, which was invariant across gender and age. Convergent validity was met with suitable values for average variance extracted. Discriminant validity was obtained by acceptable values for the heterotrait–monotrait ratio of correlations and the correlations among the three latent factors. Reliability was also supported by adequate scores on Cronbach’s alpha, Raykov’s coefficient, and intra-class correlation coefficient. Criterion validity was evidenced by a negative prediction from need frustration to active commuting to school, and a positive association of need satisfaction with active commuting to school. Results support the use of the BPNFS-ACS for the purpose of gaining deeper insight into the “dark side” of motivation for active school transport among Spanish children and adolescents.     

Validity of the Woodcock-Johnson tests of cognitive ability: A comparison with the WISC-R in LD and normal elementary students

The study addresses the external validity of the Woodcock-Johnson Tests of Cognitive Ability (WJTCA) in learning disabled (LD) elementary school children by controlling for two methodological errors Woodcock identified in previous studies: (a) Intellectual ability range was restricted for both normal and LD samples to counteract an artificial inflation of mean WISC-R scores without concomitant effect on WJTCA scores, and (b) the WISC-R was readministered during data collection. In addition, normals were used as controls for LD students. WJTCA scores were correlated and compared with WISC-R scores and reading achievement test scores in 20 normal, 20 mild-to-moderate LD, and 20 severe LD third-, fourth-, and fifth-grade students. Results indicate comparability of mean WISC-R and WJTCA Full Scale scores in the normal sample, but manifest a significantly lower WJTCA Full Scale scores in the LD samples, despite a strong degree of correlation between the two tests in each sample. The significant linear trend of increasing mean WISC-R/WJTCA discrepancy across the severity of LD strongly suggests that the lower WJTCA scores in the LD samples is a function of the instrument's achievement emphasis and refutes the possibility of systematic error in the WJTCA norms. Results suggest that the WJTCA's achievement emphasis jeopardizes its validity for assessing and classifying LD students within the currently accepted and mandated ability-achievement discrepancy model of specific learning disabilities.     

Validity Stability Across Entering College Cohorts: Exploring the temporal generalizability of local validity estimates

Local validity studies rely on the assumption that validity estimates from one incumbent sample approximate validity for future applicant pools. We test this assumption using SAT scores and high school grades as predictors of first year college grade point average across multiple college applicant pools for over 100 schools. We present evidence for substantial absolute and rank order consistency in validity estimates. However, this consistency is far less than perfect, resulting in potentially meaningful utility differences over time. In addition, observed fluctuations are not fully explained by sampling error alone.