Retesting in selection: a meta-analysis of coaching and practice effects for tests of cognitive ability

Previous studies have indicated that as many as 25% to 50% of applicants in organizational and educational settings are retested with measures of cognitive ability. Researchers have shown that practice effects are found across measurement occasions such that scores improve when these applicants retest. In this study, the authors used meta-analysis to summarize the results of 50 studies of practice effects for tests of cognitive ability. Results from 107 samples and 134,436 participants revealed an adjusted overall effect size of .26. Moderator analyses indicated that effects were larger when practice was accompanied by test coaching and when identical forms were used. Additional research is needed to understand the impact of retesting on the validity inferences drawn from test scores.     

Effects of a Private Room Versus Group Setting on Math Test Performance of College Students with ADHD

The effects of (a) private room, (b) typical classroom, and (c) high-distraction room test settings were examined on the math test performance of six college students (four females) with ADHD. Participants (aged 18–30 years) completed multiple, equivalent algebra calculation tests (100 items each) under the three test-setting conditions. A multi-element design was used to compare the number of items attempted (test access) and percentage correct (accuracy). Students also rated their level of distraction after each testing session on a Likert-type scale. Results showed that students found the high-distraction setting more distracting than other settings and that five of six students accessed more test items in the private room versus the high-distraction setting, with three of the six demonstrating nonoverlapping data across settings. A regular classroom setting did not produce a high level of distraction, nor did it restrict student access to the math tests relative to a private room setting. These findings suggest that some students with ADHD are affected negatively by highly distracting environments and may benefit from a quiet, private test setting. Single-case experimental designs may be useful for examining individual differences among students with disabilities receiving specific test accommodations.

     

Enhancing Digital Fluency through a Training Program for Creative Problem Solving Using Computer Programming

The purpose of this paper is to propose a training program for creative problem solving based on computer programming. The proposed program will encourage students to solve real‐life problems through a creative thinking spiral related to cognitive skills with computer programming. With the goal of enhancing digital fluency through this proposed training program, we investigated its effects. Two sets of experiments were performed in which 119 typical students and 30 younger, gifted students participated. Two synthetic creative problem solving tests, which had a high correlation with logical ability, scientific problem solving ability and divergent thinking ability, were developed to measure creative problem solving ability. We provided the treatment group with a paper‐based booklet with relevant problems developed specifically for that group. ANCOVA statistical procedures were used to analyze the pre‐ and post‐synthetic creative problem solving tests. The findings of our study are as follows: with typical students, the originality of the treatment group outperformed the control group, a result that was compatible with previous research. With gifted students, the fluency of the treatment group outperformed the control group, and overall creative problem solving ability was enhanced. Remarkably, fluency increased significantly, a notable difference from the results of prior studies. In conclusion, we inferred that, given the definition of digital fluency, if creative problem solving ability is enhanced by a training program for creative problem solving based on computer programming, digital fluency will ultimately be improved. In this paper, we discuss the result of fluency enhancement that contradicts prior research. We suggest that this training program could be a new learning environment for the students who have grown up with digital media.     

Reliability of a sentence completion measure of ego development.

Studied the reliability of the Washington University Sentence Completion Test by giving 51 9th graders and 26 college students the test twice, a week apart. For 9th graders the design included a test-retest group and two groups given half of the test at each session. Although test-retest correlations were high for the 9th graders, retest scores dropped significantly. With college students (a) test-retest correlations through positive and significant were lower, (b) retest scores did not change systematically, and (c) percentage agreement between test and retest scores was high. Discrepant results were related to motivational set and variance in test scores. Split-half correlations and internal consistency coefficients were high. Likelihood of lower retest scores makes problematic the use of this test for short term pretest-posttest studies seeking to stimulate ego development.     

The effects of metacognitive training versus worked‐out examples on students' mathematical reasoning

Background: The present study is rooted in a cognitive‐metacognitive approach. The study examines two ways to structure group interaction: one is based on worked‐out examples (WE) and the other on metacognitive training (MT). Both methods were implemented in cooperative settings, and both guided students to focus on the problem's essential parts and on appropriate problem‐solving strategies. Aims: The aim of the present study is twofold: (a) to investigate the effects of metacognitive training versus worked‐out examples on students' mathematical reasoning and mathematical communication; and (b) to compare the long‐term effects of the two methods on students' mathematical achievement. Sample: The study was conducted in two academic years. Participants for the first year of the study were 122 eighth‐grade Israeli students who studied algebra in five heterogeneous classrooms with no tracking. In addition, problem‐solving behaviours of eight groups (N = 32) were videotaped and analysed. A year later, when these participants were ninth graders, they were re‐examined using the same test as the one administered in eighth grade. Method: Three measures were used to assess students' mathematical achievement: a pretest, an immediate post‐test, and a delayed post‐test. ANOVA was carried out on the post‐test scores with respect to the following criteria: verbal explanations, algebraic representations and algebraic solution. In addition, chi‐square and Mann‐Whitney procedures were used to analyse cooperative, cognitive, and metacognitive behaviours. Results: Within cooperative settings, students who were exposed to metacognitive training outperformed students who were exposed to worked‐out examples on both the immediate and delayed post‐tests. In particular, the differences between the two conditions were observed on students' ability to explain their mathematical reasoning during the discourse and in writing. Lower achievers gained more under the MT than under WE condition. Conclusions: The findings indicate that the kind of task and the way group interaction is structured are two important variables in implementing cooperative learning, each of which is likely to have different effects on mathematical communication and achievement outcomes.     

RETEST EFFECTS IN OPERATIONAL SELECTION SETTINGS: DEVELOPMENT AND TEST OF A FRAMEWORK

This study proposes a framework for examining the effects of retaking tests in operational selection settings. A central feature of this framework is the distinction between within-person and between-person retest effects. This framework is used to develop hypotheses about retest effects for exemplars of 3 types of tests (knowledge tests, cognitive ability tests, and situational judgment tests) and to test these hypotheses in a high stakes selection setting (admission to medical studies in Belgium). Analyses of within-person retest effects showed that mean scores of repeat test takers were one-third of a standard deviation higher for the knowledge test and situational judgment test and one-half of a standard deviation higher for the cognitive ability test. The validity coefficients for the knowledge test differed significantly depending on whether examinees' test scores on the first versus second administration were used, with the latter being more valid. Analyses of between-person retest effects on the prediction of academic performance showed that the same test score led to higher levels of performance for those passing on the first attempt than for those passing on the second attempt. The implications of these results are discussed in light of extant retesting practice.     

IS CONSTRUCT DIFFERENTIATION LOQUACITY?

Recent studies have indicated that cognitive complexity scores, based on the quantity of constructs embedded in written essays, correlate significantly with the total number of words contained in the response. As a consequence, a controversy involving whether loquacity rather than cognitive complexity has been measured by such scoring procedures has emerged. In this study (1) the premise is advanced that the quantity of constructs approach represents subjects' willingness to write rather than personal construct systems or construal processes, (2) loquacity is explained in terms of motivational variables rather than construal processes and (3) two separate studies test the motivational hypothesis. Results of study J indicate that subjects promised credit for rigor write more words and more constructs than did their non-motivated counterparts. Moreover, the results of study 2 demonstrate that writing apprehension scores and self-reports of situational motivation predict both loquacity and cognitive complexity scores. These findings support a motivational hypothesis regarding loquacity and raise serious questions concerning cognitive complexity measurement.     

Tape‐assisted reciprocal teaching: Cognitive bootstrapping for poor decoders

Background: Students who have limited skills in decoding and comprehension and who lack motivation to read present difficulties for practitioners. Difficulties may be compounded when these students lack access to age‐appropriate and interesting text and have lost the notion of reading as a process of obtaining meaning from print. Aims: This research examined the effects of a modified reciprocal teaching intervention for readers with poor decoding skills and poor comprehension. Tapeassisted reciprocal teaching was used to help students with poor decoding skills develop cognitive and metacognitive strategies and improve their comprehension of high interest expository texts. Methods: Two single‐subject research design studies involving four groups of students were conducted. Study I involved one experimental group and Study II was a multiple baseline design involving three experimental groups. Sample: Each experimental group comprised a heterogeneous mix of six students, three with poor decoding skills and three with adequate decoding skills, all of whom showed poor comprehension. Results: As a result of the tape‐assisted reciprocal teaching, the poor decoders demonstrated improved application of cognitive and metacognitive strategies and improved comprehension. These improvements were shown on both researcherdeveloped and standardised tests as well as on maintenance and transfer measures. The students with adequate decoding skills also showed improvements in comprehension. Conclusions: The success of the intervention for poor decoders suggests that tape assisted reciprocal teaching may be seen as a form of ‘cognitive bootstrapping’ to enable poor readers to escape the cycle of reading failure and engage more meaningfully in the process of reading.     

Test anxiety and the validity of cognitive tests: A confirmatory factor analysis perspective and some empirical findings

The validity of cognitive ability tests is often interpreted solely as a function of the cognitive abilities that these tests are supposed to measure, but other factors may be at play. The effects of test anxiety on the criterion related validity (CRV) of tests was the topic of a recent study by Reeve, Heggestad, and Lievens (2009) (Reeve, C. L., Heggestad, E. D., & Lievens, F. (2009). Modeling the impact of test anxiety and test familiarity on the criterion-related validity of cognitive ability tests. Intelligence, 37, 34?41.). They proposed a model on the basis of classical test theory, and concluded on the basis of data simulations that test anxiety typically decreases the CRV. In this paper, we view the effects of test anxiety on cognitive ability test scores and its implications for validity coefficients from the perspective of confirmatory factor analysis. We argue that CRV will be increased above the effect of targeted constructs if test anxiety affects both predictor and criterion performance. This prediction is tested empirically by considering convergent validity of subtests in five experimental studies of the effect of stereotype threat on test performance. Results show that the effects of test anxiety on cognitive test performance may actually enhance the validity of tests.     

An Analysis of the Blind Variation and Selective Retention Theory of Creativity

This research study explores the relation between measures of creative potential and cognitive and motivational variables, including students' interests, perceptions, and classroom experiences. Surveys were administered to 373 students (N = 373), along with 2 questionnaires, namely the Student Information Sheet (SIS) and the Teacher Encouragement Scale (TES). The students were divided into three groups as High Creative (HC), Average Creative (AC) and Low Creative (LC) on the basis of composite test scores on the tests of creative thinking. The HC students liked Science and Mathematics subjects at school level, whereas the rest of the groups (AC and LC) preferred Indian languages Telugu and Hindi. Surprisingly, all the three groups disliked Social Studies subjects. The top 5 occupations chosen were doctor, teacher, lawyer, engineer, and police constable. When comparing the 3 groups, it is evident that each group had similar aspirations. A high percentage of students, including the HC, were found to be addicted to school achievement. They perceived academic failure as a bad experience in the class and struggled for academic excellence, which was their main goal and interest in life. The highly creative students had more favorable attitudes towards teacher encouragement than the other groups.     

Cognitive ability and motivational interventions: Their effects on performance outcomes

A laboratory experiment was conducted to examine the effects of cognitive ability, task interests, goal setting, and feedback on perceived competence, intrinsic motivation, performance, and satisfaction with performance. Data were collected from 90 introductory psychology students performing one of two computer-based tasks. Results indicated that the motivational interventions interacted with cognitive ability in relation to intrinsic motivation, perceived competence, and satisfaction with performance. Implications are discussed within the framework of cognitive resources theory and goal-setting theory.     

Activating children's thinking skills (ACTS): The effects of an infusion approach to teaching thinking in primary schools

Background Recent interest in the teaching of thinking skills within education has led to an increase in thinking skills packages available to schools. However many of these are not based on scientific evaluation ( DfEE, 1999 ). This paper endeavours to examine the effectiveness of one approach, that of infusion, to teaching thinking. Aims To investigate the impact of an infusion methodology, activating children's thinking skills (ACTS), on the cognitive, social, and emotional development of children in Year 4–6 in primary schools. This is a sister project to research being conducted in Northern Ireland ( McGuinness, 2006 ). Sample The study involved 404 children from 8 primary schools in one local authority. These were divided into 160 in the experimental group and 244 in the waiting list control group. Methods A quasi‐experimental design was used with pre‐, post‐, and delayed post‐tests to ascertain changes in children's cognitive abilities, self‐perceptions, and social/behavioural skills using quantitative measures. In addition qualitative techniques were used with pupils and teachers to evaluate effectiveness. Results The experimental group made significantly greater gains in cognitive ability skills over a 2 year period compared to the waiting list control. Qualitative data demonstrated a positive impact on children's social and emotional development. In addition teacher professional development was reported to be enhanced. Conclusions This research indicated that children's cognitive abilities can be developed following a 2 year period of the ACTS infusion intervention. While some positive effects were evidenced on the social and emotional development of children, further study will be necessary to examine these in more detail.     

Can competitive paradigms increase the validity of experiments on primate social cognition?

Experiments vary in their ability to distinguish between competing hypotheses. In tests on primate cognition the majority of this variation is due to an experimenter's ability to test primates in valid settings while providing the adequate amount of experimental control. While experimenters studying primate cognition can use methods of control perfected in captivity, it is still very unclear how to design and then objectively evaluate the external validity of new experimental paradigms. I recommend that more effort be allocated to specify how to create relevant test settings for primates. Primate social life is highly competitive. This means that all aspects of primates themselves, including their cognitive abilities, have likely been shaped by the need to out-compete conspecifics. Based on this hypothesis, sophisticated cognitive abilities of primates might best be demonstrated in competitive contexts. Thus, it is suggested that one possible measure of validity is whether investigators integrate a competitive component into their experimental designs. To evaluate this methodological prediction I review the literature on chimpanzee perspective-taking as a case study including several recent studies that include a competitive component in their experimental designs. Accepted after revision: 8 April 2001 Electronic Publication     

"When the age is in, the wit is out": Age-related self-categorization and deficit expectations reduce performance on clinical tests used in dementia assessment

This study examined the combined effects of age-based self-categorization and aging expectations on cognitive performance in a clinical context. An experimental study manipulated older adults' salient self-categorization as Younger or Older, as well as expectations that aging involves a specific memory decline versus generalized cognitive decline. Memory and general ability tests that are typically used in dementia screening were then administered. As predicted, self-categorization as Older dramatically reduced performance, but the measure on which this effect was revealed depended on aging expectations. Participants who self-categorized as Older and expected memory to decline performed worse on memory tests. Conversely, participants who self-categorized as Older and expected widespread cognitive decline performed worse on the general ability test. The clinical implications for the latter group were profound, because 70% met the diagnostic criterion for dementia, compared with an average of 14% in other conditions. The importance of self-categorization processes when interpreting performance on tests used to diagnose dementia are discussed. (PsycINFO Database Record (c) 2012 APA, all rights reserved).     

Individual and group differences in intelligence and speed of information processing

A battery of eight different reaction time (RT) tests, measuring the speed with which individuals perform various elementary cognitive processes, and a group test of scholastic aptitude (the Armed Services Vocational Aptitude Battery, ASVAB) were given to 50 black and 56 white male vocational college students. The regression of the general factor scores of the ASVAB on the RT measures yielded a shrunken multiple correlation of 0.465. Although discriminant analyses, when applied separately to the ASVAB subtests and to the RT variables, showed highly comparable overall discrimination (over 70% correct classification) between the black and white groups, factor scores derived from the general factor (labeled ‘speed of information processing’) of the RT battery show only about one-third as large a mean black-white difference as the mean group difference on the general factor scores derived from the ASVAB. Comparisons were also made between the 106 vocational college students and 100 university students of higher average academic aptitude who had previously been tested on the same RT battery (Vernon, 1983a). These groups showed marked differences on the RT variables, the largest differences occuring on the tests that required more complex cognitive processing. The more complex RT tests also correlate most highly with the psychometric measures of ability within each group. The results are consistent with the hypothesis that individual differences and the mean differences between groups in psychometric abilities and scholastic achievement are related to differences in the speed of information processing as measured in elementary cognitive tasks.     

Reducing Academic Procrastination Through a Group Treatment Program: A Pilot Study

The present study assessed a short-term group treatment program using cognitive interventions focused on students’ procrastination. A structured 90-min session program was used with 10 students (5 female, 5 male; M _age = 21.8, SD = 3.2) across 5 weeks. In the first and last session of the program participants completed a two reliable and valid procrastination scales, and then 8 weeks later in the follow up sessions filled out the same questionnaires. During the group sessions, participants identified their irrational thoughts as well as cognitive distortions associated with their procrastination tendencies. Results of a non-parametric Friedman Test revealed a significant decrease in participants’ academic procrastination score and general procrastination scores from the pre-test to follow-up test suggesting that the program was deemed to be successful.     

Relationship of laboratory and on-road tests for driving-school students and experienced drivers

There is a pressing need to evaluate driving ability without an actual driving test but no reliable, standardised procedure is available. This study investigated the resulting distortion performance evaluation for different driver groups in on-road tests due to driving skills and driving style. Two groups, 15 female driving school students and 25 experienced male drivers were examined using cognitive and psychomotor tests and standard personality questionnaires as well as an on-road driving test. The students made significantly fewer errors on the driving test than the experienced drivers. The former group's driving performance could be predicted by errors on the laboratory tests, along with personality factors, while experienced drivers' driving errors were best predicted by slowness on laboratory tests. This study suggests that, when a series of standard cognitive and psychomotor tests are considered, different measures may apply in predicting young people's ability to drive safely based on assessing driving ability of experienced drivers, assumed lowered due to distorted driving style.     

The validity of a persistence test

In order to raise the predictive efficiency of its college entrance test battery, the Educational Testing Service is working on the development of non-academic measures to supplement the standard aptitude and achievement examinations. A test of difficult number series problems was set up to measure persistence by tempting the students to give up early; the students were informed that some of the problems had no solution, and that full credit would be received by so marking them. This test was tried out and found to have some correlation with grades, while having no correlation with the other tests. Adding this test to the battery showed an appreciable rise in the battery's multiple correlation with grades.     

The Association Between Socioeconomic Status and Cognitive Development in Children Is Partly Mediated by a Chaotic Home Atmosphere

ABSTRACT

There are socioeconomic-status (SES) differences in cognitive development. Various factors have been proposed that might explain this association, and one of these factors is the home environment. The present study examined a chaotic home atmosphere as a potential mediator of the association between parental SES and cognitive development. A nationally representative sample of children in the United Kingdom was studied when children were 3 years (n = 15,590), 5 years (n = 13,802), and 7 years old (n = 12,661). At each wave, the children completed multiple cognitive tests, and parents provided information on their SES (income, education, and occupation) and the home atmosphere. Mediation effects were tested with longitudinal structural equation modeling. Direct relations between parental SES and cognitive ability were partly mediated by the home atmosphere. The proportion of mediation was 16% for the change in cognitive ability predicted by parental SES. This study suggests that a chaotic home atmosphere might partly explain the association between parental SES and cognitive development.