Improving the measurement of spatial aptitude by dynamic testing

Can spatial aptitude scores be increased by test-related training? Is construct validity influenced? The traditional psychometric view of test-related training as coaching, which threatens test validity, contrasts sharply with the view that performance modifiability measures a theoretically important construct that increases test validity (e.g., Vygotsky, 1978). This article reports on three studies of the dynamic testing of spatial ability in which training on the physical analogue of the mental folding task intervenes between a pretest and a posttest. Experiment 1 found that spatial ability can be significantly and substantially increased. Experiment 2 examined the impact of the dynamic testing procedure on the relationship of ability scores to other measures. Although sex differences in spatial ability were unchanged, the dynamic testing procedure increased predictive validity for training in text editing. Experiment 3 examined the impact on construct representation. The mathematical models of task difficulty indicated that the posttest more clearly represented rotational processing while the psychometric models supported the measurement of two abilities (i.e., initial status and gain). The implications for measuring learning ability by dynamic testing are discussed.     

When “reliability” fails,must a measure be discarded?—The case of Kinesthetic Aftereffect

Critics of Kinesthetic Aftereffect (KAE) recommend abandoning it as a personality measure largely because of poor test-retest reliability. Although no test can be valid if lacking true reliability, to discard a measure because of poor retest reliability is an oversimplification of validation procedures. This pitfall is exemplified here by a reexamination of KAE. KAE scores involve measures before (pretest) and after (test) aftereffect induction. Internal analysis of a KAE study showed: Differential bias is present; its locus is the second session pretest; its form makes second-session pretest scores functionally more similar to first- and second-session test scores and functionally more dissimilar to first-session pretest scores. Given this second session bias, the retest correlation tells us nothing about the true reliability of a one-session KAE score. However, if a measure possesses external validity, it must to some degree show true reliability. Based upon a literature review of one-session KAE validity studies, we conclude that one-session KAE scores are valid and hence show true reliability. KAE remains a promising personality measure.     

The relative error magnitude in three measures of change

Formulas for the standard error of measurement of three measures of change—simple difference scores, residualized difference scores, and the measure introduced by Tucker, Damarin, and Messick—are derived. Equating these formulas by pairs yields additional explicit formulas which provide a practical guide for determining the relative error of the three measures in any pretest-posttest design. The functional relationship between the standard error of measurement and the correlation between pretest and posttest observed scores remains essentially the same for each of the three measures despite variations in other test parameters (reliability coefficients, standard deviations), even when pretest and posttest errors of measurement are correlated.     

Review of Issues About Classical Change Scores: A Multilevel Modeling Perspective on Some Enduring Beliefs

Change scores obtained in pretest–posttest designs are important for evaluating treatment effectiveness and for assessing change of individual test scores in psychological research. However, over the years the use of change scores has raised much controversy. In this article, from a multilevel perspective, we provide a structured treatise on several persistent negative beliefs about change scores and show that these beliefs originated from the confounding of the effects of within-person change on change-score reliability and between-person change differences. We argue that psychometric properties of change scores, such as reliability and measurement precision, should be treated at suitable levels within a multilevel framework. We show that, if examined at the suitable levels with such a framework, the negative beliefs about change scores can be renounced convincingly. Finally, we summarize the conclusions about change scores to dispel the myths and to promote the potential and practical usefulness of change scores.     

Help for the Significant Others of Bulimics

A quasi-experimental design was used to assess the effects of a 5-week assistance program for the significant others of bulimics. A knowledge test, an attitude survey, and a behavior device were used to evaluate the effects of the program. One or more of the three instruments was administered to an experimental group of significant others before (pretest) and after (posttest) they attended the assistance program. One or more of the three instruments was also administered to a control group of significant others before (pretest) and after (posttest) a 5-week period. During this period they did not attend the assistance program. No significant difference was found between the pretest and posttest scores of the control group on the three instruments at the .05 probability level. Experimental group posttest scores were significantly higher than their pretest scores on the three instruments at the .05 probability level. An anonymous evaluation conducted following the program also indicated significant others who attended the program found it to be beneficial.     

The theory of test validity and correlated errors of measurement

In the theory of test validity it is assumed that error scores on two distinct tests, a predictor and a criterion, are uncorrelated. The expected-value concept of true score in the calssical test-theory model as formulated by Lord and Novick, Guttman, and others, implies mathematically, without further assumptions, that true scores and error scores are uncorrelated. This concept does not imply, however, that error scores on two arbitrary tests are uncorrelated, and an additional axiom of “experimental independence” is needed in order to obtain familiar results in the theory of test validity. The formulas derived in the present paper do not depend on this assumption and can be applied to all test scores. These more general formulas reveal some unexpected and anomalous properties of test validty and have implications for the interpretation of validity coefficients in practice. Under some conditions there is no attenuation produced by error of measurement, and the correlation between observed scores sometimes can exceed the correlation between true scores, so that the usual correction for attenuation may be inappropriate and misleading. Observed scores on two tests can be positively correlated even when true scores are negatively correlated, and the validity coefficient can exceed the index of reliability. In some cases of practical interest, the validity coefficient will decrease with increase in test length. These anomalies sometimes occur even when the correlation between error scores is quite small, and their magnitude is inversely related to test reliability. The elimination of correlated errors in practice will not enhance a test's predictive value, but will restore the properties of the validity coefficient that are familiar in the classical theory.     

Schooling and intelligence: A Methodological and logitudinal study

A general account of the adjustment problem is given within the frame of pretest-posttest design, and analysis of covariance, change score analysis and standardized change score analysis are discussed with reference to models of selection and growth. A Norwegian longitudinal investigation of the relationship between schooling and intelligence is described, and the results, generated by several adjustment techniques, showed substantial schooling effects. Thus the difference in estimated effect between a higher and a lower educational group varied from 49 to 73 per cent of the pretest standard deviation. In addition, considerable changes from pretest to posttest were observed, the mean gain amounting to 114 per cent of the pretest standard deviation. A comparison with two Swedish studies yielded similar trends in estimated schooling effect in that one study had higher and the other lower estimates than the present one.     

THE EFFECTS OF DIFFERENT REINFORCEMENT CONDITIONS ON THE TEST PERFORMANCE OF MULTIHANDICAPPED DEAF CHILDREN1

The present study explored the effects of different reinforcement conditions on the number of correct responses on the Raven Progressive Matrices. Four groups of 11- to 18-year-old multihandicapped deaf children matched on the basis of mean age and pretest scores were used. The groups were randomly assigned to any of four posttest conditions: end-of-session reinforcement, noncontingent reinforcement, delayed reinforcement, and immediate reinforcement. The mean posttest score of subjects tested under the immediate-reinforcement condition was significantly higher than that of any other group. No significant differences were observed between the mean posttest scores of the three other groups. The practical implications of using reinforcement procedures for testing purposes are discussed.     

The Use of Prospective Versus Retrospective Pretests with Child-Welfare Involved Families

The purpose of the current study is to examine the use of both a prospective and a retrospective pretest in evaluating the impact of the Strengthening Families Program on foster care involved families affected by parental substance abuse. Debate has existed in the literature for over 60 years regarding the use of retrospective pretests in assessing self-reported program impact, with key stakeholders often arguing strongly that, due to response-shift bias, sensitization, and the nature of many human service settings, retrospective pretesting may be more appropriate. However, program evaluators must also incorporate into the evaluation design funding mandates to collect data at specific points in time. In order to mitigate potential pretest-only biases and enhance the evaluation’s rigor, this study sought to address the pretest debate through the use of both a traditional, prospective pretest and a retrospective pretest. Using data provided by 411 caregivers, program effectiveness was measured in the areas of family, child and parent functioning. Statistical significance tests and effect sizes were analyzed to compare traditional prospective pretest to posttest scores and retrospective pretests to posttest scores. Findings indicate that overall, the Strengthening Families Program positively impacted family, child and parent functioning and that there were few differences between testing approaches when testing for statistical significance; however, relative differences between prospective pretests and retrospective pretests appeared more prominent in effect size computations. This research informs the longstanding debate, and suggests that program evaluators consider the advantages and disadvantages of using a retrospective pretest in design planning.     

Cognitive consequences of playing brain-training games in immersive virtual reality

The goal of the present study was to examine the effects of playing an immersive virtual reality game that included a collection of gamified cognitive tasks, Cerevrum, on specific components of cognition, including perceptual attention, mental rotation, working memory, visualization, visual field of view, and visual processing speed. Participants completed a pretest of cognitive assessments, played one of the two mini-games within Cerevrum (Stardust or Heroes) for 1.5 hr over three 30-min sessions and then completed a posttest of cognitive assessments and a questionnaire about interest and engagement during the game. An inactive control group completed only the pretest and posttest. Results showed no significant differences among the Heroes group, Stardust group, and control group on the posttest scores, even when controlled for pretest scores. These findings do not support the claim that playing brain-training games for a short period results in transfer of cognitive training to nongame venues.     

Instructor Feedback for College Writing Assignments in Introductory Classes

We compared three instructor feedback protocols to determine whether minimal instructor feedback could improve undergraduate students' writing skills in large introductory classes. The three feedback conditions were specific feedback, nonspecific feedback, and grade only. Sixty community college students participated in the study. Sections of an Introduction to Human Services course were randomly assigned to the three feedback conditions. Students wrote 9 timed essays during the semester. Pretest data were collected from the first essay at the beginning of the semester. Posttest data were taken from the final essay. Data for each student's essays consisted of the number of words written, the number of spelling errors, and the number of syntax grammar errors. Between the pretest and the posttest, students wrote 7 timed essays during class periods. Students in the specific feedback condition received a tally of spelling and grammar errors on grade criteria sheets and the location of the errors marked on their essays. In the nonspecific feedback condition, students received a tally of the errors on their grade criteria sheets but the locations of the errors were not marked on the essays. In the grade only condition, students received feedback on the number of points they had earned for each essay. Significant differences were found for pretest–posttest difference (gain) scores for the proportion of errors to the number of words written between the specific feedback condition and the grade only condition. Specific feedback decreased the proportion of errors to words written. The grade only condition maintained the same proportion of errors to words written from pretest to posttest. Significant differences were also found between difference (gain) scores for total errors between the specific feedback condition and the grade only condition and between the nonspecific feedback condition and the grade only condition.     

Effect of Concurrent Visual Feedback Frequency on Postural Control Learning in Adolescents

The purpose was to find better augmented visual feedback frequency (100% or 67%) for learning a balance task in adolescents. Thirty subjects were divided randomly into a control group, and 100% and 67% feedback groups. The three groups performed pretest (3 trials), practice (12 trials), posttest (3 trials) and retention (3 trials, 24 hours later). The reduced feedback group showed lower RMS in the posttest than in the pretest (p = 0.04). The control and reduced feedback groups showed significant lower median frequency in the posttest than in the pretest (p < 0.05). Both feedback groups showed lower values in retention than in the pretest (p < 0.05). Even when the effect of feedback frequency could not be detected in motor learning, 67% of the feedback was recommended for motor adaptation.     

Promoting balance and jumping skills in children with Down syndrome

The purpose of this study was to investigate the changes in balance and qualitative and quantitative jumping performances by 20 children with Down syndrome (3 to 6 years) on jumping lessons. 30 typical children ages 3 to 6 years were recruited as a comparison group. Before the jumping lesson, a pretest was given subjects for balance and jumping skill measures based on the Motor Proficiency and Motor Skill Inventory, respectively. Subjects with Down syndrome received 3 sessions on jumping per week for 6 weeks but not the typical children. Then, a posttest was administered to all subjects. Analysis of covariance showed the pre- and posttest differences on scores for floor walk, beam walk, and horizontal and vertical jumping by subjects with Down syndrome were significantly greater than those for the typical children.     

A consciousness-sampling analysis of test anxiety and performance

In order to evaluate cognitive-interference, reassertion, and reaction-to-performance models of test anxiety, 82 students completed the Test Anxiety Scale, provided state measures of anxiety just before and after a course examination, described their preparation for the test, and reported thought content and state anxiety up to six times during the test. Test Anxiety Scale scores were predictive of pre- and posttest state anxiety but not performance or problem-solving thought frequency during the test. Thought content was significantly but weakly correlated with performance, which was well correlated with posttest state anxiety but not with pretest anxiety. Pretest state anxiety was virtually uncorrelated with posttest state anxiety, with the correlations gradually declining during the test. Question-answering thought content correlated inversely with anxiety during the test. There was no group for whom anxiety appeared to facilitate performance. Preparation correlated only with performance. The pattern of results appears inconsistent with a cognitive-interference interpretation of test anxiety and suggests that in the naturalistic setting used, anxiety is more clearly an effect than a cause of poor performance.     

意义治疗的团体辅导对于戒毒者复吸倾向的影响

摘 要: 考察意义治疗的团体辅导对于戒毒者复吸倾向的干预效果。选择生命意义感较低的被试18名进入实验组,并选择与其分数匹配的18名被试组成对照组。后测结果表明,与控制组相比,实验组被试的价值感、效能感、成就感和生命意义感均有显著提升,复吸倾向显著降低。结论: 意义治疗的团体辅导能够显著提高戒毒人员的创造性价值类生命意义源和生命意义感,显著降低其复吸倾向。提升创造价值与对苦难的理解是提高生命意义感的有效途径。     

Development of a Unified Protocol for the Treatment of Emotional Disorders in Youth

This study evaluated the efficacy of the Mind Reading interactive computer software to remediate emotion recognition deficits in children with autism spectrum disorders (ASD). Six unmedicated children with ASD and 11 unmedicated non-clinical control subjects participated in the study. The clinical sample used the software for five sessions. The control sample was evaluated only on pre- and post-test performance to assess practice effects. Results showed that participants with ASD scored significantly higher on the posttest than on the pretest. As level of emotion difficulty increased, mean scores on the pretest decreased, indicating that difficulty level had a valid effect. The Mind Reading computer software appears to significantly improve the emotion recognition abilities in children with ASD.     

The assessment of knowledge and learning in competence spaces: The gain–loss model for dependent skills

The gain–loss model (GaLoM) is a formal model for assessing knowledge and learning. In its original formulation, the GaLoM assumes independence among the skills. Such an assumption is not reasonable in several domains, in which some preliminary knowledge is the foundation for other knowledge. This paper presents an extension of the GaLoM to the case in which the skills are not independent, and the dependence relation among them is described by a well‐graded competence space. The probability of mastering skill s at the pretest is conditional on the presence of all skills on which s depends. The probabilities of gaining or losing skill s when moving from pretest to posttest are conditional on the mastery of s at the pretest, and on the presence at the posttest of all skills on which s depends. Two formulations of the model are presented, in which the learning path is allowed to change from pretest to posttest or not. A simulation study shows that models based on the true competence space obtain a better fit than models based on false competence spaces, and are also characterized by a higher assessment accuracy. An empirical application shows that models based on pedagogically sound assumptions about the dependencies among the skills obtain a better fit than models assuming independence among the skills.     

小学一年级数感不良儿童的筛查与动态干预

首先编制测查小学一年级儿童数感发展水平的测验, 以筛查出数感发展不良儿童并进行数量转换和数量估计的动态干预。将数感不良儿童分为干预组(29人)和控制组(27人), 并以数感正常儿童作为对照组(40人), 以比较干预训练对儿童数感水平和与之密切相关的计算流畅性的促进作用。同时进一步考查数感和计算流畅性的作用方向。结果发现：(1)所编制的儿童数感发展测验信效度良好, 能有效评估一年级儿童的数感水平; (2)数感不良儿童经干预后, 其数感水平较控制组有了明显改善, 计算流畅性水平也显著提高, 甚至达到正常组儿童的水平; (3)数感对计算流畅性有明显的预测作用, 是计算流畅性的基础。     

Preliminary Validity of the Eyberg Child Behavior Inventory With Filipino Immigrant Parents

Filipinos are an understudied minority affected by significant behavioral health disparities. We evaluate evidence for the reliability, construct validity, and convergent validity of the Eyberg Child Behavior Inventory (ECBI) in 6- to 12-year-old Filipino children (N = 23). ECBI scores demonstrated high internal consistency, supporting a single-factor model (preintervention α = .91; postintervention α = .95). Results document convergent validity with the Child Behavior Checklist Externalizing scale at pretest (r = .54, p < .01) and posttest (r = .71, p < .001). We conclude that the ECBI is a promising tool to measure behavior problems in Filipino children.     

Developing Anticipation Skills in Tennis Using On-Court Instruction: Perception versus Perception and Action

On-court instruction involving either Perception–action training or Perception-only training was used to improve anticipation skill in novice tennis players. A technical instruction group acted as a control. Participants' ability to anticipate an opponent's serve was assessed pre- and posttest using established on-court measures involving frame-by-frame video analysis. The perception–action and perception-only groups significantly improved their anticipatory performance from pretest to posttest. No pretest-to-posttest differences in anticipation skill were reported for the technical instruction group. The ability to anticipate an opponent's serve can be improved through on-court instruction where the relationship between key postural cues and subsequent performance is highlighted, and both practice and feedback are provided. No significant differences were observed between the perception–action and perception-only training groups, implying that either mode of training may be effective in enhancing perceptual skill in sport.