Evidence and inference in educational assessment

Educational assessment concerns inference about students' knowledge, skills, and accomplishments. Because data are never so comprehensive and unequivocal as to ensure certitude, test theory evolved in part to address questions of weight, coverage, and import of data. The resulting concepts and techniques can be viewed as applications of more general principles for inference in the presence of uncertainty. Issues of evidence and inference in educational assessment are discussed from this perspective.Probability isn't really about numbers; it's about the structure of reasoning.Presidential address to the Psychometric Society, presented June 25, 1994, in Champaign, Illinois.Supported by (1) Contract No. N00014-91-J-4101, R&T 4421573-01, from the Cognitive Science Program, Cognitive and Neural Sciences Division, Office of Naval Research, (2) the National Center for Research on Evaluation, Standards, Student Testing (CRESST), Educational Research and Development Program, cooperative agreement number R117G10027 and CFDA catalog number 84.117G, as administered by the Office of Educational Research and Improvement, U.S. Department of Education, and (3) the Statistical and Psychometric Research Division of Educational Testing Service. I am grateful for comments and suggestions from Henry Braun, Drew Gitomer, Richard Patz, Jonathan Troper, and Howard Wainer.     

Facilitated communication: Better education through applied ideology

Facilitated communication (FC), or the provision of graduated assistance by staff in the selection of keys on a communication device by people with developmental disabilities, has become a widely used training technique. Although widespread anecdotal accounts of substantial training benefits and unanticipated communicative competence have been reported, controlled experimental and single subject studies have disclosed little evidence of unexpected competencies. We review research on FC, identify cognitive factors that may stimulate acceptance of this as a training method, and suggest procedures to be employed in individual validation situations when critical communications occur.     

Relative absence of neuropsychological deficit in patients with low grade astrocytomas

Given the current accuracy and precision of modern brain imaging technology, there is presumed to be little utility in neuropsychological assessment procedures in patients with brain tumors. The primary exposure of many clinical neuropsychologists to patients with brain tumors is during their training, in the form of didactic classroom activities, such as reviewing the brain tumor cases of early investigators in the field. Historically, these brain tumors were the more aggressive and destructive tumors, such as grade III and IV astrocytomas, that could be identified with pre CT and pre MRI technology. With current imaging technology, low grade tumors that might previously have gone undiagnosed for years can be detected and patients followed over time. This series of cases represents documentation of the very unique neuropsychological status of patients with relatively slow growing, infiltrative brain tumors classified as grade II astrocytomas. The potential relevance of neuropsychological assessment for such cases is discussed.     

IDENTIFYING INSTRUCTIONAL TASKS THAT OCCASION PROBLEM BEHAVIORS AND ASSESSING THE EFFECTS OF STUDENT VERSUS TEACHER CHOICE AMONG THESE TASKS

Levels of problem behavior were assessed when 4 students with severe disabilities received instruction on preferred versus nonpreferred tasks and when tasks of each type were chosen by the teacher rather than by the student. In Phase 1, interview and direct observation assessments were conducted to identify relative preferences for academic tasks. In Phase 2, the effects of these lower preference and higher preference tasks on the rate of problem behavior were evaluated using a multielement design. The results showed that lower preference tasks were associated with higher rates of problem behaviors and that students, when given a choice, consistently selected the tasks that had been identified through interview and direct observation as higher preference. In Phase 3, we assessed whether allowing the students to choose between pairs of lower preference tasks or between pairs of higher preference tasks reduced problem behavior relative to a condition in which the teacher selected the same tasks. For 2 of 4 students, the rates of problem behavior were lower when students (rather than the teacher) selected the lower preference activity. Higher preference tasks for 3 students were associated with relatively low rates of problem behavior regardless of whether the student or the teacher selected the task.     

Cancer Risk Counseling: How Is It Different?

Fifty-six of 80 (70%) full members of the National Society of Genetic Counselors with interest in cancer genetics responded to a 1994 survey regarding their cancer risk counseling practices. This study was undertaken to describe cancer risk counselors and the services they provide and to identify possible differences from general genetic counseling that warrant further study. Of 56 respondents, 41 (75%) were providing CRC. The components of CRC programs are described. Our results found significant differences between CRC and general genetic counseling in terms of training and experience of genetic counselors providing CRC and length and number of counseling sessions per consultand. 51% of respondents had 1–2 years of working in CRC, compared to 17% with 1–2 years experience in genetic counseling (p<0.05). Over one-third had 10 or more years genetic counseling experience. Counselors were more likely to see individuals at risk for cancer for longer counseling sessions (p<0.05) and for additional sessions (p<0.05).     

The Familial Cancer Program of the Vermont Cancer Center: Development of a Cancer Genetics Program in a Rural Area

In response to many scientific discoveries linking cancer in certain families to inherited factors, the Vermont Cancer Center established the Familial Cancer Program (FCP) in December 1993. This multifaceted program combines the expertise of clinicians and researchers in many disciplines, including genetics, oncology, psychology, and molecular biology. The program's goals are identification of families in its region with excess cancer, provision of clinical services to such families, and use of research protocols when available and appropriate. This article describes the experience of setting up a familial cancer program in a rural area and discusses both successes and challenges in such an endeavor.     

The Student Guided Functional Assessment Interview: An Analysis of Student and Teacher Agreement

Functional assessment strategies are used to guide the treatment of problem behavior with individuals who have severe disabilities. Also, researchers have extended functional assessment applications to individuals with emotional and behavioral disabilities. One such extension is the use of students as informants in the functional assessment process. The present study assessed an interview tool designed to elicit information directly from students regarding their problem behaviors. We examined agreement between students and their teachers. Results showed high agreement on the causes and functions of problem behavior with mixed agreement on support plan recommendations.     

An evaluation of methylphenidate as a potential establishing operation for some common classroom reinforcers.

We conducted reinforcer assessments for 3 boys with a diagnosis of attention deficit hyperactivity disorder who alternately received either placebo or previously prescribed methylphenidate. Our purpose was to evaluate whether methylphenidate altered the relative reinforcing effectiveness of various stimuli that are often used in classroom-based behavioral treatment programs (e.g., activities, tangible items). Results showed clear differences for some stimuli between reinforcer assessments conducted when participants had received methylphenidate compared to placebo. Results suggest that methylphenidate might act as an establishing operation for some common classroom reinforcers. Implications for the development and evaluation of behavioral treatments are discussed.     

Comprehensive school-based behavioral assessment of the effects of methylphenidate.

Individualized assessments of the effects of three doses of methylphenidate (MPH) were conducted for 2 students with attention deficit hyperactivity disorder within each child's classroom using behavioral, academic, and social measures. A double-blind, placebo-controlled, multielement design was used to evaluate the results. Results suggested that at least one or more dosages of MPH were associated with some degree of improvement for both children in each area of functioning as compared to placebo. However, the degree of improvement at times varied substantially across dosage and area of functioning. Results suggest that MPH dosage and area of child functioning are critical assessment parameters and that controlled clinical trials are necessary to optimize the effectiveness of treatment with MPH for the individual child.     

The social phobia and social interaction anxiety scales: An exploration of the dimensions of social anxiety and sex differences in structure and relations with pathology

This study sought to provide information on the Social Phobia Scale (SPS) and Social Interaction Anxiety Scale (SIAS) of Mattick and Clarke (1989) with respect to factor structure, relations with psychopathology, and sex differences. A sample of 200 university students completed the SPS and SIAS and various measures of anxiety symptoms and depression. The results from the factor analyses for the sample as a whole suggest the presence of three factors corresponding to scrutiny fears, social interaction anxiety, and a general level of discomfort in social interactions. The results for men replicated this structure. For women, the three-factor solution demonstrated a blurring between the types of anxiety-provoking situations, and a general discomfort in situations involving differences in social power. In general, the discomfort factor was not correlated with measures of pathology, raising the possibility that uneasiness in these situations represents a process that is not part of social anxiety. The distinction between scrutiny fears and social interaction anxiety was also supported by the pattern of partial correlations that suggests that the presence of scrutiny fears is a stronger predictor of psychopathology than is social interaction anxiety, especially for men.