Statistical inference for false positive and false negative error rates in mastery testing

This paper describes an asymptotic inferential procedure for the estimates of the false positive and false negative error rates. Formulas and tables are described for the computations of the standard errors. A simulation study indicates that the asymptotic standard errors may be used even with samples of 25 cases as long as the Kuder-Richardson Formula 21 reliability is reasonably large. Otherwise, a large sample would be required.This work was performed pursuant to Grant No NIE-G-78-0087 with the National Institute of Education, Department of Health, Education and Welfare, Huynh Huynh, Principal Investigator. Points of view or opinions stated do not necessarily reflect NIE position or policy and no official endorsement should be inferred. The editorial assistance of Joseph C. Saunders is gratefully acknowledged.     

4~8年级学生分数数量表征的准确性及形式

在分数学习中,分数数量表征是非常重要的方面。本研究利用数字线估计任务对四到八年级学生分数数量表征的情况进行探索。研究结果表明：随着年级升高,被试分数数量表征的准确性也随之提高;被试对单位分数和非单位分数表征的准确性存在显著差异,表现为对单位分数表征的准确性显著高于对非单位分数的表征,这种差异在低年级显著,随着年级的增长,差异逐渐消失;四到八年级学生在0~3数字线上,对分数数量的表征表现为线性形式而非对数形式,且这种线性表征形式是在六年级开始出现并随着年龄增长逐渐发展起来的。中西方儿童在分数数量表征的准确性和形式上有相似的发展路径,但是在表征准确性上中国儿童更高、线性形式出现年级上中国儿童可能更早。     

Observational drawing biases are predicted by biases in perception: Empirical support of the misperception hypothesis of drawing accuracy with respect to two angle illusions

We tested the misperception hypothesis of drawing errors, which states that drawing accuracy is strongly influenced by the perceptual encoding of a to-be-drawn stimulus. We used a highly controlled experimental paradigm in which nonartist participants made perceptual judgements and drawings of angles under identical stimulus exposure conditions. Experiment 1 examined the isosceles/scalene triangle angle illusion; congruent patterns of bias in the perception and drawing tasks were found for 40 and 60° angles, but not for 20 or 80° angles, providing mixed support for the misperception hypothesis. Experiment 2 examined shape constancy effects with respect to reproductions of single acute or obtuse angles; congruent patterns of bias in the perception and drawing tasks were found across a range of angles from 29 to 151°, providing strong support for the misperception hypothesis. In both experiments, perceptual and drawing biases were positively correlated. These results are largely consistent with the misperception hypothesis, suggesting that inaccurate perceptual encoding of angles is an important reason that nonartists err in drawing angles from observation.     

Combining speed and accuracy to assess error-free cognitive processes

Both the speed and accuracy of responding are important measures of performance. A well-known interpretive difficulty is that participants may differ in their strategy, trading speed for accuracy, with no change in underlying competence. Another difficulty arises when participants respond slowly and inaccurately (rather than quickly but inaccurately), e.g., due to a lapse of attention. We introduce an approach that combines response time and accuracy information and addresses both situations. The modeling framework assumes two latent competing processes. The first, the error-free process, always produces correct responses. The second, the guessing process, results in all observed errors and some of the correct responses (but does so via non-specific processes, e.g., guessing in compliance with instructions to respond on each trial). Inferential summaries of the speed of the error-free process provide a principled assessment of cognitive performance reducing the influences of both fast and slow guesses. Likelihood analysis is discussed for the basic model and extensions. The approach is applied to a data set on response times in a working memory test. The authors wish to thank Roger Ratcliff, Christopher Chabris, and three anonymous referees for their helpful comments, and Aureliu Lavric for providing the data analyzed in this paper.     

Application of Sequential Interval Estimation to Adaptive Mastery Testing

In this paper, we apply sequential one-sided confidence interval estimation procedures with β-protection to adaptive mastery testing. The procedures of fixed-width and fixed proportional accuracy confidence interval estimation can be viewed as extensions of one-sided confidence interval procedures. It can be shown that the adaptive mastery testing procedure based on a one-sided confidence interval with β-protection is more efficient in terms of test length than a testing procedure based on a two-sided/fixed-width confidence interval. Some simulation studies applying the one-sided confidence interval procedure and its extensions mentioned above to adaptive mastery testing are conducted. For the purpose of comparison, we also have a numerical study of adaptive mastery testing based on Wald's sequential probability ratio test. The comparison of their performances is based on the correct classification probability, averages of test length, as well as the width of the “indifference regions.” From these empirical results, we found that applying the one-sided confidence interval procedure to adaptive mastery testing is very promising.     

Base rates versus sample accuracy: competition for control in human matching to sample

People often place undue weight on specific sources of information (case cues) and insufficient weight on more global sources (base rates) even when the latter are highly predictive, a phenomenon termed base-rate neglect. This phenomenon was first demonstrated with paper-and-pencil tasks, and also occurs in a matching-to-sample procedure in which subjects directly experience case sample (cue) accuracy and base rates, and in which discrete, nonverbal choices are made. In two nonverbal experiments, subjects were exposed to hundreds of trials in which they chose between two response options that were both probabilistically reinforced. In Experiment 1, following one of two possible samples (the unpredictive sample), either response was reinforced with a .5 probability. The other sample (predictive) provided reinforcement for matching on 80% of the trials in one condition but in only 20% of the trials in another condition. Subjects' choices following the unpredictive sample were determined primarily by the contingencies in effect for the predictive sample: If matching was reinforced following the predictive sample, subjects tended to match the unpredictive sample as well; if countermatching the predictive sample was generally reinforced, subjects tended to countermatch the unpredictive sample. These results demonstrate only weak control by base rates. In Experiment 2, base rates and sample accuracy were simultaneously varied in opposite directions to keep one set of conditional probabilities constant. Subjects' choices were determined primarily by the overall accuracy of the sample, again demonstrating only weak control by base rates. The same pattern of choice occurred whether this pattern increased or decreased rate of reinforcement. Together, the results of the two experiments provide a clear empirical demonstration of base-rate neglect.     

Assessment Centre Observation Procedures: An Experimental Comparison of Traditional,Checklist and Coding Methods

Concern about the construct validity of assessment centre judgements has led to calls for research into observers’ cognitive processes. In an experiment comparing the Traditional observation procedure against use of a Behavioural Checklist and of Behavioural Coding, six outcome variables were examined. The methods were found to yield similar outcomes in terms of accuracy of judgement, accuracy of written evidence, correlation between dimension ratings, and attitude toward the method employed. However, significant between-method differences were observed in variability between observers and in their willingness to recommend a procedure. It is argued that Behavioural Coding has special merit in assessment centres; the method reduces the cognitive demands placed on assessors, and valuably structures their information-processing. Traditional objections to simultaneous observation and classification are shown to be inappropriate.     

Matching-to-sample accuracy on fixed-ratio schedules.

Pigeons performing on a matching-to-sample procedure were exposed to six fixed-ratio (FR) schedules (FR 1, 5, 10, 20, 40, and 60) of food reinforcement for correct matching responses. During both a correction and a noncorrection procedure without an intertrial interval (ITI), matching accuracy was lower on FR 1 and FR 60 than at intermediate ratios. With the FR 1 schedule, both a 5-sec and a 25-sec ITI resulted in higher matching accuracy than without an ITI; accuracy, with an ITI, was fairly constant for ratios of 1 to 20 but declined at higher ratios. The results suggest that the presence or absence of an ITI in matching to sample may account for inconsistencies obtained in earlier studies of the relationship of matching accuracy to ratio size.     

Methodological issues in judgment and decision-making research: Concurrent verbal protocol validity and simultaneous traces of process

This paper examines two dimensions of concurrent verbal protocol validity. First, whether verbalization affects process and outcome (reactivity) was examined by comparing concurrent verbal protocol traces with those from a computer search process tracing method, the latter being a complete trace of information acquisition from experimental materials. Earlier findings that verbalization affects time were confirmed. However, verbalization did not affect amount and pattern of acquisition or accuracy of judgments. Second, whether concurrent verbal protocols are complete was examined by comparing concurrent verbal protocol and computer traces that were simultaneously obtained in a treatment in which subjects verbalized as they acquired information from the computer. The verbal traces less completely captured information acquisition behavior than computer search. This suggests that, although concurrent verbal protocols provide greater insight into decision behavior than computer search, the latter is a more reliable information-acquisition trace. Thus, if information acquisition is of primary interest and if computer search activities can be naturally integrated into performing the primary task, computer search is preferred to concurrent verbal protocols. However, if information use or retrieval from long-term memory is of primary interest, concurrent verbal protocols are preferred to computer search. Finally, this paper examined whether the simultaneous use of concurrent verbal protocols and computer search provides traces of information acquisition and use that are as complete as when each method is independently applied. Although computer search tended to limit subjects verbalizations of evaluative operators, this effect may be eliminated by practice on the computer prior to collecting data.