Multiple Evaluation: A New Testing Paradigm That Exorcizes Guessing

The crux in psychometrics is how to estimate the probability that a respondent answers an item correctly on one occasion out of many. Under the current testing paradigm this probability is estimated using all kinds of statistical techniques and mathematical modeling. Multiple evaluation is a new testing paradigm using the person's own personal estimates of these probabilities as data. It is compared to multiple choice, which appears to be a degenerated form of multiple evaluation. Multiple evaluation has much less measurement error than multiple choice, and this measurement error is not in favor of the examinee. When the test is used for selection purposes as it is with multiple choice, the probability of a Type II error (unjustified passes) is almost negligible. Procedures for statistical item-and-test analyses under the multiple evaluation paradigm are presented. These procedures provide more accurate information in comparison to what is possible under the multiple choice paradigm. A computer program that implements multiple evaluation is also discussed.     

Policy evaluation: Design and utilization

In the 1960s when evaluation research came into fashion in the US, the assumption was that it would be used by public policymakers in an instrumental way for purposes of problem solving. This entailed a linear, knowledge-driven model for research utilization in which a direct relationship exists between knowing and doing (i.e., knowledge leads to action). The model combined the “rational” paradigm of organizational theory and the “scientific” paradigm of positivist philosophy of science into a consistent model of how evaluation research should be conducted and would be utilized. Both with respect to its normative and its empirical dimensions, original evaluation research model failed. Today evaluation research and its utilization are seen in quite a different perspective. However, though a parallel paradigm revolution or shift has taken place both with respect to the organization theory of evaluation research and its methodological basis, no new comprehensive and coherent evaluation research model has emerged to replace the original linear one. Erik Albaek is an associate professor of public administration at the University of Aarhus, Denmark. One of his main research areas is the utilization of social science in public policymaking.     

INVESTIGATION OF BIAS IN JOB EVALUATION RATINGS OF COMPARABLE WORTH STUDY PARTICIPANTS

An experimental study was conducted to investigate the effects of knowledge of current pay levels and perceived job gender on subsequent job evaluations. The sample consisted of 53 job evaluators in professional and scientific positions at the University of Iowa who had previously received 20 hours of training in job evaluation and participated in over 100 hours of job evaluations during the implementation of a comparable worth pay system. The hypothesis that jobs with high (manipulated) pay levels would receive higher evaluations than jobs with low (manipulated) pay levels was confirmed, although the effects were smaller than those reported in other studies. In addition, evidence of a pro-female bias was found ( p < .08) in the job evaluation ratings. Implications of these findings for job evaluation research and practice are discussed.     

人工语法范式下内隐学习的脑机制及其教育启示

人工语法范式是研究内隐学习实验的重要范式。利用脑成像技术的研究发现：人工语法范式下内隐学习发生时,额叶、枕叶、顶叶以及基底神经节与内隐学习有密切关系。该结果对当前教育教学产生一定启示,即教师要多给学生创造知识迁移的机会;培养学生的动手实践活动;重视学习过程中内隐学习与外显学习的先后顺序和有序结合。     

The organization of information retrieved from situation models

The organization of information when it is retrieved from situation models was explored using recognition and recall tests. A fan effect paradigm was used to assess organization in recognition and a clustering measure was used in recall. People memorized either object-location facts (e.g., “The pay phone is in the city hall”) or person/small location facts (e.g., “The banker is on the witness stand”). For object-location facts, a location-based organization was observed in both recall and recognition. However, for person/small location facts, a location-based organization was observed for recall, and a person-based organization was observed for recognition. Thus, the observed organization was flexible depending on the retrieval task.     

Personal deserving versus justice for others: An exploration of the justice motive

Two expriments were conducted to test implications derived from Lerner's formulation of the “justice motive.” The experimental paradigm provided subjects with an opportunity to earn money in exchange for participation in future experimental sessions. Subjects who were offered “just” pay for their efforts volunteered significantly more of their time when their efforts would help a needy victim than when they would not. Moreover, when subjects were left with just pay, they offered more of their time when a portion of their pay was to go to a victim than when they received the entire wage themselves. When the pay subjects were to receive was less than deserved, however, the opportunity to help a victim did not affect their response level. The importance of these results for understanding helping behavior is considered, as is the compatibility of these results with other theoretical models.     

Matching bias in syllogistic reasoning: Evidence for a dual-process account from response times and confidence ratings

We examined matching bias in syllogistic reasoning by analysing response times, confidence ratings, and individual differences. Roberts’ (2005) “negations paradigm” was used to generate conflict between the surface features of problems and the logical status of conclusions. The experiment replicated matching bias effects in conclusion evaluation (Stupple & Waterhouse, 2009), revealing increased processing times for matching/logic “conflict problems”. Results paralleled chronometric evidence from the belief bias paradigm indicating that logic/belief conflict problems take longer to process than non-conflict problems (Stupple, Ball, Evans, & Kamal-Smith, 2011). Individuals’ response times for conflict problems also showed patterns of association with the degree of overall normative responding. Acceptance rates, response times, metacognitive confidence judgements, and individual differences all converged in supporting dual-process theory. This is noteworthy because dual-process predictions about heuristic/analytic conflict in syllogistic reasoning generalised from the belief bias paradigm to a situation where matching features of conclusions, rather than beliefs, were set in opposition to logic.     

自我损耗的研究方法述评

自我控制是个体成功的重要因素,但个体在自我控制的同时,也要为每次控制付出代价,产生自我损耗,导致随后自我控制的失败。本文回顾了双任务范式中诱发、测量、克服自我损耗的方法,认为未来研究应探索任务的有效性、多采用三任务范式等,在克服自我损耗方面,应同时关注短期克服损耗的方法和提高整体控制能力的方法。     

Teachers do not Deal in Widgets

Although the use of merit pay, incentive pay, bonus pay, and job promotion are well-established measures in the private or corporate sector of American society, few examples of successful teacher merit pay systems exist. In 1983, the Eastern Washington University Department of Education surveyed six major private corporations to determine whether corporate merit pay programs could be applied to teachers and educational institutions. Survey findings suggest several reasons why merit pay programs for teachers will probably fail: (a) lack of monetary goals; (b) the necessity for subjective evaluation, which requires multiple measurement devices, extensive supervision, and time; (c) the absence of a many-tiered corporate structure in most educational institutions; and (d) increased supervisory direction that could stifle creativity and flexibility.     

Influence of contextual information and past prices on the willingness to pay and expected quality evaluations

This study conducted two experiments to assess consumers' willingness to pay or the expected quality in response to different levels of contradictory contextual information (regarding price and quality) and the presence of anchors in the form of past price paid. First, the results show that the contextual information was decisive in the formation of reference prices that affect the willingness to pay. Second, the anchors did not influence the evaluation of the willingness to pay and the expected quality. And third, it was observed that the coherence of the contextual information (price–quality correlation) performs a moderating role in the relation between reference prices and expected quality. These results suggest that anchors are less important than contextual information in the evaluation process of the willingness to pay and the expected quality, and that therefore neither an adjustment nor a priming mechanism is activated. We found that the coherence of the contextual information influences how individuals evaluate a service. Copyright © 2016 John Wiley & Sons, Ltd.     

情绪劳动：表层动作与深层动作，哪一种效果更好？

情绪劳动（表层动作和深层动作）,哪一种情绪劳动的效果更积极？该项研究结合“情绪调节”和“双任务”的实验范式,以指导语引起不同的情绪劳动,以电影片段诱发负情绪（以悲伤为主）,同时测量情绪劳动过程情绪的变化,并通过数学成绩和自我真实感检测情绪劳动对后续工作和心理上的作用。结果表明：（1）操作表层动作与深层动作都具有一定的难度,需要付出一定的努力;且深层动作引起更多的精力集中;（2）表层动作和深层动作都可以减弱已被诱发的悲伤情绪,深层动作引起了更大的生理激活（R-R间期增加较小）;（3）表层动作导致后续工作成绩降低,自我真实感减弱。结论：如果需要在负情绪（悲伤）状态下展现正情绪（微笑）,深层动作是更积极的情绪劳动方式     

Le renversement de préférence… renversé !

Preference reversal (PR) occurs when pairs of lottery gambles are evaluated under two different conditions: to choose the gamble preferred to play and assign minimum selling price to each gamble. The preference order is then reversed from one condition to another: while it receives the lowest selling price, the gamble less risked is preferred in condition of play. Many times replicated, this result is considered as a cognitive bias. Recent research suggests that it could be related only to the procedure used to present the gambles (i.e., the evaluation mode): either by pair (joint evaluation or JE) in condition of play, or one by one (separate evaluation or SE) in the pricing task. Then, by reversing the evaluation mode in each condition, PR should be reversed, as it is shown in this study: although the phenomenon still occurs when subjects evaluate gambles in the classical paradigm, we observe an inversion of PR when they have, on the one hand, to play in SE, and on the other hand, to sell in JE. Furthermore, an analysis not often used in this literature provides new elements, which can attest to the cognitive mechanisms underlying the phenomenon: by recording the online information (self-paced display time paradigm) we show the effect of an anchoring and adjustment process when riskiest gambles are preferred.     

Procedural and interpersonal fairness moderate the relationship between outcome fairness and acceptance of merit pay

An organizational field study (N = 257) investigated employees' acceptance of a new merit pay system as involving an assessment of whether merit pay can make their earnings more fair, compared to their earnings in the current, seniority-based pay system. We expected that improvement of unfair earnings, and consequently acceptance of merit pay, is considered likely when existing procedures that produce these earnings are unfair, because merit pay improves such procedures. We also expected improvement of unfair earnings, and increased merit pay acceptance, to be likely when employees anticipate fair performance evaluation in a new system, as indicated by fair interpersonal treatment by their supervisor. Results showed that procedural and interpersonal fairness in the existing pay system indeed moderated the relationship between fairness of current outcomes and merit pay acceptance as predicted. Implications for the introduction of merit pay in organizations and for our understanding of the different roles of procedural and interpersonal fairness in outcome evaluations are discussed.     

The use of Tholos software for combining measures of mental workload: Toward theoretical and methodological improvements

Mental workload is a concept central to a range of disciplines (including cognitive psychology and ergonomics) that has given rise to various theoretical and methodological debates. As a result, researchers have used a number of techniques for measuring mental workload. Traditionally, three categories of measurement technique have been recognized: performance measures (e.g., the dual-task paradigm), subjective measures (e.g., rating scales), and physiological measures (e.g., heart, respiration, and blink rates). Each technique has advantages and limitations; however, some limitations may prevent an accurate evaluation of the mental workload. In this article, we focus on the benefits of combining multiple measures of mental workload. However, because combining several techniques is a very complex process, we have developed the Tholos software in an attempt to reduce this complexity. This software package uses measures from each of the three categories: a dual-task paradigm with auditory signals; the National Aeronautics and Space Administration task load index (NASA-TLX) scale and its simplified version, the “raw” task load index (RTLX); and physiological (such as pupil-dilation) measurements with which our software can merge results from the dual-task paradigm. To illustrate the benefits of using Tholos, we describe a puzzle-solving experiment in which we combined multiple measures of mental workload. The results indicate the importance of combining multiple measures to build upon the theoretical and methodological foundations of mental workload.     

面向生活世界的伦理人类学

一、在规范伦理学与德性伦理学论争的背后规范伦理学与德性伦理学的论争是当代伦理学研究中的重要学术现象。规范伦理学以罗尔斯和哈贝马斯为代表,德性伦理学以麦金太尔为代表,形成了观点相互对立的两大伦理学阵营。规范伦理学以哲学思辨的方法研究伦理问题,其研究领域主要是伦理的价值或道德的应然,期望     

橡胶手错觉:拥有感研究的实验范式及其应用

橡胶手错觉是一种健康个体将非肉体的假手视为自己真实身体的一部分的体验, 这种错觉可以通过同时轻刷被试面前可见的橡胶手及其不可见的真手而产生.橡胶手错觉已成为一种研究身体拥有感的重要范式, 其产生机制可以分为“自下而上的认知匹配”与“自上而下的认知匹配”两种加工方式.前者涉及视觉与触觉刺激的同步性; 而后者涉及被试心中预存的身体意象与身体图式(包括真假手之间模态特征,位置空间的相似性).综合上述证据, 身体模型假说与个人边缘空间理论进一步为拥有感产生的复杂机制提供了整合两种加工方式的解释.橡胶手错觉范式已经被用于探索精神分裂症患者等特殊被试病理分析,错觉产生和心理特质之间的关系, 以及神经外科和术后恢复上.未来的研究应该更加重视范式本身的拓展, 应用虚拟现实技术来提高身体模态的仿真效果, 利用橡胶手拥有感的易感性作为筛选与预测身体意象障碍疾病的指标.     

Evaluation of infant vocal conditioning.

The use of the operant conditioning paradigm, as it has been applied to infant social, vocal behavior, reflects a failure to take into account the social nature of human infants over and above the rigid theoretical rationale of the paradigm. It is argued that: (1) the baseline procedure used in operant conditioning studies is methodologically and conceptually invalid; (2) the reinforcing stimulus used in social conditioning studies elicits the very response that it is assigned to reinforce; and (3) the effect of the response-reinforcer relationship is not the reinforcement (strengthening) of vocal response rate.     

Transitivity for height versus speed: To what extent do the under-7s really have a transitive capacity?

Transitive inference underpins many human reasoning competencies. The dominant task (the “extensive training paradigm”) employs many items and large amounts of training, instilling an ordered series in the reasoner's mind. But findings from an alternative “three-term paradigm” suggest transitivity is not present until 7 + years. Interestingly, a second alternative paradigm (the “spatial task”), using simultaneously displayed height relationships to form premise pairs, can uphold the 4-year estimate. However, this paradigm risks cueing children and hence is problematic. We investigated whether a height-task variant might correspond to a more ecologically valid three-term task. A total of 222 4–6-year-olds either completed a modified height task, including an increased familiarisation phase, or a computer-animated task about cartoon characters running a race in pairs. Findings confirmed that both tasks were functionally identical. Crucially, 4-year-olds were at chance on both, whereas 6-year-olds performed competently. These findings contrast with estimates from all three paradigms considered. A theoretical evaluation of our tasks and procedures against previous ones, leads us to two conclusions. First, our estimate slightly amends the 7-year estimate offered by the three-term paradigm, with the difference explained in terms of its greater relevance to child experiences. Second, our estimate can coexist alongside the 4-year estimate from the extensive training paradigm. This is because, applying a recently developed “dual-process” conception of reasoning, anticipates that extensive training benefits a species-general associative system, while the spatial paradigm and three-term paradigm can potentially index a genuinely deductive system, which has always been the target of transitive research.     

A Reverse Outcome Bias: The Influence of Multiple Reference Points on the Evaluation of Outcomes and Decisions

Two experiments examined the influence of multiple reference points on the evaluation of outcomes and decisions that lead to those outcomes. Norm theory is applied as a conceptual framework for understanding the conditions under which alternative norms may be evoked by the decision context and how they are subsequently used as reference points in the evaluation process. Of primary interest, in these studies, was how an outcome is evaluated when two reference points, the status quo and an evoked alternative, provide conflicting information about the "goodness" of the outcome (the outcome is good from the perspective of one reference point and bad from the perspective of the other). A gambling paradigm, based on regret theory, is employed to address these questions. We find that an alternative outcome is more likely to be evoked as a reference point when: (1) it is certain that another choice would have led to the alternative outcome; (2) a social comparison other receives the alternative outcome; and (3) the alternative outcome is in a different evaluative domain than the outcome received (i.e., is negative when the outcome received is positive). When these conditions hold, and the alternative outcome is used as a reference point for evaluation, the evaluations which result are quite counterintuitive: winners are rated as more regretful over their choices than losers are considered to be less satisfied with their outcomes than losers, and are rated as having made poorer quality decision than losers, who made the same choice! The latter finding represents a complete reversal of the outcome bias often observed in judgments of decisions made under uncertainty.     

Individual behavior in learning of an artificial grammar

Artificial grammar learning (AGL) is a widely used experimental paradigm that investigates how syntactic structures are processed. After a familiarization phase, participants have to distinguish strings consistent with a set of grammatical rules from strings that violate these rules. Many experiments report performance solely at a group level and as the total number of correct judgments. This report describes a systematic approach for investigating individual performance and a range of different behaviors. Participants were exposed to strings of the nonfinite grammar A ⁿ B ⁿ . To distinguish grammatical from ungrammatical strings, participants had to pay attention to local dependencies while comparing the number of stimuli from each class. Individual participants showed substantially different behavioral patterns despite exposure to the same stimuli. The results were replicated across auditory and visual sensory modalities. It is suggested that an analysis that looks at individual differences grants new insights into the processes involved in AGL. It also provides a solid basis from which to investigate sequence-processing abilities in special populations, such as patients with neurological lesions.