Design Rationale for a Complex Performance Assessment

In computer-based interactive environments meant to support learning, students must bring a wide range of relevant knowledge, skills, and abilities to bear jointly as they solve meaningful problems in a learning domain. To function effectively as an assessment, a computer system must additionally be able to evoke and interpret observable evidence about targeted knowledge in a manner that is principled, defensible, and suited to the purpose at hand (e.g., licensure, achievement testing, coached practice). This article describes the foundations for the design of an interactive computer-based assessment of design, implementation, and troubleshooting in the domain of computer networking. The application is a prototype for assessing these skills as part of an instructional program, as interim practice tests and as chapter or end-of-course assessments. An Evidence Centered Design (ECD) framework was used to guide the work. An important part of this work is a cognitive task analysis designed (a) to tap the knowledge computer network specialists and students use when they design and troubleshoot networks and (b) to elicit behaviors that manifest this knowledge. After summarizing its results, we discuss implications of this analysis, as well as information gathered through other methods of domain analysis, for designing psychometric models, automated scoring algorithms, and task frameworks and for the capabilities required for the delivery of this example of a complex computer-based interactive assessment.     

Computers and commitment to a public management decision: An experiment

Based on results of an experiment, hypotheses are tested concerning the effects of computer use on decision commitment. The experiment required subjects to make an adoption decision regarding a hypothetical government agency's innovation. Subjects could choose from a variety of information sets, some computer based, some not, before making the decision. After their decision the subjects were given “new evidence” that contradicted their initial position. Two experimental treatments included more difficult access to the computer-based information and higher cost for the computer-based information. Results indicate that access difficulty diminishes confidence in decisions and leads to lesser commitment. However, the cost of the computer information seems to have little bearing on decision commitment. Barry Bozeman is director of the Technology and Information Policy Program of the Maxwell School of Public Affairs, Syracuse University, Syracuse NY 13244-4010. There he is professor of public policy and administration and affiliate professor of engineering. His research has focused on organization theory and public management, science and technology policy, and the use of technical information in decision-making. R.F. Shangraw, Jr. is vice president of Independent Project Analysis, Inc., Great Falls, Virginia. Previously, he was assistant professor of public administration at Syracuse University. His research interests include information management, decision-making, and public policy applications of expert systems.     

An exploratory analysis of the latent structure of process data via action sequence autoencoders

Computer simulations have become a popular tool for assessing complex skills such as problem-solving. Log files of computer-based items record the human–computer interactive processes for each respondent in full. The response processes are very diverse, noisy, and of non-standard formats. Few generic methods have been developed to exploit the information contained in process data. In this paper we propose a method to extract latent variables from process data. The method utilizes a sequence-to-sequence autoencoder to compress response processes into standard numerical vectors. It does not require prior knowledge of the specific items and human–computer interaction patterns. The proposed method is applied to both simulated and real process data to demonstrate that the resulting latent variables extract useful information from the response processes.     

A Formal Model for the Study of Communication Support Systems

The lack of conceptual frameworks available for integrating research on organizational computer-based communication technologies is documented. An integrated model for studying communication support phenomena in organizations is formalized from a knowledge management perspective using set-theoretic notation. Key constructs for understanding and exploring communication support systems are identified, such as knowledge workers, knowledge management activities, communicate-abilities, and knowledge management episodes. The implications associated with the identification and formalization of each of the foregoing constructs is discussed and further research avenues are explored. The model provides researchers and system developers with a means of studying both human and computer-based knowledge workers in organizations. Two types of communication support systems are defined.     

Offside decisions by expert assistant referees in association football: Perception and recall of spatial positions in complex dynamic events

This study investigated the offside decision-making process in association football. The first aim was to capture the specific offside decision-making skills in complex dynamic events. Second, we analyzed the type of errors to investigate the factors leading to incorrect decisions. Fédération Internationale de Football Association (FIFA; n = 29) and Belgian elite (n = 28) assistant referees (ARs) assessed 64 computer-based offside situations. First, an expertise effect was found. The FIFA ARs assessed the trials more accurately than the Belgian ARs (76.4% vs. 67.5%). Second, regarding the type of error, all ARs clearly tended to raise their flag in doubtful situations. This observation could be explained by a perceptual bias associated with the flash-lag effect. Specifically, attackers were perceived ahead of their actual positions, and this tendency was stronger for the Belgian than for the FIFA ARs (11.0 vs. 8.4 pixels), in particular when the difficulty of the trials increased. Further experimentation is needed to examine whether video- and computer-based decision-making training is effective in improving the decision-making skills of ARs during the game.     

Effectiveness of a responsible conduct of research course: a preliminary study

Training in the responsible conduct of research (RCR) is required for many research trainees nationwide, but little is known about its effectiveness. For a preliminary assessment of the effectiveness of a short-term course in RCR, medical students participating in an NIH-funded summer research program at the University of California, San Diego (UCSD) were surveyed using an instrument developed through focus group discussions. In the summer of 2003, surveys were administered before and after a short-term RCR course, as well as to alumni of the courses given in the summers of 2002 and 2001. Survey responses were analyzed in the areas of knowledge, ethical decision-making skills, attitudes about responsible conduct of research, and frequency of discussions about RCR outside of class. The only statistically significant improvement associated with the course was an increase in knowledge, while there was a non-significant tendency toward improvements in ethical decision-making skills and attitudes about the importance of RCR training. The nominal impact of a short-term training course should not be surprising, but it does raise the possibility that other options for delivering information only, such as an Internet-based tutorial, might be considered as comparable alternatives when longer courses are not possible.     

Visual skills and playing positions of Olympic field hockey players

Many sports require fine spatiotemporal resolution for optimal performance. Previous studies have compared anticipatory skills and the decision-making process in athletes; however, there is little information on visual skills of elite athletes, particularly hockey players. To assess visual skills of Olympic hockey players and analyze differences by playing position, and to analyze improvement of visual skills after training, 21 Olympic field hockey players were pre- and post-tested on 11 visual tasks following a 10-wk. visual training program consisting of computer-based visual exercises. There were no mean differences at pre-test between players of different positions, suggesting that performance on these visual skills was independent of playing position. However, after training, an improvement was seen in all players (when scores were averaged across all 11 visual tasks) with goalkeepers improving significantly more than any other position. This suggests the possibility of improving visual skills even in an elite population.     

Information technology in municipal environmental policy: Automated registration,sure, but what about expert systems?

Dutch municipalities are confronted with an increased number of prescribed environmental tasks and also with a growing demand, both from the central government and environmental pressure groups, to undertake environmental activities on their own initiative. This development over-taxed the information management of most municipalities. In the past few years, computer technology was introduced to relieve part of this pressure (e.g., by automation of registration systems). In this article we present a classification of computer applications for environmental management, investigate their possible impact on the environmental knowledge and information system, and distinguish between a formal and informal knowledge domain. Special attention will be paid to expert systems.     

信息显示板（IDB）实验在消费行为研究中的应用

信息显示板（Information Display Board, IDB）实验是一种过程跟踪技术,以m×n矩阵方式呈现信息阵列,通过观测被试信息检索和浏览的一系列行为,来研究被试问题解决和决策中的信息获得（information acquisition）过程。Mouselab是常用的IDB计算机化程序。消费者是有限理性的问题解决者,因此IDB实验被用于消费行为领域的研究。应用不同的IDB实验设置,研究者探索了消费决策中消费者信息获得的模式和影响因素。     

Learning with a computer-based advisor

The subject of this paper is the design of a computer-based educational advisory system called LOLA (Leeds On-Line Advisor). Implemented in SMALLTALK-80, LOLA attempts to address some of the problems which arise when students are required to manage their own learning, a decision-making process of importance particularly in tertiary education. LOLA demonstrates how Artificial Intelligence (AI) techniques can be incorporated in the design of instructional systems. LOLA provides advice to students about what to study, how and when to study topics. The work was supervised by J.R. Hartley and benefited from many helpful discussions with K. Tait and Professor N. Sheehy.     

Knowledge management in decision making and decision support

A knowledge-oriented view of decisions and decision making is introduced, as a complement to classical perspectives and as a contribution of under-standing computer-based possibilities for relaxing strains on decision makers. This perspective includes a model of knowledge management activities performed by a decision maker and a taxonomy of knowledge types. It leads to a characterization of decision-support-system purpose, traits, and potentials that offers a basis for new research into computer-based possibilities for knowledge management. Clyde W. Holsapple holds the Rosenthal Endowed Chair in Management Information Systems and is Professor of Decision Science and Information Systems at the College of Business and Economics, University of Kentucky.     

Enhancing Skills for Everyday Problem Solving,Decision Making,and Conflict Resolution in Special Needs Students with the Support of Computer-Based Technology

Too many special needs students leave school without being able to compete effectively for jobs which require problem solving, skilled literacy, knowledge of technology, and at least adequate interpersonal skills. In this article, key problem solving and decision making skills in which special needs students are deficientare outlined and evidence is presented for the importance of teaching these skills. A specific procedure called the Personal Problem Solving Guide, utilizing readily available personal computer technology to teach children social decision making and problem solving, is described. The Guide provides formats that allow for constructive behavior in discipline situations, as well as anticipatory guidance for managing conflicts, thinking about health choices, and reviewing everyday interpersonal and academic situations before taking impulsive action. A case illustration and related information are presented to demonstrate the feasibility and benefits of using the computer as a tool for oromoting effective learning of social decision making skills and for bromotinlg the behavioral application of these skills among special needs populations.     

Life Skills: Adolescents' Perceptions of Importance and Competence

A previous classification of life skills was used to suggest issues relating to areas of adolescents' concerns involving relationships, leisure, jobs, education, preparation for the future, work, budgeting, health, social responsibility, and personal skills such as planning and decision-making. A questionnaire relating these concerns to young people's perceptions of the facets of importance, and their own knowledge, competence, sense of improvement, problems, and striving, was administered to a sample of 1,084 adolescents in senior schools and colleges of technical and further education. The results suggested a strong general factor for each facet, with some differentiation into groups of skills. The mean ratings suggested that the skills investigated were considered to be very important by young people, who in general also seemed optimistic about their competence. However, a small proportion expressed serious problems. Responses to the various facets were further examined in terms of group differences — by age, gender, educational context, ethnicity, economic status, and educational expectations. Many small but significant group differences were isolated. Finally, implications are drawn from this related research for schools and colleges in general and for guidance and counselling in particular.     

Exploring Adoption with Clients: The Need for Adoption Education Within the Genetic Counseling Profession

Genetic counselors and other health professionals may encounter adoption during any counseling session. They must be skilled in using appropriate language and understand how to approach and discuss this topic with clients. A thorough knowledge of adoption as an option for clients facing a prenatal or postnatal diagnosis is necessary when presenting individuals with non-biased information needed for informed decision-making. However, three preliminary studies have demonstrated an absence of graduate education and lack of a professional knowledge base regarding this option (Mates 2008; Oksala 2007; Perry 2003). We discuss the impact of medical professionals’ preconceptions on client decision-making, increasing early identification of fetal anomalies, deficiency of adoption knowledge and resources, and the resulting need for genetic counselors and other health professionals to develop their skills in discussing adoption with clients.     

The relationship between deliberate practice,play, and futsal in childhood and adolescence and the development of different decision-making skills in professional female soccer players

Soccer players’ ability to make efficient and quick decisions has gained more importance due to the increase in game speed in the last few years. Therefore, this study aimed to examine the relationship between the engagement in previous developmental activities in soccer and futsal with the quality and speed of decision-making skills in different phases of sport development of elite female soccer players. The sample comprised 77 elite Brazilian professional female soccer players. Players’ decision-making skills were assessed based on an objective video-based test – TacticUP®. We used a retrospective questionnaire to collect information about previous participation in different developmental activities. The results showed that engagement in deliberate practice in soccer and futsal, especially during childhood and early adolescence, is related to a better quality of offensive decision-making skills, although showing small to medium effect sizes. We highlight that deliberate practice in futsal is associated only with offensive decision-making skills with the ball and near the ball. In turn, engaging in deliberate play in soccer, mainly in childhood and early adolescence, is related to quicker offensive and defensive decision-making skills, showing mostly medium effect sizes. To the best of our knowledge, it was the first study to measure the relationship of decision-making speed with developmental activities in soccer. It is concluded that deliberate practice in soccer and deliberate play in soccer are associated with different dimensions of decision-making skills.     

核心知识系统及其对相关研究的启示

核心知识系统出现于人类个体发展和种系发展的早期,在人类复杂认知能力的发生发展中起着建构模块的作用。该文以表征物体和数量关系的两类核心知识系统的性质及二者在儿童的“数”概念形成和成人的数学思维中所起的作用为例,综述了以灵长类动物、婴儿、儿童和成人为被试的研究证据,并进而提出了核心知识系统给相关研究带来的若干启示。     

Legal aspects of decision-enhancing technologies

As advances in computer-based systems (CBS) have continued, development problems can occur which can adversely affect systems developers and users of these technologies, especially those designed to improve or affect decision making. One major difficulty concerns the legal liability that can occur as a result of defective or flawed systems development efforts. This article discusses several types of decision-enhancing technologies and how problems associated with the adequacy of testing of system design and appropriateness of warnings and directions/instructions dealing with CBS could adversely affect CBS developers from a legal perspective. Several propositions and suggested research perspectives are also presented. He received his Ph.D. in computer information systems from Arizona State University. His research interests include the impact on organizations and individuals of information systems and the potential legal liability to systems developers brought about by defective computer-based systems. His research has appeared inMIS Quarterly, Information & Management, AI Expert, AI & Society, Educational andPsychological Measurement andInformation Resources Management Journal. Kathleen Mykytyn is a consultant and research specializing in the impact of information systems and technologies on individuals and organizations. Her research has been published inMIS Quarterly, AI & Society, AI Expert, andInformation & Management.     

Improving children's mental rotation accuracy with computer game playing

The authors investigated the relation between mental rotation (MR) and computer game-playing experience. Third grade boys (n = 24) and girls (n = 23) completed a 2-dimensional MR test before and after playing computer games (during 11 separate 30-min sessions), which either involved the use of MR skills (the experimental group) or did not involve the use of MR skills (the control group). The experimental group outperformed the control group on the MR posttest but not on the pretest. Boys outperformed girls on the pretest but not on the posttest. Children whose initial MR performance was low improved after playing computer games that entailed MR skills. The findings imply that computer-based instructional activities can be used in schools to enhance children's spatial abilities.     

Computer-based systems for applying psychological knowledge

Computer-based systems have long been seen as a means for applying psychological knowledge to instructing. However, from many years of research and development on instructional design and on using computers for instructing, we learned that the current instructional techniques are based on an inadequate scientific knowledge base and are poorly implemented. In order to do better, improved techniques based on modern cognitive science must be developed, and computer-based aids are needed to assist their application. Furthermore, a standardization and distribution system is needed to identify effective instructional programs and make them available. This seems especially important if we hope to have broad impact on education and training.