Whither (European) evaluation methodology?

Quasi-experimental evaluation methods promised to provide a way of testing and improving solutions to social problems. These methods have been found wanting. Alternative approaches, including “pragmatic evaluation,” “naturalistic evaluation,” and “pluralist evaluation” are also shown to be unsatisfactory. The initial promise of the evaluation movement has been disappointed. Realistic evaluation is advocated as an alternative to existing forms of evaluation. It is rooted in some European traditions in epistemology, ontology, and social theory. It offers a framework within which rigorous outcome evaluations can be undertaken with results that can inform policy and practice. He is president of the Research Committee on Methodology of the International Sociological Association. He is seconded part-time to the Home Office Police Research Group as a research consultant. An earlier version of this paper was presented at the founding conference of the European Evaluation Society, The Hague, 1–2 December, 1994.     

The clinical approach: Triangulated program evaluation and adjustment

In the discussion about “enlightenment” or “utilization” in program evaluation, it is increasingly clear that the discussants implicitly refer to different professional contexts. Weiss, in Alkin (1990), using the “scientist” approach appears to reflect upon the academic context of the traditional university, where, beyond the land grant institutions, “enlightenment” is the honored objective. In contrast, Patton (Alkin, 1990) speaks from the clinical perspective of the organization consultant, with “utilization” the essential element in the evaluator-client relationship. Yet, those contextual differences notwithstanding, each party defends its case on the implicit assumption ofone methodological procedure of program evaluation—equally shared by both. Correcting this assumption, this article articulates major principles and methods of the “clinical” approach in program evaluation. The method has been tested in the field in western Europe and the United States. Its characteristic difference with the academic tradition is that in clinical evaluation, improving the program is part of the method. Mark van de Vall is professor of Sociology, Erasmus University Rotterdam and adjunct professor at SUNY / Buffalo.     

Thomas E. Jordan, A Review of “The Quality of Life in Seventeenth-Century Ireland”

In “The Quality of Life in Seventeenth-Century Ireland,” Thomas Jordan brings a historian’s perspective to quality-of-life studies. The book raises questions about the determinants of quality-of-life, tangible outcomes and appropriate measures to assess societal well-being.     

Disciplinary roots and branches of evaluation: Some lessons from agricultural research

Since its origins in North America in the 1960s, the field of program evaluation has grown considerably, and its concerns have broadened from accountability to program improvement, decision support, and institutional learning. Program evaluation is now commonly practiced in governmental organizations not only in North America but also in many countries of Western Europe and Oceania. Although program evaluation is a relatively new field with many controversies and lively debates, a unifying body of evaluation theory, methods, and standards is gradually emerging. Evaluation has recently been described as a “transdiscipline,” as are statistics and measurement. This article is based largely on my personal experiences working in agricultural research organizations in developing regions. Here, a number of different types of evaluation are carried out, but program evaluation as defined by Patton (1997) and as practiced by social scientists to assess public programs is largely unknown. Distinct branches of agricultural research evaluation can be identified, with disciplinary roots in the natural sciences and in agricultural economics. The most rigorous agricultural research evaluations are economic studies. Systematic internal evaluation is notably lacking. Current pressures to improve performance, transparency, and accountability are creating demands for more systematic evaluation, and many program evaluation concepts and methods would seem to be of value in agricultural research organizations. However, in the current scenario of declining funding for agricultural research, managers are yet to be convinced to expand their evaluation activities and explore unfamiliar paradigms and methods. Moreover, they are not yet convinced that social-science-based program evaluation would produce useful results. Natural scientists and economists tend to view program evaluation as “soft-science” or no science at all. Douglas Horton works at the International Service for National Agricultural Research (ISNAR) in The Netherlands. Since joining ISNAR in early 1990, he has done research, training, and advisory work on agricultural research management, with an emphasis on evaluation. Previously, for fifteen years, Horton was head of the social science department of the International Potato Center (CIP) in Peru. With colleagues at CIP and in national agricultural research organizations, he documented patterns and trends in world potato production and use, engaged in participatory technology development and assessed the impact of CIP programs. Horton received B.S. and M.S. degrees in agricultural economics from the University of Illinois, and a Ph.D. in economics from Cornell University. While at Cornell, Horton worked with W.F. Whyte, pioneer in the fields of participant observation and participatory action research.     

Does the compatibility effect in the race Implicit Association Test reflect familiarity or affect?

In the Implicit Association Test (IAT; Greenwald, McGhee, & Schwartz, 1998) involving race classification (white vs. black), an apparent compatibility effect is found between the “pleasant” attribute and the “white” category. This race IAT effect has been interpreted in terms of “implicit prejudice”—that is, more positive evaluation of whites than of blacks that is not open to consciousness. We suggested instead that the race IAT effect is better interpreted in terms of the salience asymmetry account proposed by Rothermund and Wentura (2004), whereby greater familiarity with the white category makes it more salient. Evidence that has been presented against the familiarity interpretation is considered, and alternative interpretations of findings related to the race IAT effect are discussed.     

One process is not enough! A speed-accuracy tradeoff study of recognition memory

Speed-accuracy tradeoff (SAT) methods have been used to contrast single- and dual-process accounts of recognition memory. In these procedures, subjects are presented with individual test items and are required to make recognition decisions under various time constraints. In this experiment, we presented word lists under incidental learning conditions, varying the modality of presentation and level of processing. At test, we manipulated the interval between each visually presented test item and a response signal, thus controlling the amount of time available to retrieve target information. Study— test modality match had a beneficial effect on recognition accuracy at short response-signal delays (≤300msec). Conversely, recognition accuracy benefited more from deep than from shallow processing at study only at relatively long response-signal delays (≥300 msec). The results are congruent with views suggesting that both fast familiarity and slower recollection processes contribute to recognition memory.     

What Determines Perceived Quality-of-life Impact of Mobile Phones? A Model Based on the Consumption Life Cycle

This paper reports two studies designed to identify the determinants of perceived quality-of-life impact (PQOLI) of mobile phones. We hypothesized that PQOLI of mobile phones is determined by mostly global feelings of satisfaction with mobile phones, which in turn are determined by satisfaction with a broad range of customer-related experiences—experience with the purchase of the mobile phone and service, preparing the mobile phone for personal use, using the mobile phone, owning the mobile phone, maintenance and repairs of the mobile phone, and disposal of the mobile phone (stages in the consumption life cycle). “Study 1” focused testing the model using a college student population. “Study 2” was designed to replicate the findings of “Study 1” with a more mature and diverse consumer population and extend the model by demonstrating that PQOLI has a significant predictive influence on brand loyalty, thus underscoring the managerial utility of our model. The study results are supportive of our overall model and its hypotheses.     

Placebo and the helsinki declaration — What to do?

The Helsinki Declaration is the ‘gold standard’ — a directive, not a law, on how to conduct controlled studies in humans in conformity with ethical principles. In spite of many discussions about their unsuitability some articles have remained unchanged in the most recent (sixth) revision of the Declaration. The demand to use “the best treatment” excludes use of placebo in the control group and presents an obstacle to the scientific evaluation of a number of drugs and treatments in general. The use of placebo is justified whenever its use does not cause irreversible damage or considerable suffering to the well informed patient. It must be, is, and will be used in the controlled clinical trials of treatments of a great number of diseases especially those which have a tendency to spontaneous improvement, even healing, or have a pronounced psychological component An earlier version of this paper was presented at an international conference, “Placebo: Its Action and Place in Health Research Today,” held in Warsaw, Poland on 12–13 April, 2003.     

Emotional Design; Application of a Research-Based Design Approach

In this paper, we discuss an approach to ‘design for wow’ that focuses on the emotions that constitute a wow-experience. In this approach, the eliciting conditions of these emotions are used to define a product character with a high wow-impact. In addition to the approach, a measurable wow-index is introduced. First, a concept of wow is described in which wow is explained as a combination of fascination, pleasant surprise, and desire. The eliciting conditions of these three emotions are examined and combined to a ‘wow-appraisal.’ This concept is applied in the design of a mobile telephone. A combination of qualitative and quantitative research methods was used to formulate concern themes relevant for the wow-appraisal. These themes were used to create a layered product character and a prototype of the final design. An evaluation study demonstrated that the product designed with this approach rated higher on the wow-index than alternative products.     

Evaluation in the private sector: Evolution and professionalization of quality management

This article reviews different approaches to quality management in the private sector from an evolutionary perspective. Two dimensions of this evolution are crucial: the scope of evaluation (“product” versus “system”) and the objective versus intersubjective orientation. Experiences within the private sector can be relevant for the public sector, with adaptation for public sector characteristics. Complexity of both object and process of evaluation has implications for the professionalization of the evaluation function.     

Discussions

Summary  In their paper, ‘When are thought experiments poor ones?’ (Peijnenburg and David Atkinson, 2003, Journal of General Philosophy of Science 34, 305-322.), Jeanne Peijnenburg and David Atkinson argue that most, if not all, philosophical thought experiments are “poor” ones with “disastrous consequences” and that they share the property of being poor with some (but not all) scientific thought experiments. Noting that unlike philosophy, the sciences have the resources to avoid the disastrous consequences, Peijnenburg and Atkinson come to the conclusion that the use of thought experiments in science is in general more successful than in philosophy and that instead of concocting more “recherché” thought experiments, philosophy should try to be more empirical. In this comment I will argue that Peijnenburg’s and Atkinson’s view on thought experiments is based on a misleading characterization of both, the dialectical situation in philosophy as well as the history of physics. By giving an adequate account of what the discussion in contemporary philosophy is about, we will arrive at a considerably different evaluation of philosophical thought experiments.

For I am convinced that we now find ourselves at an altogether decisive turning point in philosophy, and that we are objectively justified in considering that an end has come to the fruitless conflict of systems. We are already at the present time, in my opinion, in possession of methods which make any such conflict in principle unnecessary. What is now required is their resolute application. (Schlick, ‘The Turning Point in Philosophy’, 1930/1959, p. 54).

     

Homeopathic remedies as placebo alternatives — verification on the example of treatment of menopause-related vegetative and emotional disturbances

With the example of treatment of menopause-related vegetative and emotional disturbances, the author verifies the effectiveness of the use of Ignatia amara containing complex homeopathic remedies (IACCHR) as an alternative to placebo. Substantial improvement in psychological and psychosomatic symptoms was observed. Climacteric complaints diminished or disappeared completely in the majority of women (95.7% by patient evaluation and 96.2% by physician evaluation). Compared to standard pharmaceuticals, IACCHR treatment was tolerated better and lower risk of side effects was observed. The results obtained in this work indicate the significant therapeutic potential of this group of treatments, which is in line with the therapeutic effect of the placebo. Nevertheless, the showing of specific effects in pharmacological tests disqualifies the investigated treatments from use in a clinical trial in place of a placebo. An earlier version of this paper was presented at an international conference, “Placebo: Its Action and Place in Health Research Today,” held in Warsaw, Poland on 12–13 April, 2003.     

“Put Your Money Where Your Love Is”: Parental Aid to Adult Children

Qualitative data are used to examine parents’ support to their adult children, and their motivations and feelings about it. The sample is 40 adult children and parents from four racial/ethnic groups: African-Americans, Asian-Americans, non-Latino Whites, and Latinos. Ideals of adult independence versus interdependence emerge as determinants of feelings about aiding adult children. The ambivalence concept provides the means to elucidate the tensions between these ideologies and children’s structurally shaped circumstances. Parental assistance to adult children was seen as acceptable for education, home ownership, and on behalf of young children, especially if the child was “working hard.” Parents still provided aid, however, even if these conditions were not met. In these situations, more intrafamilial tension characterized the aid transfer.     

Wittgenstein’s ‘Non-Cognitivism’ – Explained and Vindicated

The later Wittgenstein advanced a revolutionary but puzzling conception of how philosophy ought to be practised: Philosophical problems are not to be coped with by establishing substantive claims or devising explanations or theories. Instead, philosophical questions ought to be treated ‘like an illness’. Even though this ‘non-cognitivism’ about philosophy has become a focus of debate, the specifically ‘therapeutic’ aims and ‘non-theoretical’ methods constitutive of it remain ill understood. They are motivated by Wittgenstein’s view that the problems he addresses result from misinterpretation, driven by ‘urges to misunderstand’. The present paper clarifies this neglected concept and analyses how such ‘urges’ give rise to pseudo-problems of one particular, hitherto little understood, kind. This will reveal ‘therapeutic’ aims reasonable and ‘non-theoretical’ methods necessary, in one clearly delineated and important part of philosophy. I.e.: By developing a novel account of nature and genesis of one important class of philosophical problems, the paper explains and vindicates a revolutionary reorientation of philosophical work, at the level of both aims and methods.     

Introducing ethics across the curriculum at South Dakota school of mines and technology

This paper describes how the Electrical and Computer Engineering Department at South Dakota School of Mines and Technology has chosen to integrate ethics into their curriculum. All university freshmen engineering students are introduced to ethics through the presentation of ethical dilemmas. During this exercise, students are forced to argue both sides (‘for’ and ‘against’) of a hypothetical ethical engineering dilemma. It provides a setting for great discussion with the desired outcome that they learn to carefully analyze a situation before they draw conclusions. In the sophomore year, students are introduced to methods to use the fundamental principles, the fundamental canons, and the suggested guidelines for use with the fundamental canons of ethics when analyzing appropriate action to be taken when confronted with ethical dilemmas. We currently use the ‘sophomore’ method for seniors because the sequencing is just beginning. Next year the seniors will do more indepth analysis of ethical case studies. An earlier version of this paper was presented at the Fourteenth Annual Meeting, Association for Practical and Professional Ethics, February 24–27, 2005.     

Animal experimentation

Millions of animals are used every year in oftentimes extremely painful and distressing scientific procedures. Legislation of animal experimentation in modern societies is based on the supposition that this is ethically acceptable when certain more or less defined formal (e.g. logistical, technical) demands and ethical principles are met. The main parameters in this context correspond to the “3Rs” concept as defined by Russel and Burch in 1959, i.e. that all efforts to replace, reduce and refine experiments must be undertaken. The licensing of animal experiments normally requires an ethical evaluation process, oftentimes undertaken by ethics committees. The serious problems in putting this idea into practice include inter alia unclear conditions and standards for ethical decisions, insufficient management of experiments undertaken for specific (e.g. regulatory) purposes, and conflicts of interest of ethics committees’ members. There is an ongoing societal debate about ethical issues of animal use in science. Existing EU legislation on animal experimentation for cosmetics testing is an example of both the public will for setting clear limits to animal experiments and the need to further critically examine other fields and aspects of animal experimentation. This paper was presented at the 6th International Bioethics Conference on the subject of ‘The Responsible Conduct of Basic and Clinical Research’, held in Warsaw, Poland, 3–4 June 2005. The paper has been expanded from an article to be published in the forthcoming Council of Europe publication entitled “Animal Protection and Welfare”.     

Scientific societies and research integrity: What are they doing and how well are they doing it?

Scientific societies can play an important role in promoting ethical research practices among their members, and over the past two decades several studies have addressed how societies perform this role. This survey continues this research by examining current efforts by scientific societies to promote research integrity among their members. The data indicate that although many of the societies are working to promote research integrity through ethics codes and activities, they lack rigorous assessment methods to determine the effectiveness of their efforts. An earlier version of this paper was presented at the AAAS-Office of Research Integrity (ORI) meeting in Washington DC on the theme: “The Role and Activities of Scientific Societies in Promoting Research Integrity” held on April 10–11, 2000.     

Elementary, my dear Wason: The role of problem representation in the THOG task

The present study was concerned with Wason's THOG problem, a hypothetico-deductive reasoning task for which performance over the past 20 years has typically been very poor (<20% correct). We examined the hypothesis that incorporating a quasi-visual context into the problem statement would make both the binary, symmetric tree structure and solution principle of the THOG task clearer and thus facilitate performance. A version of O'Brien et al.'s (Q J Exp Psychol 42A:329–351) Blackboard THOG problem, that specifies each branch of the tree by describing a specific location for each possible color-shape combination, was used to test this hypothesis. Substantial facilitation was both observed (68% correct) and replicated (73% correct), and it was also shown that it is necessary to provide a representation of both sides of the tree to obtain this level of facilitation. The implications of these results for human deductive reasoning are considered. Received: 1 November 2000 / Accepted: 4 May 2001