Alternatives to Randomized Experiments

ABSTRACT— Randomized experiments are preferred for making inferences about causality when they can be implemented and their assumptions are met. Yet assumptions can fail (e.g., attrition, treatment noncompliance) or randomization may be unethical or infeasible. I describe alternative design and statistical approaches that permit testing causal hypotheses and present current empirical evidence related to alternative designs. Alternative designs permit a wider range of research questions to be answered and permit more direct generalization of causal effects; however, when using such designs, estimates of the magnitude of the causal effect may be more uncertain.     

Mood in chronic disease: Questioning the answers

The main research questions concerning mood and chronic disease are about the extent, sources, and consequences of emotional distress. Answers to these questions are informative both to psychological theory and to the design of interventions aimed at improving outcomes for patients. However, as a result of practical and ethical limitations in research designs, the answers obtained are rarely definitive and typically need to be interpreted carefully. This article illustrates some of the problems using studies of patients with cancer, myocardial infarction, stroke, and other disabling conditions. The first group of studies illustrates problems in the timing of mood measurement including both timing within an interview, and issues concerned with identifying equivalent timing in the stages of disease and its treatment. The remaining studies are used to illustrate the limits of non-experimental longitudinal, and of experimental designs. It is concluded that research in this area requires the use of more than one research paradigm to achieve dependable answers.     

Mood in chronic disease: Questioning the answers

The main research questions concerning mood and chronic disease are about the extent, sources, and consequences of emotional distress. Answers to these questions are informative both to psychological theory and to the design of interventions aimed at improving outcomes for patients. However, as a result of practical and ethical limitations in research designs, the answers obtained are rarely definitive and typically need to be interpreted carefully. This article illustrates some of the problems using studies of patients with cancer, myocardial infarction, stroke, and other disabling conditions. The first group of studies illustrates problems in the timing of mood measurement including both timing within an interview, and issues concerned with identifying equivalent timing in the stages of disease and its treatment. The remaining studies are used to illustrate the limits of non-experimental longitudinal, and of experimental designs. It is concluded that research in this area requires the use of more than one research paradigm to achieve dependable answers.     

A Child's-Eye View: An Examination of Point-of-View Camera Use in Four Informal Education Settings

A point-of-view (POV) camera is a video recording device attached to the person of a research participant. The cameras collect observational data from the individual wearing them and those in close proximity. The cameras can be used to collect data that provide researchers with an immersive view of visitor experience by documenting visits from the perspective of the individual. The purpose of this article is to outline the benefits and weaknesses of using POV cameras for answering broad visitor studies research questions, compare their use to other common methods for visitor studies research, and examine their use within four informal educational settings—a zoo, a museum exhibit, a bus tour, and an afterschool program. In particular, the POV cameras are useful for capturing data beyond a single exhibit, recording the dialogue and gestures of individual visitors, and documenting interactions among group members. POV cameras may be especially useful for answering questions about hands-on activities such as building or tinkering. Limitations that should be considered when attempting to use POV cameras include participants tampering with the recording, the inability to see facial expressions, and the limited angle of captured video. Care should also be taken to minimize the discomfort of visitors in studies utilizing POV cameras.     

The Impact of Facilitation by Museum Educators on Family Learning at Interactive Math Exhibits: A Quasi-Experimental Study

Although discussions of museums often revolve around exhibits, educators in these spaces have the potential to create in-depth, social learning experiences beyond what is possible at exhibits alone. There is still little empirical research, however, to inform how we understand, approach, and improve museum facilitation practices. In this study, we sought to address this gap by quantifying the impact of facilitation by trained educators working with visitors at interactive museum exhibits and comparing this to visitor engagement and learning outcomes for families without educator support. Using a quasi-experimental design, we measured the impact of staff facilitation on visitor engagement time, mathematical reasoning, math awareness, satisfaction, and intergenerational communication across three different exhibits, four trained educators, and two experimental conditions. Multivariate regression modeling showed that staff facilitation had a positive impact on engagement time, mathematical reasoning, and satisfaction, a negative impact on intergenerational communication, and no impact on math awareness.     

Individual differences methods for randomized experiments

Experiments allow researchers to randomly vary the key manipulation, the instruments of measurement, and the sequences of the measurements and manipulations across participants. To date, however, the advantages of randomized experiments to manipulate both the aspects of interest and the aspects that threaten internal validity have been primarily used to make inferences about the average causal effect of the experimental manipulation. This article introduces a general framework for analyzing experimental data to make inferences about individual differences in causal effects. Approaches to analyzing the data produced by a number of classical designs and 2 more novel designs are discussed. Simulations highlight the strengths and weaknesses of the data produced by each design with respect to internal validity. Results indicate that, although the data produced by standard designs can be used to produce accurate estimates of average causal effects of experimental manipulations, more elaborate designs are often necessary for accurate inferences with respect to individual differences in causal effects. The methods described here can be diversely applied by researchers interested in determining the extent to which individuals respond differentially to an experimental manipulation or treatment and how differential responsiveness relates to individual participant characteristics.     

Developmental psychology in the real world: A paradigm of parent education

Twenty years ago, William Kessen, Greta Fein and I developed and tested a model of parent education, a model which involved variation in curricular content, didactic approaches, and child outcomes–experimentally contrasted. Our experience yielded some valuable lessons about the extent to which researchers can influence parents' behaviour, and parents their children's development. In the present article I suggest that these lessons might be useful for researchers now as they were for us then. I present some observations about the research that developmental psychologists have done in the intervening two decades and suggest that we could learn much by using such a model of parent education to answer questions about the effets of parents' behaviour on children's development. Carefully done, parent education studies can yield valuable information about many of the questions raised in recent correlational research. Parent training research can suggest hypotheses that can be tested with data collected non-interventively and analysed in causal model analyses–and vice versa. What is more, because one reason for studying parent effects is so that we can offer guidance to parents about rearing their children, this design has a particular advantage; it is both the medium and the message.     

The intervention selection bias: an underrecognized confound in intervention research

Selection bias can be the most important threat to internal validity in intervention research, but is often insufficiently recognized and controlled. The bias is illustrated in research on parental interventions (punishment, homework assistance); medical interventions (hospitalization); and psychological interventions for suicide risk, sex offending, and juvenile delinquency. The intervention selection bias is most adequately controlled in randomized studies or strong quasi-experimental designs, although recent statistical innovations can enhance weaker designs. The most important points are to increase awareness of the intervention selection bias and to systematically evaluate plausible alternative explanations of data before making causal conclusions.     

Stable Causal Relationships Are Better Causal Relationships

We report three experiments investigating whether people's judgments about causal relationships are sensitive to the robustness or stability of such relationships across a range of background circumstances. In Experiment 1, we demonstrate that people are more willing to endorse causal and explanatory claims based on stable (as opposed to unstable) relationships, even when the overall causal strength of the relationship is held constant. In Experiment 2, we show that this effect is not driven by a causal generalization's actual scope of application. In Experiment 3, we offer evidence that stable causal relationships may be seen as better guides to action. Collectively, these experiments document a previously underappreciated factor that shapes people's causal reasoning: the stability of the causal relationship.     

Conceptualizing the Visitor Experience: A Review of Literature and Development of a Multifaceted Model

ABSTRACT

Although commonly considered the core product offered by visitor attractions such as museums, zoos, and heritage sites, the visitor experience has proved a difficult construct to both define and measure. This article reviews the concept of visitor experience, drawing from literature in tourism and leisure research as well as museum and visitor studies, and identifies a number of issues on which the literature presents multiple perspectives. By clarifying these issues, this article takes a first step toward building a shared vocabulary to describe and measure visitor experiences. The article presents a conceptual scheme that describes relationships among key factors, and a multifaceted model of the visitor experience that offers a way of characterizing both its content and intensity. The article thus provides a basis for future research designed to capture this elusive phenomenon.     

Learning,prediction and causal Bayes nets

Recent research in cognitive and developmental psychology on acquiring and using causal knowledge uses the causal Bayes net formalism, which simultaneously represents hypotheses about causal relations, probability relations, and effects of interventions. The formalism provides new normative standards for reinterpreting experiments on human judgment, offers a precise interpretation of mechanisms, and allows generalizations of existing theories of causal learning. Combined with hypotheses about learning algorithms, the formalism makes predictions about inferences in many experimental designs beyond the classical, Pavlovian cue-->effect design.     

Methodological Challenges in Causal Research on Racial and Ethnic Patterns of Cognitive Trajectories: Measurement, Selection, and Bias

Research focused on understanding how and why cognitive trajectories differ across racial and ethnic groups can be compromised by several possible methodological challenges. These difficulties are especially relevant in research on racial and ethnic disparities and neuropsychological outcomes because of the particular influence of selection and measurement in these contexts. In this article, we review the counterfactual framework for thinking about causal effects versus statistical associations. We emphasize that causal inferences are key to predicting the likely consequences of possible interventions, for example in clinical settings. We summarize a number of common biases that can obscure causal relationships, including confounding, measurement ceilings/floors, baseline adjustment bias, practice or retest effects, differential measurement error, conditioning on common effects in direct and indirect effects decompositions, and differential survival. For each, we describe how to recognize when such biases may be relevant and some possible analytic or design approaches to remediating these biases.     

Normative data for idiomatic expressions

Idiomatic expressions such as kick the bucket or go down a storm can differ on a number of internal features, such as familiarity, meaning, literality, and decomposability, and these types of features have been the focus of a number of normative studies. In this article, we provide normative data for a set of Bulgarian idioms and their English translations, and by doing so replicate in a Slavic language the relationships between the ratings previously found in Romance and Germanic languages. Additionally, we compared whether collecting these types of ratings in between-subjects or within-subjects designs affects the data and the conclusions drawn, and found no evidence that design type affects the final outcome. Finally, we present the results of a meta-analysis that summarizes the relationships found across the literature. As in many previous individual studies, we found that familiarity correlates with a number of other features; however, such studies have shown conflicting results concerning literality and decomposability ratings. The meta-analysis revealed reliable relationships of decomposability with a number of other measures, such as familiarity, meaning, and predictability. Conversely, literality was shown to have little to no relationship with any of the other subjective ratings. The implications for these relationships in the context of the wider experimental literature are discussed, with a particular focus on the importance of attaining familiarity ratings for each sample of participants in experimental work.     

Addressing confounding errors when using non-experimental, observational data to make causal claims

In their recent book, Is Inequality Bad for Our Health?, Daniels, Kennedy, and Kawachi claim that to “act justly in health policy, we must have knowledge about the causal pathways through which socioeconomic (and other) inequalities work to produce differential health outcomes.” One of the central problems with this approach is its dependency on “knowledge about the causal pathways.” A widely held belief is that the randomized clinical trial (RCT) is, and ought to be the “gold standard” of evaluating the causal efficacy of interventions. However, often the only data available are non-experimental, observational data. For such data, the necessary randomization is missing. Because the randomization is missing, it seems to follow that it is not possible to make epistemically warranted claims about the causal pathways. Although we are not sanguine about the difficulty in using observational data to make warranted causal claims, we are not as pessimistic as those who believe that the only warranted causal claims are claims based on data from (idealized) RCTs. We argue that careful, thoughtful study design, informed by expert knowledge, that incorporates propensity score matching methods in conjunction with instrumental variable analyses, provides the possibility of warranted causal claims using observational data.     

Improving causal inference from naturalistic designs: Self-selection and other issues in evaluating alcohol control policies

The capacity of naturalistic research designs to contribute to causal understanding is assessed in relation to evaluating alcohol control policies. Recent conceptual writings on evaluation research emphasize the importance of causal fields and complex models in making probabilistic causal inferences. Several new statistical approaches that promise help in drawing generalizable causal inferences from suitably developed naturalistic and quasi-experimental studies are discussed. Structural equation modeling augmented by automated heuristic search procedures, econometric techniques for selection bias adjustment, and theory-driven outcome pattern analysis are considered. Two examples from alcohol control policy evaluations are given. One involved use of selectivity adjustments to evaluate a highly selectivity-conditioned environmentally-based alcohol abuse prevention program. Many naturalistic evaluations are improved by modeling self-selected program exposure as well as program response. A second evaluation, now in progress, assesses impacts of the U.S. national alcohol warning labels policy. Problems facing the investigation are enumerated, highlighting the need for theorizing about mechanisms of influence, developing multiple lines of evidence, and examining patterns of outcomes for fit with theoretical expectations.     

A Quasi-Experimental Analysis of the Association Between Family Income and Offspring Conduct Problems

The study presents a quasi-experimental analysis of data on 9,194 offspring (ages 4–11 years old) of women from a nationally representative U.S. sample of households to test the causal hypotheses about the association between family income and childhood conduct problems (CPs). Comparison of unrelated individuals in the sample indicated a robust inverse association, with the relation being larger at higher levels of income and for male offspring, even when statistical covariates were included to account for measured confounds that distinguish different families. Offspring also were compared to their siblings and cousins who were exposed to different levels of family income in childhood to rule out unmeasured environmental and genetic factors confounded with family income as explanations for the association. In these within-family analyses, boys exposed to lower family income still exhibited significantly higher levels of CPs. When considered in the context of previous studies using different designs, these results support the inference that family income influences CPs, particularly in males, through causal environmental processes specifically related to earnings within the nuclear family. This research was supported by grant R01-MH070025 to Benjamin B. Lahey. The research was approved by the Institutional Review Board at Indiana University. Resubmitted to the Journal of Abnormal Child Psychology on April 21, 2008 and October 20, 2008.     

Investigating Sustained Visitor/Museum Relationships: Employing Narrative Research in the Field of Museum Visitor Studies

ABSTRACT

Museums today are faced with a number of challenges and are currently searching for ways to attract and retain audiences and engage with the public. In order to address these issues, researchers have called for studies to help construct a more complete understanding of the visitor experience. Few studies employ methodological approaches that provide in-depth accounts of the museum experience. Narrative methodology is one means by which researchers may access rich accounts of the multi-faceted nature of audience relationships with museums. This article reports preliminary findings from a narrative research study that aimed to explore sustained visitor/museum relationships. The narrative methodology adopted for this study provides novel insights into the nature of these relationships and extends our understanding of those museum experiences that build and strengthen relationships with visitors.     

The co-therapy relationship: Mutuality,agreement and client outcome

This project, utilizing a pretest and posttest design, studied interactions between co-therapists and the relationships of their interactions to client outcome. Six co-therapist pairs at a counseling center saw two clients in multiple treatments yielding twelve cases. The study investigated co-therapist relationships in regard to: (a) their level of mutuality or caring for their partners, (b) their ratings of the quality of their relationships, and (c) their agreement as to perceptions and behavior during sessions. Also investigated were: the relation of co-therapist interaction to client outcome and the personal growth of the therapists over the course of therapy. Results supported success claims of the literature concerning co-therapy.     

Learning the form of causal relationships using hierarchical bayesian models

People learn quickly when reasoning about causal relationships, making inferences from limited data and avoiding spurious inferences. Efficient learning depends on abstract knowledge, which is often domain or context specific, and much of it must be learned. While such knowledge effects are well documented, little is known about exactly how we acquire knowledge that constrains learning. This work focuses on knowledge of the functional form of causal relationships; there are many kinds of relationships that can apply between causes and their effects, and knowledge of the form such a relationship takes is important in order to quickly identify the real causes of an observed effect. We developed a hierarchical Bayesian model of the acquisition of knowledge of the functional form of causal relationships and tested it in five experimental studies, considering disjunctive and conjunctive relationships, failure rates, and cross-domain effects. The Bayesian model accurately predicted human judgments and outperformed several alternative models.