An alternative approach to analysis of mental states in experimental social cognition research

Establishing the mental states that affect human behavior is a primary goal of experiments on social cognitive processes. Such mental states can be manipulated only indirectly; therefore, after delivering a manipulation, researchers attempt to verify that the mental state of interest, the representation of a mental state, was in fact changed by the manipulation and that this change caused the observed effect. The usual procedure is to examine mean differences in a measure of the mental state of interest (a manipulation check) among experimental conditions and to infer whether the manipulation was effective. We describe a procedure that strengthens the construct validity of manipulations and, hence, causal inferences in experiments that focus on mental states using analyses familiar to most researchers. This procedure employs a traditional manipulation check that assesses the relationship between manipulations and mental states but, additionally, tests the relationship between the manipulation check and dependent measure.     

Marginal mean weighting through stratification: a generalized method for evaluating multivalued and multiple treatments with nonexperimental data

Propensity score matching and stratification enable researchers to make statistical adjustment for a large number of observed covariates in nonexperimental data. These methods have recently become popular in psychological research. Yet their applications to evaluations of multi-valued and multiple treatments are limited. The inverse-probability-of-treatment weighting method, though suitable for evaluating multi-valued and multiple treatments, often generates results that are not robust when only a portion of the population provides support for causal inference or when the functional form of the propensity score model is misspecified. The marginal mean weighting through stratification (MMW-S) method promises a viable nonparametric solution to these problems. By computing weights on the basis of stratified propensity scores, MMW-S adjustment equates the pretreatment composition of multiple treatment groups under the assumption that unmeasured covariates do not confound the treatment effects given the observed covariates. Analyzing data from a weighted sample, researchers can estimate a causal effect by computing the difference between the estimated average potential outcomes associated with alternative treatments within the analysis of variance framework. After providing an intuitive illustration of the theoretical rationale underlying the weighting method for causal inferences, the article demonstrates how to apply the MMW-S method to evaluations of treatments measured on a binary, ordinal, or nominal scale approximating a completely randomized experiment; to studies of multiple concurrent treatments approximating factorial randomized designs; and to moderated treatment effects approximating randomized block designs. The analytic procedure is illustrated with an evaluation of educational services for English language learners attending kindergarten in the United States.     

Diffusion versus linear ballistic accumulation: different models but the same conclusions about psychological processes?

Quantitative models for response time and accuracy are increasingly used as tools to draw conclusions about psychological processes. Here we investigate the extent to which these substantive conclusions depend on whether researchers use the Ratcliff diffusion model or the Linear Ballistic Accumulator model. Simulations show that the models agree on the effects of changes in the rate of information accumulation and changes in non-decision time, but that they disagree on the effects of changes in response caution. In fits to empirical data, however, the models tend to agree closely on the effects of an experimental manipulation of response caution. We discuss the implications of these conflicting results, concluding that real manipulations of caution map closely, but not perfectly to response caution in either model. Importantly, we conclude that inferences about psychological processes made from real data are unlikely to depend on the model that is used.     

Predictive versus diagnostic causal learning: Evidence from an overshadowing paradigm

Causal directionality belongs to one of the most fundamental aspects of causality that cannot be reduced to mere covariation. This paper is part of a debate between proponents of associative theories, which claim that learners are insensitive to the causal status of cues and outcomes, and proponents of causal-model theory, which postulates an interaction of assumptions about causal directionality and learning. Some researchers endorsing the associationist view have argued that evidence for the interaction between cue competition and causal directionality may be restricted to two-phase blocking designs. Furthermore, from the viewpoint of causal-model theory, blocking designs carry the potential problem that the predicted asymmetries of cue competition are partly dependent on asymmetries of retrospective inferences. The present experiments use a one-phase overshadowing paradigm that does not allow for retrospective inferences and therefore represents a more unambiguous test of sensitivity to causal directionality. The results strengthen causal-model theory by clearly demonstrating the influence of causal directionality on learning. However, they also provide evidence for boundary conditions for this effect by highlighting the role of the semantics of the learning task.     

Experimental and Quasi-Experimental Designs in Visitor Studies: A Critical Reflection on Three Projects

Identifying causal relationships is an important aspect of research and evaluation in visitor studies, such as making claims about the learning outcomes of a program or exhibit. Experimental and quasi-experimental approaches are powerful tools for addressing these causal questions. However, these designs are arguably underused in visitor studies. In this article, we offer examples of the use of experimental and quasi-experimental designs in science museums to aide investigators interested in expanding their methods toolkit and increasing their ability to make strong causal claims about programmatic experiences or relationships among variables. Using three designs from recent research (fully randomized experiment, posttest only quasi-experimental design with comparison condition, and posttest with independent pretest design), we discuss challenges and tradeoffs related to feasibility, participant experience, alignment with research questions, and internal and external validity. We end the article with broader reflections on the role of experimental and quasi-experimental designs in visitor studies.     

Addressing two blind spots of commonly used experimental designs: The Highly‐Repeated Within‐Person approach

Two well documented but still neglected blind spots of often‐used study designs limit a researcher's ability to make inferences about psychological phenomenon. First, typical designs focus on effects of conditions at the group level and are not able to assess the extent to which effects characterize each participant in the study. This blind spot can lead to erroneous (or incomplete) conclusions about the effects of manipulations both for a given participant and at the group level. Second, commonly used research designs often use a limited sample of stimuli, constraining conclusions to the particular stimuli. This blind spot can lead to non‐replication when different stimuli are used. We propose that the Highly‐Repeated Within‐Person (HRWP) approach helps mitigate these limitations. Using a study on the effects of anti‐smoking messages, we illustrate how the HRWP approach helps alert researchers when the conclusions at the group level may not apply to all (or any) participant, quantifies the heterogeneity of effects of manipulations across people, and increases confidence regarding the generalizability of the effects. We discuss how the HRWP approach may help conceptualize issues of replicability in a new light.     

Testing mediational models with longitudinal data: questions and tips in the use of structural equation modeling

R. M. Baron and D. A. Kenny (1986; see record 1987-13085-001) provided clarion conceptual and methodological guidelines for testing mediational models with cross-sectional data. Graduating from cross-sectional to longitudinal designs enables researchers to make more rigorous inferences about the causal relations implied by such models. In this transition, misconceptions and erroneous assumptions are the norm. First, we describe some of the questions that arise (and misconceptions that sometimes emerge) in longitudinal tests of mediational models. We also provide a collection of tips for structural equation modeling (SEM) of mediational processes. Finally, we suggest a series of 5 steps when using SEM to test mediational processes in longitudinal designs: testing the measurement model, testing for added components, testing for omitted paths, testing the stationarity assumption, and estimating the mediational effects.     

Causal learning across domains

Five studies investigated (a) children's ability to use the dependent and independent probabilities of events to make causal inferences and (b) the interaction between such inferences and domain-specific knowledge. In Experiment 1, preschoolers used patterns of dependence and independence to make accurate causal inferences in the domains of biology and psychology. Experiment 2 replicated the results in the domain of biology with a more complex pattern of conditional dependencies. In Experiment 3, children used evidence about patterns of dependence and independence to craft novel interventions across domains. In Experiments 4 and 5, children's sensitivity to patterns of dependence was pitted against their domain-specific knowledge. Children used conditional probabilities to make accurate causal inferences even when asked to violate domain boundaries.     

Proceeding From Observed Correlation to Causal Inference: The Use of Natural Experiments

ABSTRACT— This article notes five reasons why a correlation between a risk (or protective) factor and some specified outcome might not reflect environmental causation. In keeping with numerous other writers, it is noted that a causal effect is usually composed of a constellation of components acting in concert. The study of causation, therefore, will necessarily be informative on only one or more subsets of such components. There is no such thing as a single basic necessary and sufficient cause. Attention is drawn to the need (albeit unobservable) to consider the counterfactual (i.e., what would have happened if the individual had not had the supposed risk experience). Fifteen possible types of natural experiments that may be used to test causal inferences with respect to naturally occurring prior causes (rather than planned interventions) are described. These comprise five types of genetically sensitive designs intended to control for possible genetic mediation (as well as dealing with other issues), six uses of twin or adoptee strategies to deal with other issues such as selection bias or the contrasts between different environmental risks, two designs to deal with selection bias, regression discontinuity designs to take into account unmeasured confounders, and the study of contextual effects. It is concluded that, taken in conjunction, natural experiments can be very helpful in both strengthening and weakening causal inferences.     

Amazon Mechanical Turk in Organizational Psychology: An Evaluation and Practical Recommendations

Purpose

Amazon Mechanical Turk is an increasingly popular data source in the organizational psychology research community. This paper presents an evaluation of MTurk and provides a set of practical recommendations for researchers using MTurk.

Design/Methodology/Approach

We present an evaluation of methodological concerns related to the use of MTurk and potential threats to validity inferences. Based on our evaluation, we also provide a set of recommendations to strengthen validity inferences using MTurk samples.

Findings

Although MTurk samples can overcome some important validity concerns, there are other limitations researchers must consider in light of their research objectives. Researchers should carefully evaluate the appropriateness and quality of MTurk samples based on the different issues we discuss in our evaluation.

Implications

There is not a one-size-fits-all answer to whether MTurk is appropriate for a research study. The answer depends on the research questions and the data collection and analytic procedures adopted. The quality of the data is not defined by the data source per se, but rather the decisions researchers make during the stages of study design, data collection, and data analysis.

Originality/Value

The current paper extends the literature by evaluating MTurk in a more comprehensive manner than in prior reviews. Past review papers focused primarily on internal and external validity, with less attention paid to statistical conclusion and construct validity—which are equally important in making accurate inferences about research findings. This paper also provides a set of practical recommendations in addressing validity concerns when using MTurk.

     

Assessment of accurate self-knowledge

Despite ongoing theoretical interest in the accuracy of self-knowledge and its implications for mental health, few researchers have yet to tackle this topic directly. This may be due, in part, to several factors that make assessing individual differences in accurate self-knowledge especially difficult. In this article, we present a method for the assessment of accurate self-knowledge that relies on information gathered from the self, knowledgeable others, and observations of behavior in the laboratory, and we provide psychometric support for this newly developed assessment procedure. Specifically, we present evidence for internal consistency reliability, convergent and discriminant validity, and criterion-related validity. Other researchers interested in studying the accuracy of self-knowledge might wish to adopt this procedure in their own research endeavors.     

Learning,prediction and causal Bayes nets

Recent research in cognitive and developmental psychology on acquiring and using causal knowledge uses the causal Bayes net formalism, which simultaneously represents hypotheses about causal relations, probability relations, and effects of interventions. The formalism provides new normative standards for reinterpreting experiments on human judgment, offers a precise interpretation of mechanisms, and allows generalizations of existing theories of causal learning. Combined with hypotheses about learning algorithms, the formalism makes predictions about inferences in many experimental designs beyond the classical, Pavlovian cue-->effect design.     

Finding the Cause: Verbal Framing Helps Children Extract Causal Evidence Embedded in a Complex Scene

In making causal inferences, children must both identify a causal problem and selectively attend to meaningful evidence. Four experiments demonstrate that verbally framing an event (“Which animals make Lion laugh?”) helps 4-year-olds extract evidence from a complex scene to make accurate causal inferences. Whereas framing was unnecessary when evidence was isolated, children required it to extract and reason about evidence embedded in a more complex scene. Subtler framing stating the causal problem, but not highlighting the relevant variables, was equally effective. Simply making the causal relationship more perceptually obvious did facilitate children's inferences, but not to the level of verbal framing. These results illustrate how children's causal reasoning relies on scaffolding from adults.     

Independence and dependence in human causal reasoning

Causal graphical models (CGMs) are a popular formalism used to model human causal reasoning and learning. The key property of CGMs is the causal Markov condition, which stipulates patterns of independence and dependence among causally related variables. Five experiments found that while adult’s causal inferences exhibited aspects of veridical causal reasoning, they also exhibited a small but tenacious tendency to violate the Markov condition. They also failed to exhibit robust discounting in which the presence of one cause as an explanation of an effect makes the presence of another less likely. Instead, subjects often reasoned “associatively,” that is, assumed that the presence of one variable implied the presence of other, causally related variables, even those that were (according to the Markov condition) conditionally independent. This tendency was unaffected by manipulations (e.g., response deadlines) known to influence fast and intuitive reasoning processes, suggesting that an associative response to a causal reasoning question is sometimes the product of careful and deliberate thinking. That about 60% of the erroneous associative inferences were made by about a quarter of the subjects suggests the presence of substantial individual differences in this tendency. There was also evidence that inferences were influenced by subjects’ assumptions about factors that disable causal relations and their use of a conjunctive reasoning strategy. Theories that strive to provide high fidelity accounts of human causal reasoning will need to relax the independence constraints imposed by CGMs.     

All data collection and analysis methods have limitations: reply to Rabbitt (2011) and Raz and Lindenberger (2011)

The commentaries on my article contain a number of points with which I disagree but also several with which I agree. For example, I continue to believe that the existence of many cases in which between-person variability does not increase with age indicates that greater variance with increased age is not inevitable among healthy individuals up to about 80 years of age. I also do not believe that problems of causal inferences from correlational information are more severe in the cognitive neuroscience of aging than in other research areas; I contend instead that neglect of these problems has led to confusion about neurobiological underpinnings of cognitive aging. I agree that researchers need to be cautious in extrapolating from cross-sectional to longitudinal relations, but I also note that even longitudinal data are limited with respect to their ability to support causal inferences.     

Conditional reasoning, representation, and empirical evidence on a concrete task

This study examined interactions between empirical data, internal representations, and reasoning performance on a conditional reasoning task using a concrete apparatus. Subjects were asked an initial series of questions in order to determine the pattern of inferences they made after simple exposure to the apparatus. They were subsequently shown two different experimental manipulations designed to provide data about the internal structure of the apparatus without giving information about specific inferences. Some subjects did change their reasoning in response to the new data, although most remained stable throughout the experiment. These results are consistent with the idea that reasoning may require generation of an internal representation of a problem space. It was also concluded that the relation between reasoning and empirical evidence cannot be understood without supposing that evidence is often interpreted by subjects according to their reasoning patterns.     

What will Phil do next? : Spontaneously inferred traits influence predictions of behavior

Past research has shown that perceivers intentionally may make trait inferences about others and use this information to make predictions about these others' future behaviors. Other research has also shown that people can make trait inferences without intent—that is, spontaneously. However, one unexplored avenue is whether spontaneous trait inferences (STI), affect how perceivers predict others' will behavior. Three studies explored this issue. Results from Studies 1 and 2 showed that: (1) exposure to trait-implicative behaviors describing an actor influences subsequent behavior predictions made about the actor in a trait-consistent manner, and (2) predictions occurred regardless of behavior recall, implying that the behavior predictions were derived from prior trait inferences and not from behavior recall. Results from Study 3 bolstered this conclusion by showing that behavior predictions were similar regardless of whether subjects were explicitly instructed to make inferences or not, but that a manipulation known to interfere with inference generation (lie detection instructions) muted behavior predictions. Results from Study 3 also suggested that the prediction effects had both automatic and controlled components, and that reductions observed in the lie detection condition of Study 3 were caused by alterations in the automatic influence of trait knowledge to the behavior predictions. These results suggest that STI may be causal inferences about the actors' dispositions.     

Alternatives to Randomized Experiments

ABSTRACT— Randomized experiments are preferred for making inferences about causality when they can be implemented and their assumptions are met. Yet assumptions can fail (e.g., attrition, treatment noncompliance) or randomization may be unethical or infeasible. I describe alternative design and statistical approaches that permit testing causal hypotheses and present current empirical evidence related to alternative designs. Alternative designs permit a wider range of research questions to be answered and permit more direct generalization of causal effects; however, when using such designs, estimates of the magnitude of the causal effect may be more uncertain.     

因果变化模式与因果联结强度在幼儿因果推理中的作用

对93名幼儿进行了五种因果变化模式的因果推理题目的测试。结果表明:(1)在不同的因果变化模式下,被试进行因果推理的成绩存在差异,且在对于这五类题目的掌握上具有一定的顺序。(2)被试在同样因果变化模式题目的表现之间具有较高的相似性,而在因果联结强度相同的题目之间则具有显著的差异。(3)被试对于各题目回答的正确率并不随原因与结果联结次数的增多而提高。(4)即使是在观察到的刺激完全一致的情况下,被试的回答仍会因因果变化模式的差异及主试对于题目解释的不同而存在差别。     

Analogical and category-based inference: a theoretical integration with Bayesian causal models

A fundamental issue for theories of human induction is to specify constraints on potential inferences. For inferences based on shared category membership, an analogy, and/or a relational schema, it appears that the basic goal of induction is to make accurate and goal-relevant inferences that are sensitive to uncertainty. People can use source information at various levels of abstraction (including both specific instances and more general categories), coupled with prior causal knowledge, to build a causal model for a target situation, which in turn constrains inferences about the target. We propose a computational theory in the framework of Bayesian inference and test its predictions (parameter-free for the cases we consider) in a series of experiments in which people were asked to assess the probabilities of various causal predictions and attributions about a target on the basis of source knowledge about generative and preventive causes. The theory proved successful in accounting for systematic patterns of judgments about interrelated types of causal inferences, including evidence that analogical inferences are partially dissociable from overall mapping quality.