Arbitrary metrics in psychology

Many psychological tests have arbitrary metrics but are appropriate for testing psychological theories. Metric arbitrariness is a concern, however, when researchers wish to draw inferences about the true, absolute standing of a group or individual on the latent psychological dimension being measured. The authors illustrate this in the context of 2 case studies in which psychologists need to develop inventories with nonarbitrary metrics. One example comes from social psychology, where researchers have begun using the Implicit Association Test to provide the lay public with feedback about their "hidden biases" via popular Internet Web pages. The other example comes from clinical psychology, where researchers often wish to evaluate the real-world importance of interventions. As the authors show, both pursuits require researchers to conduct formal research that makes their metrics nonarbitrary by linking test scores to meaningful real-world events.     

Conditions for the acceptance of deontic conditionals.

Recent psychological research has investigated how people assess the probability of an indicative conditional. Most people give the conditional probability of q given p as the probability of if p then q. Asking about the probability of an indicative conditional, one is in effect asking about its acceptability. But on what basis are deontic conditionals judged to be acceptable or unacceptable? Using a decision theoretic analysis, we argue that a deontic conditional, of the form if p then must q or if p then may q, will be judged acceptable to the extent that the p & q possibility is preferred to the p & not-q possibility. Two experiments are reported in which this prediction was upheld. There was also evidence that the pragmatic suitability of permission rules is partly determined by evaluations of the not-p & q possibility. Implications of these results for theories of deontic reasoning are discussed.     

The frequentist implications of optional stopping on Bayesian hypothesis tests

Null hypothesis significance testing (NHST) is the most commonly used statistical methodology in psychology. The probability of achieving a value as extreme or more extreme than the statistic obtained from the data is evaluated, and if it is low enough, the null hypothesis is rejected. However, because common experimental practice often clashes with the assumptions underlying NHST, these calculated probabilities are often incorrect. Most commonly, experimenters use tests that assume that sample sizes are fixed in advance of data collection but then use the data to determine when to stop; in the limit, experimenters can use data monitoring to guarantee that the null hypothesis will be rejected. Bayesian hypothesis testing (BHT) provides a solution to these ills because the stopping rule used is irrelevant to the calculation of a Bayes factor. In addition, there are strong mathematical guarantees on the frequentist properties of BHT that are comforting for researchers concerned that stopping rules could influence the Bayes factors produced. Here, we show that these guaranteed bounds have limited scope and often do not apply in psychological research. Specifically, we quantitatively demonstrate the impact of optional stopping on the resulting Bayes factors in two common situations: (1) when the truth is a combination of the hypotheses, such as in a heterogeneous population, and (2) when a hypothesis is composite—taking multiple parameter values—such as the alternative hypothesis in a t-test. We found that, for these situations, while the Bayesian interpretation remains correct regardless of the stopping rule used, the choice of stopping rule can, in some situations, greatly increase the chance of experimenters finding evidence in the direction they desire. We suggest ways to control these frequentist implications of stopping rules on BHT.     

Who’s afraid of the big bad wolf? The interweaving of values and science

The role of values in the scientific process is widely debated. But evidence and values cannot be neatly separated. Instead, values infuse the entire scientific process, starting with the choice of research questions. Research avenues are selected based on prior beliefs about the workings of the world. In fact, informally assigned prior probabilities and normalizing constants play an essential role in distinguishing causes from correlations and ignoring irrelevant associations that would otherwise be suggested by raw data. But since these initial probabilities often have a heterogeneous pedigree – often not all that went into their formation is known or knowable – it becomes difficult to cleanly separate evidence from values. At the same time, this reality does not undermine science’s unique status in generating knowledge. This is because it is not the evidence per se generated by science, but rather the method by which science generates it, that justifies science’s privileged role.     

Hail the impossible: p-values, evidence, and likelihood

Significance testing based on p-values is standard in psychological research and teaching. Typically, research articles and textbooks present and use p as a measure of statistical evidence against the null hypothesis (the Fisherian interpretation), although using concepts and tools based on a completely different usage of p as a tool for controlling long-term decision errors (the Neyman-Pearson interpretation). There are four major problems with using p as a measure of evidence and these problems are often overlooked in the domain of psychology. First, p is uniformly distributed under the null hypothesis and can therefore never indicate evidence for the null. Second, p is conditioned solely on the null hypothesis and is therefore unsuited to quantify evidence, because evidence is always relative in the sense of being evidence for or against a hypothesis relative to another hypothesis. Third, p designates probability of obtaining evidence (given the null), rather than strength of evidence. Fourth, p depends on unobserved data and subjective intentions and therefore implies, given the evidential interpretation, that the evidential strength of observed data depends on things that did not happen and subjective intentions. In sum, using p in the Fisherian sense as a measure of statistical evidence is deeply problematic, both statistically and conceptually, while the Neyman-Pearson interpretation is not about evidence at all. In contrast, the likelihood ratio escapes the above problems and is recommended as a tool for psychologists to represent the statistical evidence conveyed by obtained data relative to two hypotheses.     

Exploring MMPI-Rorschach relationships

Limitations exist in the current literature examining Minnesota Multiphasic Personality Inventory (MMPI)-Rorschach interrelationships. In this article I attempt to identify several issues to be addressed by researchers in this area. These include restricting conclusions about MMPI-Rorschach relationships to research using the Comprehensive System; considering the relative reliability and validity of these tests; comparing the predictive power of the Rorschach and MMPI; examining the effects extreme outliers have on statistical analyses; and determining the appropriate level of analysis for research, the level of individual test scores, or psychological constructs. The implications of these issues for research design, statistical analysis, and applied clinical work are discussed.     

Detecting and predicting changes

When required to predict sequential events, such as random coin tosses or basketball free throws, people reliably use inappropriate strategies, such as inferring temporal structure when none is present. We investigate the ability of observers to predict sequential events in dynamically changing environments, where there is an opportunity to detect true temporal structure. In two experiments we demonstrate that participants often make correct statistical decisions when asked to infer the hidden state of the data generating process. However, when asked to make predictions about future outcomes, accuracy decreased even though normatively correct responses in the two tasks were identical. A particle filter model accounts for all data, describing performance in terms of a plausible psychological process. By varying the number of particles, and the prior belief about the probability of a change occurring in the data generating process, we were able to model most of the observed individual differences.     

警惕心理学研究中的统计误用

正确地使用统计方法对心理学研究至关重要。但心理学研究中存在着诸多统计误用现象。该着重分析了心理学研究中常见的一些统计误用现象：小或有偏样本的使用、在概率的理解上存在偏差、显性检验问题、夸大的统计图、相关分析的误用等等,并在此基础上提出了提高研究自身的统计素养、强调理论在心理学研究中的重要性、运用多样化的研究方法等应对之策。     

解释水平视角下的乐观偏差效应

研究基于解释水平理论,考查心理距离对乐观偏差效应的影响。实验从心理距离的事件概率维度、时间距离维度、空间距离维度入手,采用自编的材料对677名被试进行测量,探讨心理距离的远近对乐观偏差产生的影响。结果表明,被试对未来生活事件的判断存在总体上的乐观偏差。在较远的心理距离条件下,被试表现出更大的乐观偏差;而在较近的心理距离中,乐观偏差效应明显减小。但在时间距离维度以及消极事件的概率维度上,表现出与总体乐观偏差不一致的现象。     

When personal history repeats itself: Decomposing memories for recurring events

Retrospective reports of personal events constitute the principal data source for much applied social science research. Such reports are known to be subject to biases and errors of inference. In health surveys, for example, events associated with chronic conditions are particularly likely to be under-reported (Jabine, 1987). In this series of studies we apply perspectives and tools from psychological studies of memory to the problem of understanding and improving reports of past untilization of health care. In two studies we found that subjects report using different strategies to answer questions about how often they have seen a doctor over the past 12 months, depending on the type of doctor's visit and its frequency. A third study, in which subjects' recall was checked against medical records, provided evidence that individual health events within a group of recurring, similar events are much less likely than non-recurring events to be recalled. In Experiment 4 we tested the efficacy of cognitively oriented intervention techniques designed to (1) help subjects decompose their molar memories of groups of similar events into individual events and (2) place all remembered health events into a personal timeline. The intervention more than doubled the proportion of recurring events on the medical record that was recalled. In addition, the proportion of events that could be dated, and the accuracy of dates provided, both rose.     

Effects of Situational Judgment Test Format on Reliability and Validity

Single-response situational judgment tests (SRSJTs) differ from multiple-response SJTs (MRSJTS) in that they present test takers with edited critical incidents and simply ask test takers to read over the action described and evaluate it according to its effectiveness. Research comparing the reliability and validity of SRSJTs and MRSJTs is thus far extremely limited. The study reported here directly compares forms of a SRSJT and MRSJT and explores the reliability, convergent validity, and predictive validity of each format. Results from this investigation present preliminary evidence to suggest SRSJTs may produce internal consistency reliability, convergent validity, and predictive validity estimates that are comparable to those achieved with many traditional MRSJTs. We conclude by discussing practical implications for personnel selection and assessment, and future research in psychological science more broadly.     

Assessing positive and negative changes in the aftermath of adversity: psychometric evaluation of the changes in outlook questionnaire

The Changes in Outlook Questionnaire (CiOQ; S. Joseph, R. Williams, & W. Yule, 1993) is a 26-item self-report measure that was designed to assess positive and negative changes in the aftermath of adversity. This article had 3 aims: 1st, to investigate the factor structure of the CiOQ; 2nd, to test for internal consistency reliability and convergent and discriminant validity; and, 3rd, to investigate the association between positive and negative changes in outlook, posttraumatic stress, and psychological distress. Three studies are reported. Study 1 provides evidence that positive and negative changes are statistically separable and that the 2-factor model is a better fit than the 1-factor model. Studies 2 and 3 provide evidence for internal consistency reliability, convergent and discriminant validity of the CiOQ, and its associations with posttraumatic stress and psychological distress. In conclusion, the CiOQ has much promise for research on responses to stressful and traumatic events.     

Exploring MMPI-Rorschach Relationships

Limitations exist in the current literature examining Minnesota Multiphasic Personality Inventory (MMPI)—Rorschach interrelationships. In this article I attempt to identify several issues to be addressed by researchers in this area. These include restricting conclusions about MMPI—Rorschach relationships to research using the Comprehensive System; considering the relative reliability and validity of these tests; comparing the predictive power of the Rorschach and MMPI; examining the effects extreme outliers have on statistical analyses; and determining the appropriate level of analysis for research, the level of individual test scores, or psychological constructs. The implications of these issues for research design, statistical analysis, and applied clinical work are discussed.     

Distress in spouses of Vietnam veterans: associations with communication about deployment experiences

Emerging literature shows a consistent pattern of relationship and psychological distress in spouses or partners of combat veterans with symptoms of posttraumatic stress disorder (PTSD). One possible mechanism of partners' distress posited in clinical literature is that excessive discussion of traumatic events from deployment may have negative effects on partners. At the extreme, some partners are suggested to develop PTSD-like symptoms, or secondary traumatic stress. Despite these hypotheses, there have been few empirical tests of the effects of communicating about such events. In a sample of 465 combat veterans and their spouses who participated in the Family Interview Component of the National Vietnam Veterans Readjustment Study, we explored how the extent of couples' deployment-related communication was associated with partner relationships and psychological distress, and whether such associations were moderated by the severity of veterans' PTSD symptoms. Results showed that Vietnam-specific communication correlated negatively with relationship distress, but the effect was negligible after controlling for overall communication in the relationship. On the other hand, Vietnam-specific communication did not correlate with psychological distress, but the association was significantly moderated by veterans' PTSD symptom severity. Specifically, communication about Vietnam was increasingly and positively associated with partners' psychological distress as veterans' symptoms of PTSD rose into the clinical range, but nonsignificantly and negatively associated with such distress as PTSD symptoms decreased below this level. The findings support previous clinical recommendations that couples' discussions of potentially traumatic events be approached cautiously, and they suggest a need to attend to the content of couples' communications when conducting dyadic interventions for PTSD.     

Understanding conjunction effects in probability judgments: the role of implicit mental models

Subjective probability judgments often violate a normative principle in that the conjunction of two events is judged to be more likely than the probability of either of the two events occurring separately. Most previous explanations of these conjunction effects have assumed that probability judgments depend on some psychological relation (e.g. representativeness) between the constituents mentioned explicitly in the stimulus information. In contrast, the present approach highlights the fundamental role of implicitly inferred information. Participants are assumed to transform the explicit stimulus information into implicit mental models in their attempt to make sense of the experimental task. Probability judgments should then reflect the degree of activation of such a mental model in memory given a set of propositions, rather than the quantitative fit or likelihood of the propositions themselves. Two studies are reported which provide converging evidence for the proposed mental model approach. In the first study, using graded conjunctions of one to five propositions, probability judgments are shown to vary as a function of the activation of a mental model rather than the likelihood of the component events. In a second study, a priming procedure is employed to activate mental models that either fit an event conjunction or do not, leading to an increase or decrease of conjunction effects in probability judgment. Copyright © 1999 John Wiley & Sons, Ltd.     

Discovering syntactic deep structure via Bayesian statistics

In the Bayesian framework, a language learner should seek a grammar that explains observed data well and is also a priori probable. This paper proposes such a measure of prior probability. Indeed it develops a full statistical framework for lexicalized syntax. The learner's job is to discover the system of probabilistic transformations (often called lexical redundancy rules) that underlies the patterns of regular and irregular syntactic constructions listed in the lexicon. Specifically, the learner discovers what transformations apply in the language, how often they apply, and in what contexts. It considers simpler systems of transformations to be more probable a priori. Experiments show that the learned transformations are more effective than previous statistical models at predicting the probabilities of lexical entries, especially those for which the learner had no direct evidence.     

Methodology is more than research design and technology

The Society for Computers in Psychology has been at the forefront of disseminating information about advances in computer technology and their applications for psychologists. Although technological advances, as well as clean research designs, are key contributors to progress in psychological research, the justification of methodological rules for interpreting data and making theory choices is at least as important. Historically, methodological beliefs and practices have been justified through intuition and logic, an approach known asfoundationism. However,naturalism, a modern approach in the philosophy of science inspired by the work of Thomas S. Kuhn, indicates that all aspects of scientific practice, including its methodology, should be evaluated empirically. This article examines implications of the naturalistic approach for psychological research methods in general and for the current debate that is often framed as one of qualitative versus quantitative methods.     

差数显著性t检验与元分析的对比研究

利用计算机构造被试总体、模拟实验研究程序进行抽样研究,探讨显著性ｔ检验方法与元分析方法在检验实验结果数据方面的差异。在模拟实验过程中,ｔ验受到显著性水平、样本容量和总体效果大小的影响,因此最终影响了统计推断的可靠性,建议：在进行显著性检验过程中,应对统计检验能力进行估计;元分析方法以样本为元素对总体进行推断,因此具有很高的准确性和可靠性,它将很有可能成为今后心理学研究的重要统计工具。