Confidence intervals and replication: where will the next mean fall?

Confidence intervals (CIs) give information about replication, but many researchers have misconceptions about this information. One problem is that the percentage of future replication means captured by a particular CI varies markedly, depending on where in relation to the population mean that CI falls. The authors investigated the distribution of this percentage for varsigma known and unknown, for various sample sizes, and for robust CIs. The distribution has strong negative skew: Most 95% CIs will capture around 90% or more of replication means, but some will capture a much lower proportion. On average, a 95% CI will include just 83.4% of future replication means. The authors present figures designed to assist understanding of what CIs say about replication, and they also extend the discussion to explain how p values give information about replication.     

Types of inconsistency in health-state utility judgments

In making judgments of health-related quality of life, respondents often compare the relative magnitude of two intervals between health states, such as the interval between normal health and blindness compared to that between normal health and death. We examined two ways of comparing such intervals: person-trade-off (PTO)—in which the judgment concerns matching numbers of people so that two changes are equivalent—and direct judgment of the ratio. Both measures showed ratio inconsistency (a ratio that should be the product of two other ratios is too high) and superadditivity (two ratios that should add to 1 are too high). Some responses in both methods implied that two intervals which should have been different (because they shared a top or bottom point, but differed on the other point) were nevertheless viewed by subjects as being of equal size. These equality responses were more common when death was the bottom (worse end) of both intervals being compared (e.g., the interval between death and blindness is perceived as being the same size as the interval between death and normal health) than when any other condition was at the bottom or when the condition common to the two intervals was at the top. A second experiment indicated that subjects really do consider the intervals to be equal. Our findings argue for giving subjects a chance to reflect on such apparent inconsistencies in practical utility elicitation.     

More than 50% or Less than 70% Chance: Pragmatic Implications of Single‐Bound Probability Estimates

Probability estimates can be given as ranges or uncertainty intervals, where often only one of the interval bounds (lower or upper) is specified. For instance, a climate forecast can describe La Niña as having “more than 70% chance” or “less than 90% chance” of occurring. In three experiments, we studied how research participants perceived climate‐related forecasts expressed with lower‐bound (“over X% chance”) or upper‐bound (“under Y% chance”) probability statements. Results indicate that such single‐bound statements give pragmatic information in addition to the numeric probabilities they convey. First, the studies show that these statements are directional, leading the listeners' attention in opposite directions. “Over” statements guide attention towards the possible occurrence of the event and are explained by reasons for why it might happen, while “under” statements direct attention to its possible non‐occurrence and are more often explained by reasons for why the target event might not appear, corresponding to positive (it is possible) versus negative (it is uncertain) verbal probabilities. Second, boundaries were found to reveal the forecaster's beliefs and could be perceived as indicative of an increasing or a decreasing trend. Single‐bound probability estimates are therefore not neutral communications of probability level but might “leak” information about the speaker's expectations and about past and future developments of the forecast. Copyright © 2017 John Wiley & Sons, Ltd.     

Overconfidence in interval estimates: What does expertise buy you?

People’s 90% subjective confidence intervals typically contain the true value about 50% of the time, indicating extreme overconfidence. Previous results have been mixed regarding whether experts are as overconfident as novices. Experiment 1 examined interval estimates from information technology (IT) professionals and UC San Diego (UCSD) students about both the IT industry and UCSD. This within-subjects experiment showed that experts and novices were about equally overconfident. Experts reported intervals that had midpoints closer to the true value—which increased hit rate—and that were narrower (i.e., more informative)—which decreased hit rate. The net effect was no change in hit rate and overconfidence. Experiment 2 showed that both experts and novices mistakenly expected experts to be much less overconfident than novices, but they correctly predicted that experts would provide narrower intervals with midpoints closer to the truth. Decisions about whether to consult experts should be based on which aspects of performance are desired.     

Hedonic Evaluation over Short and Long Retention Intervals: The Mechanism of the Peak–End Rule

The peak–end rule is used to explain how people make retrospective hedonic evaluations. This study advances our understanding of its mechanism by identifying the different effects of the rule on such evaluations over short and long retention intervals. The results of two experiments show that (i) respondents constructed their retrospective hedonic evaluations on the basis of the peak and end affects only over a short retention interval, not over a long one; and (ii) respondents relied on episodic information to construct their evaluations over a short retention interval, whereas they relied on both semantic and episodic information to construct their evaluations over a long retention interval. Our study also suggests that the “watershed” between short and long retention intervals is likely to lie between 3 and 7 weeks. Copyright © 2012 John Wiley & Sons, Ltd.     

Duration of dogs' (<Emphasis Type="Italic">Canis familiaris</Emphasis>) working memory in search for disappearing objects

Two experiments explored the duration of dogs' working memory in an object permanence task: a delay was introduced between the disappearance of a moving object behind a box and the beginning of the search by the animal. In experiment 1, the dogs were tested with retention intervals of 0, 10, 30, and 60 s. Results revealed that the dogs' accuracy declined as a function of the length of the retention interval but remained above chance for each retention interval. In experiment 2, with new subjects, longer retention intervals (0, 30, 60, 120, and 240 s) were presented to the dogs. Results replicated findings from experiment 1 and revealed that the dogs' accuracy remained higher than chance level with delays up to 240 s. In both experiments, the analysis of errors also showed that the dogs searched as a function of the proximity of the target box and were not subject to intertrial proactive interference. In the discussion, we explore different alternatives to explain why dogs' search behaviour for hidden objects decreased as a function of the retention intervals. Electronic Publication     

Intuitive Political Theory: People's Judgments About How Groups Should Decide

Societies must make collective decisions even when citizens disagree, and they use many different political processes to do so. But how do people choose one way to make a group decision over another? We propose that the human mind contains an intuitive political theory about how to make collective decisions, analogous to people's intuitive theories about language, physics, number, minds, and morality. We outline a simple method for studying people's intuitive political theory using scenarios about group decisions, and we begin to apply this approach in three experiments. Participants read scenarios in which individuals in a group have conflicting information (Experiment 1), conflicting interests (Experiment 2), and conflicting interests between a majority and a vulnerable minority who have more at stake (Experiment 3). Participants judged whether the group should decide by voting, consensus, leadership, or chance. Overall, we find that participants prefer majority‐rule voting over consensus, leadership, and chance when a group has conflicting interests or information. However, participants' support for voting is considerably diminished when the group includes a vulnerable minority. Hence, participants showed an intuitive understanding of Madison's concerns about tyranny of the majority.     

The fallacy of placing confidence in confidence intervals

Interval estimates – estimates of parameters that include an allowance for sampling uncertainty – have long been touted as a key component of statistical analyses. There are several kinds of interval estimates, but the most popular are confidence intervals (CIs): intervals that contain the true parameter value in some known proportion of repeated samples, on average. The width of confidence intervals is thought to index the precision of an estimate; CIs are thought to be a guide to which parameter values are plausible or reasonable; and the confidence coefficient of the interval (e.g., 95 %) is thought to index the plausibility that the true parameter is included in the interval. We show in a number of examples that CIs do not necessarily have any of these properties, and can lead to unjustified or arbitrary inferences. For this reason, we caution against relying upon confidence interval theory to justify interval estimates, and suggest that other theories of interval estimation should be used instead.     

An alternative to null-hypothesis significance tests

The statistic p(rep) estimates the probability of replicating an effect. It captures traditional publication criteria for signal-to-noise ratio, while avoiding parametric inference and the resulting Bayesian dilemma. In concert with effect size and replication intervals, p(rep) provides all of the information now used in evaluating research, while avoiding many of the pitfalls of traditional statistical inference.     

Tune recognition with reduced pitch and interval information

Twenty subjects were tested on their ability to recognize simple tunes from which rhythm information had been removed. Only the first phrase of each tune was presented. The purpose of the experiment was (a) to determine whether stimuli containing only high harmonics can evoke a sense of musical pitch, and (b) to provide a set of data in normal subjects with which the performance of deaf subjects whose auditory nerve is stimulated electrically can be compared. Each subject was tested on five sets of stimuli presented in a counterbalanced order. These stimuli were (I) pulse trains high-pass filtered at 2 kHz, with repetition rates in the range of 100-200 p.p.s.; (2) as in (I) but high-pass filtered at 4 kHz; (3) sinusoids with musical intervals compressed, so that the “octave” was a ratio of I:I·3; (4) sinusoids with the musical intervals expanded, so that the “octave” was a ratio of I:4; (5) sinusoids of a constant frequency in which the normal frequency changes were translated into intensity changes, each semitone being represented by a 3 dB change in level. The results indicate that a pattern of intensity changes does not support tune recognition, and that, although the pitch contour alone allows reasonable performance, subjects do use musical interval information in recognizing tunes. Stimuli containing only high harmonics can provide such interval information, and thus can evoke a sense of musical pitch. Preliminary results from a deaf subject stimulated electrically with an electrode on the surface of the cochlea indicate that such stimulation can also evoke a sense of musical pitch. It is concluded that musical pitch information can be carried in the time-pattern of nerve impulses in the auditory nerve.     

Spike coding from the perspective of a neurone

In this paper, we compare existing methods for quantifying the coding capacity of a spike train, and review recent developments in the application of information theory to neural coding. We present novel methods for characterising single-unit activity based on the perspective of a downstream neurone and propose a simple yet universally applicable framework to characterise the order of complexity of neural coding by single units. We establish four orders of complexity in the capacity for neural coding. First-order coding, quantified by firing rates, is conveyed by frequencies and is thus entirely described by first moment processes. Second-order coding, represented by the variability of interspike intervals, is quantified by the log interval entropy. Third-order coding is the result of spike motifs that associate adjacent inter-spike intervals beyond chance levels; it is described by the joint interval histogram, and is measured by the mutual information between adjacent log intervals. Finally, nonstationarities in activity represent coding of the fourth-order that arise from the effects of a known or unknown stimulus.     

单维测验合成信度三种区间估计的比较

已有许多研究建议使用合成信度来估计测验信度, 并报告其置信区间。有三种方法或途径可以计算单维测验合成信度的置信区间, 包括Bootstrap法、Delta法和直接用统计软件(如LISREL)输出的标准误进行计算。本文通过模拟研究进行比较, 发现Delta法与Bootstrap法得到的置信区间相当接近, 但用LISREL输出的标准误计算的与Bootstrap法得到的结果相差很大。推荐用Delta法估计合成信度的置信区间(使用Mplus容易实现), 但不能直接用LISREL输出的标准误来计算。举例说明了如何计算单维测验的合成信度以及用Delta法计算其置信区间。     

Do the Weak Stand a Chance? Distribution of Resources in a Competitive Environment

When two agents of unequal strength compete, the stronger one is expected to always win the competition. This expectation is based on the assumption that evaluation of performance is complete, hence flawless. If, however, the agents are evaluated on the basis of only a small sample of their performance, the weaker agent still stands a chance of winning occasionally. A theoretical analysis indicates that, to increase the chance of this happening the weaker agent ought to give up on enough occasions so that he or she can match the stronger agent on the remaining ones. We model such a competition in a game, present its game-theoretic solution, and report an experiment, involving 144 individuals, in which we tested whether players (both weak and strong) are actually sensitive to their relative strengths and know how to allocate their resources accordingly. Our results indicate that they do.     

Optimal and Most Exact Confidence Intervals for Person Parameters in Item Response Theory Models

The common way to calculate confidence intervals for item response theory models is to assume that the standardized maximum likelihood estimator for the person parameter θ is normally distributed. However, this approximation is often inadequate for short and medium test lengths. As a result, the coverage probabilities fall below the given level of significance in many cases; and, therefore, the corresponding intervals are no longer confidence intervals in terms of the actual definition. In the present work, confidence intervals are defined more precisely by utilizing the relationship between confidence intervals and hypothesis testing. Two approaches to confidence interval construction are explored that are optimal with respect to criteria of smallness and consistency with the standard approach.     

Constructing Confidence Intervals for Effect Size Measures of an Indirect Effect

Confidence intervals for an effect size can provide the information about the magnitude of an effect and its precision as well as the binary decision about the existence of an effect. In this study, the performances of five different methods for constructing confidence intervals for ratio effect size measures of an indirect effect were compared in terms of power, coverage rates, Type I error rates, and widths of confidence intervals. The five methods include the percentile bootstrap method, the bias-corrected and accelerated (BCa) bootstrap method, the delta method, the Fieller method, and the Monte Carlo method. The results were discussed with respect to the adequacy of the distributional assumptions and the nature of ratio quantity. The confidence intervals from the five methods showed similar results for samples of more than 500, whereas, for samples of less than 500, the confidence intervals were sufficiently narrow to convey the information about the population effect sizes only when the effect sizes of regression coefficients defining the indirect effect are large.     

Spatial memory in rats after 25 hours

We investigated the time course of spatial-memory decay in rats using an eight-arm radial maze. It is well established that performance remains high with retention intervals as long as 4 h, but declines to chance with a 24-h retention interval (Beatty, W. W., & Shavalia, D. A. (1980b). Spatial memory in rats: time course of working memory and effect of anesthetics. Behavioral & Neural Biology, 28, 454-462). It is possible that 24 h reflects a genuine retention limitation of rat spatial memory. Alternatively, it may be possible to identify factors that might support memory performance even after very long delays. The current experiment was conducted to test the above two hypotheses. We evaluated performance using two intertrial intervals (24 and 48 h) and two retention intervals (1 and 25 h). Increasing the intertrial interval produced an approximately constant increase in performance for both retention intervals. This improvement is consistent with a trial-spacing effect (i.e., the superiority of spaced over massed trials). Rat spatial memory apparently lasts at least 25 h.     

Replicability and randomization test logic in behavior analysis

Randomization tests are a class of nonparametric statistics that determine the significance of treatment effects. Unlike parametric statistics, randomization tests do not assume a random sample, or make any of the distributional assumptions that often preclude statistical inferences about single‐case data. A feature that randomization tests share with parametric statistics, however, is the derivation of a p‐value. P‐values are notoriously misinterpreted and are partly responsible for the putative “replication crisis.” Behavior analysts might question the utility of adding such a controversial index of statistical significance to their methods, so it is the aim of this paper to describe the randomization test logic and its potentially beneficial consequences. In doing so, this paper will: (1) address the replication crisis as a behavior analyst views it, (2) differentiate the problematic p‐values of parametric statistics from the, arguably, more useful p‐values of randomization tests, and (3) review the logic of randomization tests and their unique fit within the behavior analytic tradition of studying behavioral processes that cut across species.     

Accuracy in parameter estimation for targeted effects in structural equation modeling: sample size planning for narrow confidence intervals

In addition to evaluating a structural equation model (SEM) as a whole, often the model parameters are of interest and confidence intervals for those parameters are formed. Given a model with a good overall fit, it is entirely possible for the targeted effects of interest to have very wide confidence intervals, thus giving little information about the magnitude of the population targeted effects. With the goal of obtaining sufficiently narrow confidence intervals for the model parameters of interest, sample size planning methods for SEM are developed from the accuracy in parameter estimation approach. One method plans for the sample size so that the expected confidence interval width is sufficiently narrow. An extended procedure ensures that the obtained confidence interval will be no wider than desired, with some specified degree of assurance. A Monte Carlo simulation study was conducted that verified the effectiveness of the procedures in realistic situations. The methods developed have been implemented in the MBESS package in R so that they can be easily applied by researchers.     

Sample Size Planning for the Squared Multiple Correlation Coefficient: Accuracy in Parameter Estimation via Narrow Confidence Intervals

Methods of sample size planning are developed from the accuracy in parameter approach in the multiple regression context in order to obtain a sufficiently narrow confidence interval for the population squared multiple correlation coefficient when regressors are random. Approximate and exact methods are developed that provide necessary sample size so that the expected width of the confidence interval will be sufficiently narrow. Modifications of these methods are then developed so that necessary sample size will lead to sufficiently narrow confidence intervals with no less than some desired degree of assurance. Computer routines have been developed and are included within the MBESS R package so that the methods discussed in the article can be implemented. The methods and computer routines are demonstrated using an empirical example linking innovation in the health services industry with previous innovation, personality factors, and group climate characteristics.