Sample size and correlational inference

In 4 studies, the authors examined the hypothesis that the structure of the informational environment makes small samples more informative than large ones for drawing inferences about population correlations. The specific purpose of the studies was to test predictions arising from the signal detection simulations of R. B. Anderson, M. E. Doherty, N. D. Berg, and J. C. Friedrich (2005). The results of a simulation study in the present article confirmed and extended previous theoretical claims (R. B. Anderson et al., 2005) that in a yes/no correlation detection task, small-sample advantages should occur but should be restricted to particular decision conditions. In 3 behavioral studies, participants viewed larger or smaller samples of data pairs and judged whether each sample had been drawn from a population characterized by a zero correlation or from one characterized by a greater-than-zero correlation. Consistent with traditional statistical theory, accuracy tended to be greater for larger than for smaller samples, though there was a small-sample advantage in 1 experimental condition. The results are discussed in relation to alternative theoretical and behavioral paradigms such as those of Y. Kareev (e.g., 2005) and K. Fiedler and Y. Kareev (2006).     

Sample size and the detection of means: A signal detection account

Using statistical theory as a basis, Kareev (e.g., 1995) claimed that people's ability to correctly infer the existence of a population correlation should be greater for small than for large samples. Simulations by R. B. Anderson, Doherty, Berg, and Friedrich (2005) identified conditions favoring small samples but could not determine whether such an advantage was due to sampling skew, variance, or central tendency displacement. In the present study, we investigated theoretical effects of sample size (n) on the detection of population means under circumstances in which sampling variance is unconfounded with skew or central tendency displacement. The results demonstrate an extremely limited, criterion-specific, small-sample advantage that was attributable to n-related sampling variance and that occurred only with highly conservative, suboptimal criterion placement.     

Capacity limitations and the detection of correlations: comment on Kareev (2000)

Y. Kareev (2000) argued that the limited capacity of working memory may be an adaptive advantage for the early detection of useful correlations. His analysis indeed suggests that the optimal sample size is close to G. A. Miller's (1956) "magical number 7 +/- 2." The authors point out logical and statistical limitations of Y. Kareev's (2000) analysis, including that it neglects that the adaptive value is not determined by the hit rate but by the posterior probability of hit and that only signal trials are considered. The authors' analysis demonstrates that when these limitations are corrected for, the alleged benefit for small samples does not occur, and larger samples imply considerable improvement in the detection of correlations.     

Information overload or search-amplified risk? Set size and order effects on decisions from experience

How do changes in choice-set size influence information search and subsequent decisions? Moreover, does information overload influence information processing with larger choice sets? We investigated these questions by letting people freely explore sets of gambles before choosing one of them, with the choice sets either increasing or decreasing in number for each participant (from two to 32 gambles). Set size influenced information search, with participants taking more samples overall, but sampling a smaller proportion of gambles and taking fewer samples per gamble, when set sizes were larger. The order of choice sets also influenced search, with participants sampling from more gambles and taking more samples overall if they started with smaller as opposed to larger choice sets. Inconsistent with information overload, information processing appeared consistent across set sizes and choice order conditions, reliably favoring gambles with higher sample means. Despite the lack of evidence for information overload, changes in information search did lead to systematic changes in choice: People who started with smaller choice sets were more likely to choose gambles with the highest expected values, but only for small set sizes. For large set sizes, the increase in total samples increased the likelihood of encountering rare events at the same time that the reduction in samples per gamble amplified the effect of these rare events when they occurred—what we call search-amplified risk. This led to riskier choices for individuals whose choices most closely followed the sample mean.     

Estimating psychometric functions in forced-choice situations: significant biases found in threshold and slope estimations when small samples are used

When a theoretical psychometric function is fitted to experimental data (as in the obtaining of a psychophysical threshold), maximum-likelihood or probit methods are generally used. In the present paper, the behavior of these curve-fitting methods is studied for the special case of forced-choice experiments, in which the probability of a subject's making a correct response by chance is not zero. A mathematical investigation of the variance of the threshold and slope estimators shows that, in this case, the accuracy of the methods is much worse, and their sensitivity to the way data are sampled is greater, than in the case in which chance level is zero. Further, Monte Carlo simulations show that, in practical situations in which only a finite number of observations are made, the mean threshold and slope estimates are significantly biased. The amount of bias depends on the curve-fitting method and on the range of intensity values, but it is always greater in forced-choice situations than when chance level is zero.     

It's in the sample: the effects of sample size and sample diversity on the breadth of inductive generalization

Developmental studies have provided mixed evidence with regard to the question of whether children consider sample size and sample diversity in their inductive generalizations. Results from four experiments with 105 undergraduates, 105 school-age children (M = 7.2 years), and 105 preschoolers (M = 4.9 years) showed that preschoolers made a higher rate of projections from large samples than from small samples when samples were diverse (Experiments 1 and 3) but not when samples were homogeneous (Experiment 4) and not when the task required a choice between two samples (Experiment 2). Furthermore, when a property occurred in large and diverse samples, preschoolers exhibited a broad pattern of projection, generalizing the property to items from categories not represented in the evidence. In contrast, adults followed a normative pattern of induction and never attributed properties to items from categories not represented in the evidence. School-age children showed a mixed pattern of results.     

On the role of differential sample behaviors in matching-to-sample

Pigeons were trained on matching-to-sample (MTS) with differential sample-response requirements that were identical with respect to two pairs of sample stimuli but were either correlated or uncorrelated with correct choice. Experiment 1A showed that birds in the uncorrelated condition were slower to reach criterion levels of accuracy than birds in the correlated condition in spite of their equivalent sample discriminations. However, correlated birds were more disrupted in their matching performances than the uncorrelated birds when subsequently switched to nondifferential sample-response requirements (Experiment 1B). Experiment 2 showed that differential sample behaviors also generated higher levels of accuracy on delayed MTS when correlated with choice, and that accuracy in this condition did not differ as a function of whether the samples were hues or lines. Sample dimension did affect memory performance, on the other hand, in the uncorrelated condition. In Experiment 3, reversing differential sample-response requirements for one pair of samples substantially reduced matching accuracy in the correlated group but had almost no effect in the uncorrelated group. These findings demonstrate that differential sample behaviors directly control pigeons' matching performances and also overshadow conditional stimulus control by the samples when these behaviors are predictive of correct choice. The facilitation in matching produced by differential sample behaviors apparently arises from the additional cue these behaviors provide, not because they enhance sample discriminability.     

Simple predictions fueled by capacity limitations: when are they successful?

Counterintuitively, Y. Kareev, I. Lieberman, and M. Lev (1997) found that a lower short-term memory capacity benefits performance on a correlation detection task. They assumed that people with low short-term memory capacity (low spans) perceived the correlations as more extreme because they relied on smaller samples, which are known to exaggerate correlations. The authors consider, as an alternative hypothesis, that low spans do not perceive exaggerated correlations but make simpler predictions. Modeling both hypotheses in ACT-R demonstrates that simpler predictions impair performance if the environment changes, whereas a more exaggerated perception of correlation is advantageous to detect a change. Congruent with differences in the way participants make predictions, 2 experiments revealed a low capacity advantage before the environment changes but a high capacity advantage afterward, although this pattern of results surprisingly only existed for men.     

Statistical judgments are influenced by the implied likelihood that samples represent the same population

When sample information is combined, it is generally considered normative to weight information based on larger samples more heavily than information based on smaller samples. However, if samples appear likely to have been drawn from different subpopulations, it is reasonable to combine estimates of these subpopulation means (typically, the sample means) without weighting these estimates by sample size. This study investigated whether laypeople are influenced by the likelihood of samples coming from the same population when determining how to combine information. In two experiments we show that (1) implied binomial variability affected participants’ judgments of the likelihood that a sample was drawn from a given population, (2) participants' judgments were more affected by sample size when samples were implied to be drawn randomly from a general population, compared to when they were implied to be drawn from different subpopulations, and (3) people higher in numeracy gave more normative responses. We conclude that when determining how to weight and combine samples, laypeople use not only the provided data, but also information about likelihood and sampling processes that these data imply.     

Sample size and the detection of correlation--a signal detection account: comment on Kareev (2000) and Juslin and Olsson (2005)

Simulations examined the hypothesis that small samples can provide better grounds for inferring the existence of a population correlation, p, than can large samples. Samples of 5, 7, 10, 15, or 30 data pairs were drawn either from a population with p=0 or from one with p>0. When decision accuracy was assessed independently for each level of the decision criterion, there was a criterion-specific small-sample advantage. For liberal criteria, accuracy was greater for large than for small samples, but for conservative criteria, the opposite result occurred. There was no small-sample advantage when accuracy was measured as the area under a receiver operating characteristic curve or as the posterior probability of a hit. The results show that small-sample advantages can occur, but under limited conditions.     

Sex Differences in Risk Taking: Repeated Sessions on a Computer-Simulated Task

Two experiments were conducted to investigate sex differences in risktaking behavior on a computer-generated and controlled task. Male and female subjects faced a video display of simulated mine fields with varying numbers and patterns of dots representing mines in the fields. In Experiment 1, they estimated the probability that a tank might successfully cross 100 mine fields when starting from an unknown point below each field. This was followed by the risk-taking task in which they decided whether to send a tank across each of the fields. The participants were tested on the risk-taking task once in Experiment 1 and over four separate sessions in Experiment 2. Scores, based on decision outcomes, and decision latencies were recorded. No significant sex differences were found in the participants' ability to estimate probabilities or in their total scores for decisions made on the risk-taking task. In low probability-of-success situations, women initially took greater risks than men but took longer to make their decisions; in all subsequent sessions, this pattern was reversed, with men taking greater risks and a longer time to make decisions. The findings support the growing body of evidence that men are more inclined to take risks than women in a variety of situations, indicate the importance of obtaining data over repeated sessions when investigating sex differences in risk taking, and demonstrate that computer-simulated tasks can provide a valid means for laboratory studies of sex differences in risk taking.     

The development of emergent differential sample behavior in pigeons

Three experiments attempted to replicate Manabe, Kawashima, and Staddon's (1995) finding of emergent differential sample behavior in budgerigars that has been interpreted as evidence of functional equivalence class formation. In Experiments 1 and 2, pigeons initially learned two-sample/ two-alternative matching to sample in which comparison presentation was contingent on pecking one sample on a differential-reinforcement-of-low-rate (DRL) schedule and the other on a fixed-ratio (FR) schedule. Later, two new samples were added to the task. Comparison presentation on these trials occurred after the first sample peck following a predetermined interval (Experiment 1) or after completion of either the DRL or FR requirement, whichever occurred first (Experiment 2). Experiment 1 found no evidence for emergent spaced versus rapid responding to the new samples as they established conditional control over the familiar choices. By contrast, differential responding did emerge for some pigeons in Experiment 2, with responding to each new sample coinciding with the pattern explicitly conditioned to the original sample occasioning the same comparison choice. This emergent effect, however, disappeared for most pigeons with continued training. Experiment 3 systematically replicated Experiment 2 using differential peck location as the sample behavior. Differential location pecking emerged to the new samples for most pigeons and remained intact throughout training. Our findings demonstrate a viable pigeon analogue to the budgerigar emergent calling paradigm and are discussed in terms of equivalence- and non-equivalence-based processes.     

Dyadic judgments based on conflicting samples: The failure to ignore invalid input

The present research demonstrates a so far unrecognized impediment of group performance, metacognitive myopia (Fiedler, 2012). Judges and decision-makers follow the given samples of information uncritically and neglect the metacognitive assessment of the samples' validity. Applying this notion to dyadic judgments, we instructed dyads to jointly estimate conditional probabilities p (Win|A) and p (Win|B) of Lotteries A and B. One person per dyad experienced a valid sample (winning rates conditional on lotteries). The other person experienced an invalid, reverse sample (lotteries conditional on winning). Whereas valid samples provide unbiased estimates of lotteries' winning probabilities, invalid samples can greatly misrepresent the association of winning and lotteries (depending on lottery base rates). Across three experiments, metacognitive myopia—both at the individual and at the dyadic group level—prevented participants from discriminating valid and invalid samples. Group judgments were biased toward erroneous implications of invalid samples, reflecting an equality bias among unequal group members.     

Determining the Temporal Limits of a Visual Sample for Visual Regulation

The author examined the minimum amount of time needed for vision to increase aiming accuracy and decrease movement duration. Participants selected when they would receive a visual sample during aiming movements by pressing a switch held with the left hand. The sample was one of the following durations: 40 ms, 30 ms, 20 ms, 10 ms, or 0 ms (no vision). Decreased accuracy in the no-vision condition compared to the vision conditions was observed when the duration of the impending sample was unknown (Experiment 1). Samples 40 ms in duration were sufficient to decrease endpoint variability when the duration of the sample was known before the movement (Experiment 2). These results indicate that short visual samples can be used to decrease movement time and increase accuracy and that knowledge of the impending visual context can impact the individual's subsequent behavior.     

DOES EFFORT PLAY A ROLE IN THE EFFECT OF RESPONSE REQUIREMENTS ON DELAYED MATCHING TO SAMPLE?

The possible role of "effort" in the accuracy of pigeons' performance on a delayed matching-to-sample procedure was investigated by examining the effects of response requirements that accompanied a trial-initiating stimulus and that accompanied a sample stimulus. In the first experiment, the effect of varying the size of a fixed-ratio requirement for responses during an initiating stimulus was compared to that of varying a similar requirement for responses during the sample stimulus. Accuracy increased reliably with increases in the ratio scheduled during the sample stimulus, but was not significantly affected by increases in the ratio scheduled on the key during the initiating stimulus. In another phase of Experiment 1, sample duration was held constant while the ratio requirement was varied during the initiating stimulus. Again, accuracy of matching to sample was not significantly affected by the size of the ratio scheduled during the initiating stimulus. Experiment 2 provided a systematic replication of these results in another group of pigeons and included a more detailed analysis of responding. These results support the view that increases in sample-response requirement facilitate accuracy of delayed matching by increasing the durations of exposure to the sample stimuli, and do not support a role of effort in the sample-response effect. In Experiment 3, the facilitative effect of responses on the sample but not of those on the initiating stimulus was replicated using a simultaneous matching-to-sample procedure. This finding provides further evidence against an interpretation of response-requirement effects that appeals to effort; the finding also suggests that sample exposure might affect initial discrimination of the sample rather than remembering the sample.     

Effects of training condition on the contribution of specific items to relational processing in baboons (Papio papio)

Relational processing involves learning about the relationship between or among stimuli, transcending the individual stimuli, so that abstract knowledge generalizable to novel situations is acquired. Relational processing has been studied in animals as well as in humans, but little attention has been paid to the contribution of specific items to relational thinking or to the factors that may affect that contribution. This study assessed the intertwined effects of item and relational processing in nonhuman primates. Using a procedure that entailed both expanding and contracting sets of pictorial items, we trained 13 baboons on a two-alternative forced-choice task, in which they had to distinguish horizontal from vertical relational patterns. In Experiment 1, monkeys engaged in item-based processing with a small training set size, and they progressively engaged in relation-based processing as training set size was increased. However, in Experiment 2, overtraining with a small stimulus set promoted the processing of item-based information. These findings underscore similarities in how humans and nonhuman primates process higher-order stimulus relations.     

Monkeys match and tally quantities across senses

We report here that monkeys can actively match the number of sounds they hear to the number of shapes they see and present the first evidence that monkeys sum over sounds and sights. In Experiment 1, two monkeys were trained to choose a simultaneous array of 1-9 squares that numerically matched a sample sequence of shapes or sounds. Monkeys numerically matched across (audio-visual) and within (visual-visual) modalities with equal accuracy and transferred to novel numerical values. In Experiment 2, monkeys presented with sample sequences of randomly ordered shapes or tones were able to choose an array of 2-9 squares that was the numerical sum of the shapes and sounds in the sample sequence. In both experiments, accuracy and reaction time depended on the ratio between the correct numerical match and incorrect choice. These findings suggest monkeys and humans share an abstract numerical code that can be divorced from the modality in which stimuli are first experienced.     

Does decision quality (always) increase with the size of information samples? Some vicissitudes in applying the law of large numbers

Adaptive decision making requires that contingencies between decision options and their relative assets be assessed accurately and quickly. The present research addresses the challenging notion that contingencies may be more visible from small than from large samples of observations. An algorithmic account for such a seemingly paradoxical effect is offered within a satisficing-choice framework. Accordingly, a choice is only made when the sample contingency describing the relative evaluation of the 2 options exceeds a critical threshold. Small samples, because of the high dispersion of their sampling distribution, facilitate above-threshold contingencies. Across a broad range of parameters, the resulting small-sample advantage in terms of hits is stronger than their disadvantage in false alarms. Computer simulations and experiments support the model predictions. The relative advantage of small samples is most apparent when information loss is low, when the threshold is high relative to the ecological contingency, and when the sampling process is self-truncated.