Sampling Bias and Other Methodological Threats to the Validity of Health Survey Research

Data from a longitudinal occupational health survey of professional fire fighters were used to explore the potential impact of two types of methodological bias: sample selection and reactivity. No significant differences on demographic variables were observed between the group who first responded after a within-study change in survey administration format (Delayed) and respondents who had completed surveys since the study's inception (Initial). However, statistically significant differences in the study's 26 outcome measures provided some evidence that between-group differences did exist and that an administration format type of response bias was also potentially present. The effect sizes associated with the 37 observed significant differences ranged from small to medium. These results provide a context for a reexamination of standard techniques for the identification and interpretation of survey research biases. Methods are suggested to strengthen tests for selection bias and to minimize the impact of response biases.     

Can't stop believing: inhibitory control and resistance to misleading testimony

Why are some young children consistently willing to believe what they are told even when it conflicts with first‐hand experience? In this study, we investigated the possibility that this deference reflects an inability to inhibit a prepotent response. Over the course of several trials, 2.5‐ to 3.5‐year‐olds (N = 58) heard an adult contradict their report of a simple event they had both witnessed, and children were asked to resolve this discrepancy. Those who repeatedly deferred to the adult's misleading testimony had more difficulty on an inhibitory control task involving spatial conflict than those who responded more skeptically. These results suggest that responding skeptically to testimony that conflicts with first‐hand experience may be challenging for some young children because it requires inhibiting a normally appropriate bias to believe testimony.     

Group cognitive-behavior therapy for patients failing to respond to pharmacotherapy for panic disorder: a clinical case series.

The last decade has brought exciting advances to the treatment of panic disorder and agoraphobia, and a variety of cognitive-behavioral and pharmacologic treatment strategies offer clear benefit to patients. Nonetheless, treatment nonresponse continues to be a chronic problem, and additional strategies are needed to aid patients who do not respond fully to initial interventions. In the present study, we use 'services' research to document the clinical response of pharmacotherapy nonresponders to a standard program of brief, group cognitive-behavioral therapy. Patients responded well, regardless of whether they had received a full, adequate trial of pharmacotherapy. In addition to its application as an initial treatment for panic disorder, routine application of cognitive-behavioral therapy in pharmacologic treatment algorithms is encouraged, with attention to referral of pharmacotherapy nonresponders to cognitive-behavioral therapy.     

Maternal sensory sensitivity and response bias in detecting change in infant facial expressions: maternal self-efficacy and infant gender labeling

This study examined the role of maternal cognitive sets regarding infant gender on maternal response; participants were 69 mothers, each with a 6-month-old infant. Signal detection methodology was used to assess maternal sensory sensitivity and response bias to subtle changes in positive and negative infant facial expressions as a function of maternal self-efficacy, operationalized by low, moderate, and high illusory control, and maternal perceptions of infant gender, experimental manipulated through labeling. For the positive expression, mothers who received the female manipulation exhibited greater sensory sensitivity and responded with a less conservative response bias compared with mothers who received the male manipulation. For the negative expression, only mothers with high illusory control were differentially sensitive to the manipulation; those who received the male manipulation exhibited less sensory sensitivity than those who received the female manipulation. Results are discussed in the context of the gendered socialization of emotion.     

Survey Nonrespondents as Bad Soldiers: Examining the relationship between organizational citizenship and survey response behavior

The present paper examines whether organizational survey response behavior relates to organizational citizenship behavior (OCB). In Study 1, we examined the relationship between OCB dimensions and survey completion intentions in a field setting (n=228). In Study 2, we compared respondents and different groups of nonrespondents on OCB dimensions using an experimental design in a university setting (n=622). Survey response and OCBs were found to be related. OCB research may suffer from nonresponse bias, with organizational surveys failing to capture the opinions of those least likely to engage in OCBs.     

Evidence for episodic retrieval of inadequate prime responses in auditory negative priming

Four experiments are reported in which the mechanisms underlying auditory negative priming were investigated. In Experiments 1A and 1B, preprime-prime intervals and prime-probe intervals were manipulated. The ratio between the 2 intervals determined the size of the negative priming effect. Results are compatible with the episodic retrieval account, according to which the retrieval of inappropriate response information associated with the previous distractor slows down responding when that stimulus becomes the target. Experiment 2 tested a variant of this account, according to which the retrieval of the prime response rather than the retrieval of nonresponse information interferes with responding. Consistent with this variant, participants erroneously responded with the prime response more frequently in the ignored repetition condition than in the control condition. Experiment 3 replicated this finding and generalized it to the visual modality. The authors conclude that the retrieval of the inappropriate prime response is a determinant of the negative priming phenomenon.     

The paired associate learning task: Is it an externally valid instrument for assessing methylphenidate response in children with attention deficit disorder?

The Paired Associate Learning Task (PAL-T) is a laboratory-based instrument commonly used for assessing psychostimulant response (and specifically cognitive nonresponse or toxicity) in children with Attention Deficit Disorder/Hyperactivity (ADDH). The present study examined the PAL-T's sensitivity in detecting overall and between-dose methylphenidate (MPH) effects, its operating characteristics, and its relationship with academic performance in 36 children with ADDH who underwent a placebo controlled trial of MPH at four dose levels. Children's PAL-T performance under active medication resembled but was uncorrelated with direct observations of their academic performance in the classroom. Negative predictive power (NPP) rates indicated that cognitive nonresponse to MPH did not portend a poor response in children's academic efficiency under identical dose conditions in the classroom. Clinical implications are discussed.The authors are grateful to the undergraduate and graduate student members of the Children's Learning Clinic who assisted with the conduct of this study.     

Profiling active and passive nonrespondents to an organizational survey

In this field study (N = 405) population profiling was introduced to examine general and specific classes of nonresponse (active vs. passive) to a satisfaction survey. The active nonrespondent group (i.e., purposeful nonresponders) was relatively small (approximately 15%). Active nonrespondents, in comparison with respondents, were less satisfied with the entity sponsoring the survey and were less conscientious. Passive nonrespondents (e.g., forgot), who represented the majority of nonrespondents, were attitudinally similar to respondents but differed with regard to personality. Nonresponse bias does not appear to be a substantive concern for satisfaction type variables--the typical core of an organizational survey. If the survey concerns topics strongly related to Conscientiousness and Agreeableness, the respondent sample may not be representative of the population.     

A novel assessment of behaviors associated with body dissatisfaction and disordered eating

The purpose of the present study was to compare response rates to questions assessing behaviors associated with body dissatisfaction and disordered eating on a standard self-report questionnaire that was nominally anonymous to an unmatched count questionnaire that allowed for true response anonymity. Five hundred sixty-seven undergraduate students were asked about disordered eating and body image-related behaviors using one of two response formats; either a conventional true-false questionnaire or an unmatched count questionnaire that did not require participants to directly respond to any sensitive questions. Both men and women had significantly different rates of endorsement between the two methods of assessment on the majority of the questions, suggesting that degree of anonymity and format of response may affect endorsements of these potentially sensitive topics. These results contribute to the existing literature highlighting the necessity of understanding self-report bias in determining accurate rates of sensitive behaviors.     

Using Response Times for Joint Modeling of Response and Omission Behavior

Abstract

For adequate modeling of missing responses, a thorough understanding of the nonresponse mechanisms is vital. As a large number of major testing programs are in the process or already have been moving to computer-based assessment, a rich body of additional data on examinee behavior becomes easily accessible. These additional data may contain valuable information on the processes associated with nonresponse. Bringing together research on item omissions with approaches for modeling response time data, we propose a framework for simultaneously modeling response behavior and omission behavior utilizing timing information for both. As such, the proposed model allows (a) to gain a deeper understanding of response and nonresponse behavior in general and, in particular, of the processes underlying item omissions in LSAs, (b) to model the processes determining the time examinees require to generate a response or to omit an item, and (c) to account for nonignorable item omissions. Parameter recovery of the proposed model is studied within a simulation study. An illustration of the model by means of an application to real data is provided.     

Who responds to surveys? Assessing the effects of nonresponse in cross-sectional dyadic research

The purpose of this study was to adapt Goodman and Blum's procedure for assessing the effects of nonrandom sampling in longitudinal research to a cross-sectional study of dyads. To demonstrate the impact of survey nonresponse on means, variances, and relationships between variables, data were collected from a sample of 311 recruiter-applicant pairs. Of the variables included in this investigation, only one (recruiter competence) significantly predicted the likelihood of responding. In addition, there were differences in the means and variances of respondents and nonrespondents on some of the variables included in this study. However, there were no differences in the relationships between independent and dependent variables for the respondent group compared with the entire group of respondents and nonrespondents. In sum, the results suggest that nonresponse does not bias the relationships between variables.     

Probabilistic inferences,discrimination, and stimulus interference in comparative judgement

Comparative judgements of successive stimuli involve constant errors. Here, two theories of these errors are considered. For the wave theory of discrimination, constant errors directly originate from discrimination operations. For the inferential theory, the same errors are due to probabilistic inferences drawn after discrimination. Previous studies suggest that the second of two successively compared stimuli has more influence on the comparative response. The results of the present experiment, in which successive lines as comparison stimuli were used, confirm this suggestion. The wave theory does not explain these results, while the inferential theory explains them in terms of probabilistic inferences. Analysis of response frequencies and response times indicates that there was a response bias in favour of the longer response. Interfering lines appearing before the first comparison line significantly affected response frequency, response time, and discriminability, whereas interpolated lines significantly affected only response frequency and response time. The inferential theory does not explain these effects on response time, while the wave theory explains them as due to modifications of response bias. These results suggest the possibility that the two theories describe independent sources of constant errors.     

Maximizing the response rate in surveys may be a mistake

A random postal sample of 400 Australians was carried out with the assistance of two conventional response-rate maximization techniques: a preliminary letter and a second mailing of the questionnaire aimed at non-respondents to the first questionnaire. The response-rate enhancement techniques appeared to have doubled the response rate from a usual 25% for a one-wave mailing to 47% on the present occasion. The present data, however, showed severe effects of acquiescent response bias (‘saying yes to anything’), thus severely damaging the meaningfulness of the results. It is concluded that the use of response-rate enhancement techniques is not only unnecessary but is also in fact counter-productive.     

Les représentations de l’entretien structuré chez les recruteurs

We studied recruiters’ representations of the structured interview. In Study 1, 90 recruiters were interviewed. They define “structure” mainly as a schema or list of questions. The terms “structured” and “unstructured” also have different connotations. This can create a social desirability bias in surveys on selection practices, causing an overestimation of the real prevalence of structured interviewing. In Study 2, 59 recruiters responded to a questionnaire. Attitudes towards structured interviews are positive, but representations diverge from scientific definitions, especially regarding unstructured interviews. Recruiters’ representations allow flexibility in practice while maintaining a positive self-image.     

Sex differences: A study of the ear of the beholder

Male and female adult Ss, while performing an unrelated task, heard a single infant awakening. The infant was labeled a girl or a boy. The dependent variable was how quickly Ss responded to the crying infant. Females responded more quickly to the girl than to the boy. Males responded equally slowly to boy and girl. A questionnaire on sex differences in fragility showed that females and males tend to disagree with the proposition that girls are more fragile than boys. Thus, the females' faster response to a crying girl cannot be accounted for by a belief in sex-differential fragility. Correlations between fragility beliefs and response times to an infant indicate that sex-differential belief-behavior relationships are not the same for males and females.     

The good,the bad and the mad: Response bias in self-report measures

This study set out to examine the susceptibility of five extensively used, self-report measures to response set bias. Subjects were requested either to fake good, (give a good impression), fake bad, (give a bad impression), fake mad, (give an impression of mental instability) or respond honestly. Subjects who faked good had significantly higher Extraversion, Lie and Social Desirability scores but lowest Neuroticism, Psychoticism and Social Anxiety scores. Subjects who faked bad had significantly lower Extraversion and higher Psychoticism and Social Anxiety scores. Fake mad subjects scored higher on Self-Monitoring and Locus of Control. Four of the eight scales showed significant differences between subjects faking bad and those faking mad. The results are discussed in terms of questionnaire design and respondent's motivation.     

Swine Flu: A Field Study of Self-Serving Biases

Self-serving biases were examined as an explanation for swine flu inoculation behavior. Subjects who had recently had the opportunity to obtain the shot were surveyed by questionnaire. As predicted, most persons held biased beliefs that their own health was superior to that of others. Those getting or intending to get the shot had less health bias, had recently experienced a confidence-shaking event, and felt that the shot might make a difference in their health. The results are seen as having implications both for self-serving bias research and for disease control.     

Object-based attention is mediated by collinearity of targets

Same-object bias occurs when tasks associated with processing a single object are faster than tasks associated with two objects. Over five experiments we assessed whether same-object bias is mediated by the collinearity of the targets. Participants decided whether two targets, presented either within a single object or across two objects, were the same or different. Results showed that same-object bias only occurred when targets appeared on the same straight line within the same object. When targets appeared in the same object but were separated by an angle or corner, within-object facilitation was eliminated or greatly reduced. In a final experiment, response times to two targets that were collinear but on separate objects were responded to faster than were noncollinear targets on the same object. This suggests that collinearity between targets mediates the effect found in this paradigm, at least to a greater extent than colour grouping.     

Object-based attention is mediated by collinearity of targets

Same-object bias occurs when tasks associated with processing a single object are faster than tasks associated with two objects. Over five experiments we assessed whether same-object bias is mediated by the collinearity of the targets. Participants decided whether two targets, presented either within a single object or across two objects, were the same or different. Results showed that same-object bias only occurred when targets appeared on the same straight line within the same object. When targets appeared in the same object but were separated by an angle or corner, within-object facilitation was eliminated or greatly reduced. In a final experiment, response times to two targets that were collinear but on separate objects were responded to faster than were noncollinear targets on the same object. This suggests that collinearity between targets mediates the effect found in this paradigm, at least to a greater extent than colour grouping.