Sampling Bias and Other Methodological Threats to the Validity of Health Survey Research

Data from a longitudinal occupational health survey of professional fire fighters were used to explore the potential impact of two types of methodological bias: sample selection and reactivity. No significant differences on demographic variables were observed between the group who first responded after a within-study change in survey administration format (Delayed) and respondents who had completed surveys since the study's inception (Initial). However, statistically significant differences in the study's 26 outcome measures provided some evidence that between-group differences did exist and that an administration format type of response bias was also potentially present. The effect sizes associated with the 37 observed significant differences ranged from small to medium. These results provide a context for a reexamination of standard techniques for the identification and interpretation of survey research biases. Methods are suggested to strengthen tests for selection bias and to minimize the impact of response biases.