Nonresponse in Employee Attitude Surveys: A Group-Level Analysis

Purpose

Given the common practice of using employee attitude surveys as a group-level intervention, this study used a group-level approach to examine the relationship between group satisfaction and group nonresponse.

Design/Methodology/Approach

Samples from four large organizations enabled job satisfaction scores to be aggregated to the work group level and correlated with group-level response rates. Additional regression analysis was conducted to control for a number of confounding variables at the group level.

Findings

Aggregate job satisfaction showed significant associations with group-level response rates across each of the samples examined. Work groups with higher aggregate job satisfaction had significantly higher response rates. Regression analyses showed that, in addition to job satisfaction, work group size, heterogeneity in tenure, and heterogeneity in gender composition all had significant effects on response rates.

Implications

Social influence processes may operate at the group level to increase homogeneity of job-relevant attitudes and similarity in survey response behavior. Future research should be designed to investigate the effects of group-level variables on nonresponse.

Originality/Value

The current study adds to the literature by demonstrating that work group variables may play an important role in explaining nonresponse in employee attitude surveys. Because the processes underlying survey response are likely to be different at different levels of analysis, the investigation of nonresponse as a group-level phenomenon creates new opportunities for research and practice.     

Survey return rates as a function of priority versus first-class mailing

Prior research indicates survey procedures that signal significance and individualized mailings have higher response rates. Thus, it was hypothesized that surveys delivered via Priority mail would result in higher return rates than surveys delivered via First-Class. 260 surveys were sent to individuals randomly selected from lists of licensed physical and behavioral healthcare providers in Alaska and New Mexico. Half of the selected individuals were assigned randomly to receive mailings using Priority mail, the other half received First-Class mailings. Return rate was 39% for First-Class and 35% for Priority. Z tests of proportion indicated no statistically significant differences between methods. Given increased costs with no resultant increase in response rate, sending surveys to potential participants via Priority mail does not appear warranted.     

Response rate,speed, and completeness: A comparison of Internet-based and mail surveys

Because of their speed and accessibility, the use of on-line research tools has grown considerably in recent years. The present study compared two ways of delivering surveys: Internet-based and mail delivery methods. Although Internet-basedand mail surveys achieved a similar response rate, Internet-based surveys may be more effective than mail surveys in a setting such as when the target population has both e-mail and Internet access.     

A Comparison of a Subjective and Statistical Method for Establishing Score Comparability in an Organizational Culture Survey

Purpose

The purpose of this study was to compare the results of a subjective and a statistical method of detecting non-comparable items in 14 language-translated forms of a survey measuring organizational culture.

Design/Methodology/Approach

Data were obtained from a large multinational organization using a 60-item organizational culture survey. Each of 14 language-translated forms were administered to members of their respective language groups and compared to the original English (United States) form. Subjective reviews were conducted using fluent bilingual organizational members whom flagged items they did not believe were comparable. Statistical analyses using an item response theory (IRT) approach were used to detect problematic items from 14 samples with sizes ranging from 304 to 3,014. Detection patterns from these two approaches were compared.

Findings

The subjective approach identified far less items as problematic and did not agree with the statistical approach.

Implications

Our results suggest that the subjective approach as a pre-screening adaptation procedure has little added value over a careful translation/back-translation procedure.

Originality/Value

The use of language-adapted organizational surveys has become increasingly important to multinational organizations. In examining scores from such surveys the establishment of score comparability is essential. IRT analyses are often used to determine whether scores are comparable across language translations. It is also common for subjective reviews of item content to be utilized prior to statistical techniques to determine if the translated items are comparable to the original form. No research known to authors has compared modern statistical and subjective approaches to addressing these issues.     

Demographic Question Placement: Effect on Item Response Rates and Means of a Veterans Health Administration Survey

Purpose

This study examined whether demographic question placement affects demographic and non-demographic question completion rates, non-demographic item means, and blank questionnaire rates using a web-based survey of Veterans Health Administration employees.

Methodology

Data were taken from the 2010 Voice of the Veterans Administration Survey (VoVA), a voluntary, confidential, web-based survey offered to all VA employees. Participants were given two versions of the questionnaires. One version had demographic questions placed at the beginning and the other version had demographic questions placed at the end of the questionnaire.

Findings

Results indicated that placing demographic questions at the beginning of a questionnaire increased item response rate for demographic items without affecting the item response rate for non-demographic items or the average of item mean scores.

Implications

In addition to validity issues, a goal for surveyors is to maximize response rates and to minimize the number of missing responses. It is therefore important to determine which questionnaire characteristics affect these values. Results of this study suggest demographic placement is an important factor.

Originality/Value

There are various opinions about the most advantageous location of demographic questions in questionnaires; however, the issue has rarely been examined empirically. This study uses an experimental design and a large sample size to examine the effects of demographic placement on survey response characteristics.     

Effects of question format and collection mode on the accuracy of retrospective surveys of health risk behavior: a comparison with daily sexual activity diaries.

OBJECTIVE: To use prospective diaries to assess the effects of alternative survey question formats and data collection modes on the accuracy of retrospective self-reports of sexual behavior for the same time period. DESIGN: Over a 3-month period, 493 adults completed and returned by mail daily diaries of their sexual activities. Participants then returned to complete a retrospective survey of their sexual activities during the same 3-month time period. Participants were randomized to 1 of 6 retrospective experimental survey conditions that represented the combinations of 2 question formats (in aggregate and by partner) and 3 collection modes (paper questionnaire, computer-assisted self-interview [CASI], and audio-enhanced CASI). MAIN OUTCOME MEASURES: Frequencies of sexual activities and condom use reported in surveys were compared with what participants had recorded in their prospective daily diaries. RESULTS: Relative to activity recorded in daily diaries, participants more often underreported than overreported (61% vs. 28%, p < .05) sexual behavior in retrospective surveys. Greater survey-to-diary consistency in reported behavior was achieved with by-partner question format (p < .032) and CASI collection modes (p < .05) used in surveys. However, median rates of error in survey-to-diary comparisons of reported behavior were as high as 40%. CONCLUSION: Even after refinements in survey question format and use of CASI techniques, there remained substantial error in participants' retrospective reports of sexual practices. These error rates may be comparable to, or even larger than, the expected effect size of some intervention studies. Further research is needed to improve the accuracy of sexual behavior reports.     

Effects of Personalized Versus Generic Implementation of an Intra-Organizational Online Survey on Psychological Anonymity and Response Behavior: A Field Experiment

Purpose

In the present study, we sought to investigate the effects of online survey implementation strategies on perceived anonymity and employee response behavior in organizational surveys.

Design/Methodology/Approach

A field experiment was conducted to compare two commonly used online survey implementation strategies (N = 815). One group of employees received a personalized invitation to the survey and a log-in password, while the other group received a general invitation and did not have to provide a password.

Findings

The results showed that the applied implementation strategies had no substantial effects on perceived anonymity. Moreover, there were no significant effects on nonresponse or the responses of survey participants to closed-ended and open-ended survey questions.

Implications

The present study supposes that online surveys are not a uniform phenomenon and that differences in the implementation of online surveys need to be considered. However, the findings indicate that the use of specified-personalized implementation strategies does not necessarily lead to a substantial decrease in perceived anonymity or automatically result in reduced data quality. Thus, in many cases, the investigated online survey implementation strategies are unlikely to cause serious reductions in perceptions of anonymity and quality of responses to organizational online surveys.

Originality/Value

In spite of the frequent use of online surveys in organizations, little is known about the consequences of online implementation strategies for perceptions of anonymity and response behavior. This study is one of the few empirical examinations of the psychological consequences of different online implementation strategies frequently used in organizational surveying.     

Differences in responses to Web and paper surveys among school professionals

This experiment investigated the effects of survey administration mode (Web vs. paper and pencil) and survey length (short or long) on the responses of a large sample of primary and secondary school professionals in the U.S. The 812 participants in this study were part of an initial random sample of 1,000 individuals representing 5.81% of the membership of a national professional organization. The participants were randomly assigned to each of the four treatment conditions. Results indicated (1) substantially lower response rates for Web surveys than for same-length paper surveys; (2) a higher response rate for short surveys than for long surveys with paper, but not Web, questionnaires; and (3) a younger age for Web respondents, as compared with their paper counterparts. In light of prior research, we suggest that paper-and-pencil methods be used for surveying professionals in primary and secondary school settings.     

Effect of respondent code location on survey response rate

Survey response rates were examined for differences based on the location of the respondent code, a numerical code placed on the survey versus a numerical code placed on the return envelope. A four-page survey on criteria and procedures used when hiring health education faculty members was mailed to program coordinators of university health education programs (N = 223). Half of the coordinators (n = 112) received coded surveys (surveys that had a written overt three-digit code at the top right corner of the first page of the survey). The other half of the coordinators (n = 111) received coded return envelopes (return envelopes with a six-digit code typed directly under the return mailing address). A total of 132 coordinators returned completed surveys (62%). Response rate did not differ significantly based on placement of respondent code. Survey researchers are thus recommended to devote their efforts to other inducement strategies to increase response rates.     

Response Rates in Organizational Science, 1995–2008: A Meta-analytic Review and Guidelines for Survey Researchers

Purpose  

This study expands upon existing knowledge of response rates by conducting a large-scale quantitative review of published response rates. This allowed a fine-grained comparison of response rates across respondent groups. Other unique features of this study are the analysis of response enhancing techniques across respondent groups and response rate trends over time. In order to aid researchers in designing surveys, we provide expected response rate percentiles for different survey modalities.     

German Abstract

The magnitude of the cultivation effect for perceptual estimates of social reality has been shown to be affected by a number of contextual factors such as source priming and motivation to process information during judgment construction, and these contextual factors have been linked to the use of heuristic processing strategies when constructing judgments of frequency and probability ( L. J. Shrum, in press ). An experiment that manipulated data collection method explored the implications of these findings. A random sample of general population respondents were randomly assigned to either telephone or mail survey conditions. Because telephone surveys generally result in greater heuristic processing than mail surveys, telephone surveys were expected to produce larger cultivation effects than mail surveys. Results showed that not only were the magnitude of the estimates of affluence, crime, vice, marital discord, and occupations generally greater in the telephone than in the mail survey but the correlation of the estimates with amount of viewing was also greater in the telephone than in the mail survey. The implications for measuring the cultivation effect are discussed.     

Survey return rates for multiple-authored versus single-authored covering letters

Differences in response rates to a three-wave mail survey were assessed when the covering letters were signed by three versus one researcher. 375 surveys including covering letters signed by three researchers and 375 surveys including covering letters signed by one researcher were sent. A follow-up mailing was used to increase the response rate. 199 surveys were returned (53.5%) in which the covering letter had three researchers, and 218 when the covering letter had one researcher (58.7%), not a significantly different distribution.     

How Does More Attention to Subjective Well-Being Affect Subjective Well-Being?

There is an ongoing debate as to whether pursuing happiness is beneficial for people’s subjective well-being (SWB). To address this question, we tested whether attention to SWB – measured by participation in SWB surveys – is related to experienced SWB in two longitudinal studies. The initial study was conducted from November to December 2013 (N?=?129), and the replication study, three years later from November to December 2016 (N?=?120). The studies include two groups: one group (the control group) answered three SWB surveys over 4 weeks, and the other group (the experimental group) followed the same procedure but additionally tracked their SWB in detail using the experience sampling method four times a day and the day reconstruction method once a day using a smartphone application for two weeks to heighten their attention to their SWB. Both studies show higher SWB scores at later measurements compared to the first ones.     

Similarity and Helping Behavior on the Web: The Impact of the Convergence of Surnames Between a Solicitor and a Subject in a Request Made by E‐Mail1

Similarity between a solicitor and a subject traditionally enhances helping behavior. An experiment was carried out in a computer‐mediated context. Fifty students received an e‐mail containing a 40‐question survey on their food habits, which required 15 to 20 min of their time to respond. This questionnaire came from a hypothetical student of the university in which the participants were registered. In half of the cases, the surname of the solicitor, which appeared in his or her electronic address, was the same as the surname of the target. Results show that compliance to the request was significantly higher in the same‐sumame condition than in the different‐surname condition. The response delay was significantly shorter in the same‐surname condition than in control condition.     

The Confluence of Technology and Narrative Approaches in Group Work

Abstract

This article illustrates procedures for using e-journals as a creative and adjunctive approach in group work. Incorporating e-mail based journaling as an ancillary form of group interaction allows members to communicate via written channels, and creates new ways for clients to relate in the group. This article outlines how leaders can use narrative theory to construct e-mail based writing exercises in group settings. Borrowing from established narrative techniques, specific interventions are provided for using interactive e-mails in each group stage. Finally, detailed instructions for establishing encrypted and confidential e-mail communications are provided, and implications and limitations for establishing e-mail based communication systems are overviewed.     

Sampling Bias and Other Methodological Threats to the Validity of Health Survey Research

Data from a longitudinal occupational health survey of professional fire fighters were used to explore the potential impact of two types of methodological bias: sample selection and reactivity. No significant differences on demographic variables were observed between the group who first responded after a within-study change in survey administration format (Delayed) and respondents who had completed surveys since the study's inception (Initial). However, statistically significant differences in the study's 26 outcome measures provided some evidence that between-group differences did exist and that an administration format type of response bias was also potentially present. The effect sizes associated with the 37 observed significant differences ranged from small to medium. These results provide a context for a reexamination of standard techniques for the identification and interpretation of survey research biases. Methods are suggested to strengthen tests for selection bias and to minimize the impact of response biases.     

Agreement by design: The effect of visual harmony on responses to surveys

The present study is one of the first to empirically examine how the visual harmony of a questionnaire can lead to measurement bias. Researchers often employ questionnaires with Likert scales to measure constructs. In this note, we examine how the design of the survey instrument, specifically, its visual harmony, can impair measurement accuracy. Two studies investigate effects of visual harmony in surveys on responses to Likert scales using paper and pencil surveys. Applying an established customer relationship management model, Study 1 employs a survey of female visitors to a grocery store (n = 115). Switching to a product and brand innovation context, Study 2 employs a survey of male and female members of a consumer panel (n = 180) to examine responses to a new e‐scooter. Across studies, results indicate that assessing important consumer response constructs through visually more harmonious surveys can lead to more positive response patterns, lower scale reliability, and questionable validity, especially with females. Although these effects do not occur uniformly across measures and samples, they occur regardless of consumers' past experience with completing questionnaires, their familiarity with questionnaire design, and the naturalness and elaborateness of the visual design. Relating specific elements (e.g., text boxes, type font, shapes, and images) and relational properties of design (e.g., balance, symmetry, and coherence) to consumers' overall perception of harmony aids marketers and researchers in achieving intermediate levels to obtain realistic, reliable, and valid results.     

Sampling the Reluctant Participant: A Random‐Sample Response‐Rate Study of Men and Sexual Coercion1

Past research has documented the difficulty associated with mail surveys in general, and with the rates of male participation, especially if the topic involves sensitive materials. Our study varied incentive value and whether participants received a letter of introduction. We adopted most of the stringent survey‐design characteristics recommended by Dillman (1991). The results suggest that a letter of introduction significantly increased the response rate. Additionally, a smaller monetary incentive did as well as did a larger one. Despite a maximum participation rate of only 22%, the sample was successful in representing men across the age span. Some important differences from men in the general population were found on demographic and attitudinal variables, however, the generalizability was still improved well beyond the college male.     

Pornography Consumption and Satisfaction: A Meta‐Analysis

A classic question in the communication literature is whether pornography consumption affects consumers' satisfaction. The present paper represents the first attempt to address this question via meta‐analysis. Fifty studies collectively including more than 50,000 participants from 10 countries were located across the interpersonal domains of sexual and relational satisfaction and the intrapersonal domains of body and self satisfaction. Pornography consumption was not related to the intrapersonal satisfaction outcomes that were studied. However, pornography consumption was associated with lower interpersonal satisfaction outcomes in cross‐sectional surveys, longitudinal surveys, and experiments. Associations between pornography consumption and reduced interpersonal satisfaction outcomes were not moderated by their year of release or their publication status. But analyses by sex indicted significant results for men only.     

Ethicists’ and Nonethicists’ Responsiveness to Student E‐mails: Relationships Among Expressed Normative Attitude,Self‐Described Behavior,and Empirically Observed Behavior

Do professional ethicists behave any morally better than other professors do? Do they show any greater consistency between their normative attitudes and their behavior? In response to a survey question, a large majority of professors (83 percent of ethicists, 83 percent of nonethicist philosophers, and 85 percent of nonphilosophers) expressed the view that “not consistently responding to student e‐mails” is morally bad. A similarly large majority of professors claimed to respond to at least 95 percent of student e‐mails. These professors, and others, were sent three e‐mails designed to look like queries from students. Ethicists’ e‐mail response rates were not significantly different from the other two groups’. Expressed normative view correlated with self‐estimated rate of e‐mail responsiveness, especially among the ethicists. Empirically measured e‐mail responsiveness, however, was at best weakly correlated with self‐estimated e‐mail responsiveness; and professors’ expressed normative attitude was not significantly correlated with empirically measured e‐mail responsiveness for any of the three groups.