Cognitive burden of survey questions and response times: A psycholinguistic experiment

An important objective in survey question design is to write clear questions that respondents find easy to understand and to answer. This contribution identifies the factors that influence question clarity. Theoretical and empirical evidence from psycholinguistics suggests that specific text features (e.g., low‐frequency words (LFRWs), left‐embedded syntax) cause comprehension difficulties and impose a high cognitive burden on respondents. To examine the effect of seven different text features on question clarity, an online experiment was conducted in which well‐formulated questions were compared to suboptimal counterparts. The cognitive burden of the questions was assessed with response times. Data quality was compared in terms of drop‐out rates and survey satisficing behaviour. The results show that at least six of the text features are relevant for the clarity of a question. We provide a detailed explanation of these text features and advise survey designers to avoid them when crafting questions. Copyright © 2009 John Wiley & Sons, Ltd.     

Timepieces: Components of Survey Question Response Latencies

Political scientists have increasingly found that the time respondents require to answer survey questions is a useful measure of the strength of political attitudes. However, questions remain about the collection, use, and interpretation of response time data— particularly latencies collected during telephone interviews, far removed from the controlled environment of a laboratory setting. This paper uses a theoretical model of question response to decompose survey response time data into three hypothesized elements—components of response time attributable to baseline individual differences, systematic question effects, and the accessibility of the attitudes being targeted. These findings have implications for the study of political attitudes and other mental constructs, as well as practical implications for public opinion survey researchers using response time data.     

Deliberation from Within: Changing One's Mind During an Interview

This article examines whether a public opinion survey can improve the quality of political attitudes. More specifically, we argue that simply positioning a summary attitudinal question after a balanced series of relevant items can increase people's ability to answer in a way that better reflects their underlying interests, values, and predispositions. By manipulating the location of the vote preference question in two separate national election campaign surveys, we find that there are fewer undecided respondents when the question is asked at the end of the survey rather than early on, that some people are changing their mind during the questionnaire, that a larger set of determinants is structuring late‐survey vote choice, and that voting preferences based on the later question are a better predictor of the actual vote. The findings carry important lessons for students of deliberation and of citizen decision making.     

Detecting nonadherence without loss in efficiency: A simple extension of the crosswise model

In surveys concerning sensitive behavior or attitudes, respondents often do not answer truthfully, because of social desirability bias. To elicit more honest responding, the randomized-response (RR) technique aims at increasing perceived and actual anonymity by prompting respondents to answer with a randomly modified and thus uninformative response. In the crosswise model, as a particularly promising variant of the RR, this is achieved by adding a second, nonsensitive question and by prompting respondents to answer both questions jointly. Despite increased privacy protection and empirically higher prevalence estimates of socially undesirable behaviors, evidence also suggests that some respondents might still not adhere to the instructions, in turn leading to questionable results. Herein we propose an extension of the crosswise model (ECWM) that makes it possible to detect several types of response biases with adequate power in realistic sample sizes. Importantly, the ECWM allows for testing the validity of the model’s assumptions without any loss in statistical efficiency. Finally, we provide an empirical example supporting the usefulness of the ECWM.     

Tailored Interviewing: An Application of Item Response Theory for Personality Measurement

Despite recent technological advances in mass survey administration, surveys seeking to measure personality traits still employ procedures identical to the traditional paper-and-pencil scales, in which every respondent is asked exactly the same items, regardless of his or her trait level. We present an empirical application of personality measurement in which the number and sequence of scale items are tailored to the trait level of each respondent and show that the total number of questions asked of each respondent could be reduced substantially, with a measurable and controllable increase in the standard error of measurement.     

Gender and response effects in telephone interviews about gender characteristics

This study investigates the effect of gender on responses to questions about sex role orientations. Literature on gender effects in survey interviews, job interviews and counseling studies, and experiments leads to hypotheses that male interviewers will elicit more response effects than female interviewers, especially from females; that female respondents will show desirability effects more than males; and that respondents, especially female respondents, will disclose liberal orientations more to female than to male interviewers. Findings show that male interviewers do not elicit more response effects, especially among females, and there seem to be desirability effects operating instead of conformity effects, especially for female interviewers. Females, and low-power, highly educated males exhibit more liberal responses to female interviewers. This study demonstrates that in areas where interviewer—respondent question content is related to salient dimensions, response effects are observed and constitute potentially damaging sources of error.     

Sex differences in reporting sensitive behavior: A comparison of interview methods

While telephone surveys generally are considered to be comparable to in-person interviews for the collection of survey data, their utility for the acquisition of sensitive information from respondents remains in question. This paper examines the effect of interview method upon self-reports of a threatening topic—substance use—for male and female college students. It is hypothesized that males may be less likely to admit sensitive behaviors in telephone interviews, compared to face-to-face encounters, due to an interaction of a characteristic of the telephone medium (i.e., the lack of balanced exchange of identifying information between the interviewer and respondent) and aspects of the male sex role, which discourages self-revelation. Findings indicate that differences in reported use of both legal and illegal substances by interview method were greater among males than females. These gender differences in reporting behavior by survey method remained after statistical controls were applied.     

Attitude alignment in marriage and cohabitation: The case of sex‐role attitudes

Alignment of attitudes in personal relationships has not often been examined with survey data. This paper examines changes in sex‐role attitudes using a longitudinal and dyadic survey among 374 recently married and cohabiting persons. The paper tests 2 general hypotheses: (a) respondents will become more egalitarian (traditional) when the partner is initially more egalitarian (traditional) and (b) attitude alignment will be stronger when the relational relevance of the attitude is greater, which is defined as the degree to which the attitude topic has consequences for the internal functioning of the relationship. Both hypotheses are partly supported. The analyses show positive effects of the partner's attitudes on later attitudes of the respondent while controlling for prior respondent attitudes, confirming the notion of attitude alignment. In addition, alignment in sex‐role attitudes appears stronger when conditions in the household are such that a traditional division of labor becomes more problematic. Differences by gender suggest that husbands change more often in the direction of the wife than vice versa.     

Discernment and Denial: Nanotechnology Researchers’ Recognition of Ethical Responsibilities Related to Their Work

To what extent do nanotechnology researchers discern specific work-related ethical responsibilities that are incumbent upon them? A questionnaire was designed and administered to answer this question. Analysis of responses to 11 ethical responsibility statements (ERSs) by 213 researchers at the Stanford Nanofabrication Facility revealed widespread agreement about a number of work-related ethical responsibilities and substantial divergence in the views about several others. Explanations of this divergence are proposed. A new variable is defined that gauges the respondent’s overall level of discernment of the ethical responsibilities referenced in the ERSs. The mean discernment level score for respondents who had taken a course that included discussion of ethical issues closely related to science, technology, or engineering was significantly higher than for those who had not. Further, among respondents who had taken such a course, the mean discernment level score for those who had taken an ethics course devoted to exploration of ethical issues closely related to science, technology, or engineering was significantly higher than for those who had taken a technical science or engineering course that typically pays only fleeting attention to such issues. Implications of these findings are discussed.     

Effects of minor phrasing variations in traffic-related questionnaires – Comparison of objective equivalences and respondents’ subjective statements

This paper examines questionnaire-related issues of variations in the wording of questions and the objective accuracy of the response, in terms of three traffic-related phenomena (accident frequency, incident frequency, and speeding). ca. 900 survey respondents, living along four major arterial streets in Malmö, Sweden, estimated the occurrence of the phenomena in question. The dispatch consisted of two parallel questionnaires with minor variations in question wording which showed significant differences in the given estimates of accident and incident frequency depending on the phrasing. The correspondence between the subjective estimates and objectively measured accident and speeding figures is discussed along with the potential impact of wording, the phenomenon’s characteristics and response scales. Increased knowledge of these factors could ease communication with the public in the municipal traffic-planning process.     

Repetition and Reconfiguration: Demand‐Based Confabulation in Initial Eyewitness Memory

Repeated recall, as a result of repeated questioning, is typical of situations involving eyewitness evidence. The present study addressed questions of eyewitness performance with reference to repeated questioning in an initial interview, in a format based on actual police procedures. This experiment focused on eyewitness accuracy, eyewitness confidence, and the addition of false details to eyewitness accounts. Ninety‐two adult respondents were asked to recall all the information they could from a single viewing of a scene depicting an assailant aiming a handgun at a victim. This initial question was followed by three additional questions, in which respondents were asked to report any additional details they could recall. On average, respondents provided several times as many correct as false details to the initial question. However, performance deteriorated significantly to the three subsequent questions; on average, across the three subsequent questions, witnesses recalled nearly as much false as accurate information. Witness confidence was positively related to amount of accurate information recalled. However, confidence was also positively related to the number of instances of erroneous recall. These results indicate that reconfigurative dynamics begin to operate, producing confabulated responses in response to questioning demands, as early as the initial interview. This work may also help to clarify the critical relationship between accuracy and confidence in eyewitness reportage, at least within the framework reflected by the present research. Copyright © 2012 John Wiley & Sons, Ltd.     

TSUNAMI: Simultaneous Understanding,Answering, and Memory Interaction for Questions

Question processing involves parsing, memory retrieval, question categorization, initiation of appropriate answer-retrieval heuristics, answer formulation, and output. Computational and psychological models have traditionally treated these processes as separate, sequential, independent, and in pursuit of a single answer type at a time. Here this view is challenged and the implications of a theory in which question processes operate simultaneously on multiple question interpretations are explored. A highly interactive model is described in which an expectation-driven parser generates multiple question candidates, including partially-specified candidates. Question candidates act as constraints for a matcher which activates memory items. An answer retrieval process examines question candidates and the active portions of memory in an attempt to generate answer candidates. Answer candidates are examined by an output process that derives the final answer. These processes run simultaneously and interact. Three experiments on human question answering are also described which provide evidence that working memory load during question reading is affected by processes related to answer retrieval.     

Item Randomized-Response Models for Measuring Noncompliance: Risk-Return Perceptions,Social Influences,and Self-Protective Responses

Randomized response (RR) is a well-known method for measuring sensitive behavior. Yet this method is not often applied because: (i) of its lower efficiency and the resulting need for larger sample sizes which make applications of RR costly; (ii) despite its privacy-protection mechanism the RR design may not be followed by every respondent; and (iii) the incorrect belief that RR yields estimates only of aggregate-level behavior but that these estimates cannot be linked to individual-level covariates. This paper addresses the efficiency problem by applying item randomized-response (IRR) models for the analysis of multivariate RR data. In these models, a person parameter is estimated based on multiple measures of a sensitive behavior under study which allow for more powerful analyses of individual differences than available from univariate RR data. Response behavior that does not follow the RR design is approached by introducing mixture components in the IRR models with one component consisting of respondents who answer truthfully and another component consisting of respondents who do not provide truthful responses. An analysis of data from two large-scale Dutch surveys conducted among recipients of invalidity insurance benefits shows that the willingness of a respondent to answer truthfully is related to the educational level of the respondents and the perceived clarity of the instructions. A person is more willing to comply when the expected benefits of noncompliance are minor and social control is strong. The authors are grateful to the reviewers whose suggestions helped to improve the clarity of the paper substantially. The authors also wish to thank the Dutch Ministry of Social Affairs and Employment for making the reported data available. This research was supported in parts by grants from the Social Sciences and Humanities Research Council of Canada and the Canadian Foundation of Innovation.     

Favourable and unfavourable conditions for children's confidence judgments

Two studies are presented in which favourable and unfavourable conditions for children's meta‐cognitive monitoring processes are examined. Previously reported findings have shown that especially children's uncertainty monitoring (in contrast to certainty monitoring) poses specific problems for children in their elementary school years. When interviewing children about an observed event, answerable and unanswerable questions in two question formats (unbiased and misleading) were used, and 8‐ and 10‐year‐old children as well as adults were asked to rate their confidence on a three‐point scale concerning each response. Results of Study 1 show that accuracy instructions and the option to answer with ‘I don't know’ inflate children's level of confidence because uncertain answers are withheld. Results of Study 2 revealed that children's difficulty with uncertainty monitoring may lie in a cognitive overload during the interview because immediate confidence judgments were less precise and less adequate compared with delayed confidence judgments. Participants' rating of their uncertainty after having erroneously provided an answer to an unanswerable question proved that children aged 8 years and older are able to experience and report levels of uncertainty but, as was shown for answerable questions, these emerging competencies are dependent on favourable task conditions.     

CONTEMPORARY ISSUES IN THE ANALYSIS OF DATA:

Abstract— We asked active psychological researchers to answer a survey regarding the following data-analytic issues: (a) the effect of reliability on Type I and Type II errors, (b) the interpretation of interaction, (c) contrast analysis, and (d) the role of power and effect size in successful replications. Our 551 participants (a 60% response rate) answered 59% of the questions correctly; 46% accuracy would be expected according to participants' response preferences alone. Accuracy was higher for respondents with higher academic ranks and for questions with "no" as the right answer. It is suggested that although experienced researchers are able to answer difficult but basic data-analytic questions at better than chance levels, there is also a high degree of misunderstanding of some fundamental issues of data analysis.     

Sensitive questions in surveys

Psychologists have worried about the distortions introduced into standardized personality measures by social desirability bias. Survey researchers have had similar concerns about the accuracy of survey reports about such topics as illicit drug use, abortion, and sexual behavior. The article reviews the research done by survey methodologists on reporting errors in surveys on sensitive topics, noting parallels and differences from the psychological literature on social desirability. The findings from the survey studies suggest that misreporting about sensitive topics is quite common and that it is largely situational. The extent of misreporting depends on whether the respondent has anything embarrassing to report and on design features of the survey. The survey evidence also indicates that misreporting on sensitive topics is a more or less motivated process in which respondents edit the information they report to avoid embarrassing themselves in the presence of an interviewer or to avoid repercussions from third parties.     

Self-Enhancement and Social Responsibility: On Caring More, but Doing Less, Than Others

Self-enhancement biases have been found in a variety of self-rated skills, traits, and abilities, yet past research has not examined whether people show such biases in ratings of their social concern and activism. In the present paper, we report the results of two surveys on this question. In the first survey, 549 adults rated their level of concern and activism on one of six different issues (e.g., homelessness). The results showed a general pattern of self-enhancement in professed concern but self-deprecation in activism. In the second survey, a random-digit dialing method was used to contact a representative sample of the general public. A total of 817 respondents rated their level of concern and activism on the issues of environmental protection, animal welfare, and world hunger. The second survey also explored two techniques for debiasing self-enhancement in concern: one based on a cognitive consistency model, and one based on the salience of others' actions. Findings from the second survey replicated those of the first, and both debiasing techniques failed to reduce self-ratings of concern. Moreover, a sizable number of respondents said that they would do more if others showed more concern. These results are consistent with a social dilemma in which citizens feel a personal sense of concern, but are reluctant to act until others show greater concern.     

Methodological considerations in developing measures of HIV risk-relevant behaviors and attitudes: An empirical illustration

Abstract

This paper describes the pilot testing of a survey designed to assess the prevalence of HIV exposure risk-relevant behaviors and associated attitudes, beliefs and knowledge in a heterogeneous, ethnically diverse, sexually active population (the US Army). The specific objectives for the pilot test were to: (1) define optimal survey question formats for behavioral and attitude items; (2) identify questions that were poorly worded or that would not contribute to the overall objectives of the survey; (3) assess the extent to which respondents could accurately follow instructions; (4) identify appropriate and culturally relevant terminology for sensitive items; (5) determine the appropriate timeframe for recall of simple and complex sexual behaviors; (6) assess the degree of respondent honesty and belief in the anonymity of the survey results.     

Methods of data collection,perceptions of risks and losses,and motivation to give truthful answers to sensitive survey questions

Two experiments examined survey respondents' concerns about risks and losses when answering sensitive questions in a survey. In Experiment 1 subjects watched one of eight different versions of a videotaped staged interview of a woman being asked about abortion and drunk driving. Experiment 2 was an actual survey interview about women's health issues, including abortion. In Experiment 2, a large proportion of the sample had a recent abortion for which validation information from the clinic was available. Results of both experiments indicate that, when responding to questions about sensitive topics, survey respondents are concerned about risks and losses other than those covered by standard confidentiality assurances and that these other concerns affect truthful reporting. Experiment 1 showed the importance of privacy in surveys of sensitive behaviours. The experiments suggest that under some circumstances interviewer‐administration of sensitive questions may produce data as good as or better than self‐administration. Copyright © 1999 John Wiley & Sons, Ltd.     

Measuring Religiosity of East Asians: Multiple Religious Belonging,Believing, and Practicing

Social surveys normally assume that respondents adhere to a single religious faith in belonging, believing, and practicing congruently. Some surveys even take religious identity as the singular measure of religiosity and examine its relationship with other variables. This practice, however, fails to capture nonexclusive and hybrid religiosity, which is arguably the traditional and normal pattern in East Asia while becoming increasingly common in the West. We have developed a new set of survey questions and conducted a survey among East Asian international students at an American university. The findings show that multiple religious belonging, believing, and practicing are quite common, the level of believing and participating in religions varies substantially, and no confession-based single measure of religious identity or practice is sufficient for measuring religiosity. We recommend this set of improved measures of religiosity be adopted in future surveys in East Asia and probably in the West as well.