Do procedures for verbal reporting of thinking have to be reactive? A meta-analysis and recommendations for best reporting methods

Since its establishment, psychology has struggled to find valid methods for studying thoughts and subjective experiences. Thirty years ago, Ericsson and Simon (1980) proposed that participants can give concurrent verbal expression to their thoughts (think aloud) while completing tasks without changing objectively measurable performance (accuracy). In contrast, directed requests for concurrent verbal reports, such as explanations or directions to describe particular kinds of information, were predicted to change thought processes as a consequence of the need to generate this information, thus altering performance. By comparing performance of concurrent verbal reporting conditions with their matching silent control condition, Ericsson and Simon found several studies demonstrating that directed verbalization was associated with changes in performance. In contrast, the lack of effects of thinking aloud was merely suggested by a handful of experimental studies. In this article, Ericsson and Simon's model is tested by a meta-analysis of 94 studies comparing performance while giving concurrent verbalizations to a matching condition without verbalization. Findings based on nearly 3,500 participants show that the "think-aloud" effect size is indistinguishable from zero (r = -.03) and that this procedure remains nonreactive even after statistically controlling additional factors such as task type (primarily visual or nonvisual). In contrast, procedures that entail describing or explaining thoughts and actions are significantly reactive, leading to higher performance than silent control conditions. All verbal reporting procedures tend to increase times to complete tasks. These results suggest that think-aloud should be distinguished from other methods in future studies. Theoretical and practical implications are discussed.     

Usability evaluation with screen reader users: a video presentation of the PCTA’s experimental setting and rules

In the study entitled “Web usability evaluation with screen reader users: Implementation of the Partial Concurrent Thinking Aloud technique” (Federici et al. 2010), we have proposed a modified protocol of usability evaluation technique for blind users, which integrates the features of the concurrent and the retrospective techniques. This new technique, called partial concurrent thinking aloud (PCTA), while respecting the properties of classic verbal protocols, overcomes the structural interference and the limits of concurrent and retrospective protocols when used with screen reader users. In order to facilitate understanding and acquisition of the PCTA’s technique for practitioners and researchers, we have video recorded three different verbal protocols by visualizing five experimental sections. In the first two videos, we have compared a concurrent with a retrospective’s verbal protocol of a sighted user, showing the difference of the verbalizations provided by the user in these two conditions. The third video shows the structural interference of the screen reader, during a blind user concurrent thinking aloud. In the last two videos, we show the difference of a blind user behaviour when PCTA or retrospective protocol is adopted. The videos clearly visualize the advantage of the PCTA use in respect of the two other protocols. In conclusion, the visualization of the PCTA technique confirms that this new verbal protocol promotes and guarantees a more user-driven usability assessment with disabled people, by better involving screen reader users, overcoming the structural interference and the limits of the concurrent and retrospective protocols.     

The suppression of exciting thoughts

We examined how the suppression of an exciting thought influences sympathetic arousal as indexed by skin conductance level (SCL). Subjects were asked to think aloud as they followed instructions to think about or not to think about various topics. Experiment 1 showed that trying not to think about sex, like thinking about sex, elevates SCL in comparison to thinking about or not thinking about less exciting topics (e.g., dancing). Experiment 2 revealed that the suppression of the thought of sex yielded SCL elevation whether or not subjects believed their think-aloud reports would be private or public, and it also revealed that the effect dissipated over the course of a few minutes. Experiment 3 found such dissipation again but showed that subsequent intrusions of the suppressed exciting thought are associated with further elevations in SCL over 30 min. Because such an association was not found when subjects were trying to think about the exciting thought, it was suggested that the suppression of exciting thoughts might be involved in the production of chronic emotional responses such as phobias and obsessive preoccupations.     

Thinking aloud performance: Individual consistencies over tasks

Previous research (Gilhooly, 1987) indicated considerable individual consistency in quantity and quality of thinking aloud performance over three 8-puzzle tasks. The present study investigated the consistency of individual thinking aloud performance over two types of tasks. One type was largely spatial (the 8-puzzle) and the other largely verbal (the Poisoned Foods task). Considerable individual consistency in thinking aloud performance was found within each task type and, more importantly, between types of task. Thus, subjects who produced lengthy and complete protocols on the 8-puzzle problems also did so in the Poisoned Foods task. These results are discussed in the context of Ericsson and Simon’s (1984) model of thinking aloud.     

Thinking aloud performance: Individual consistencies over tasks

Previous research (Gilhooly, 1987) indicated considerable individual consistency in quantity and quality of thinking aloud performance over three 8-puzzle tasks. The present study investigated the consistency of individual thinking aloud performance over two types of tasks. One type was largely spatial (the 8-puzzle) and the other largely verbal (the Poisoned Foods task). Considerable individual consistency in thinking aloud performance was found within each task type and, more importantly, between types of task. Thus, subjects who produced lengthy and complete protocols on the 8-puzzle problems also did so in the Poisoned Foods task. These results are discussed in the context of Ericsson and Simon’s (1984) model of thinking aloud.     

The use of verbal protocols as data: An analysis of insight in the candle problem

In the present study, we examined the use of verbal protocols as data in the study of the cognitive processes underlying insight. Fifty-eight Temple University undergraduates attempted to solve Duncker's (1945) candle problem either silently or while thinking aloud. Solution rates, solving times, and solution types were comparable between conditions, suggesting that verbal overshadowing (Schooler, Ohlsson, & Brooks, 1993) did not occur when the participants attempted to solve the candle problem. Subsequent analysis of verbal protocols provided a catalogue of solutions generated by the participants, as well as empirical support for the occurrence of impasse and restructuring. Although restructuring was present in the majority of protocols, including those of the participants who later produced the box solution, the presence of impasse occurred with less frequency and was not associated with production of the box solution. These results provide information concerning how the candle problem is solved and suggest that verbalization can be used to examine how individuals solve insight problems and to evaluate existing theories of insight.     

Typing versus thinking aloud when reading: Implications for computer-based assessment and training tools

The goal of this study was to assess the impact of modality of production of think-aloud protocols on reading strategies. Readers in two studies spoke or typed protocols for narrative or science texts and completed comprehension tests for each text. Human judges identified the presence of paraphrasing, bridging inferences, and elaborating within the protocols. Reading comprehension skill was assessed with the Nelson-Denny test. With respect to narrative texts, paraphrasing and bridging were less frequent when readers were typing than when they were thinking aloud. With respect to science texts, less-skilled readers made bridging inferences more frequently when typing than when speaking. Conversely, skilled readers generated more paraphrases than bridges when typing thoughts but not when speaking. These results have implications for computer-based tools for reading assessment and intervention.     

Intimate distances: William James’ introspection, Buddhist mindfulness, and experiential inquiry

The recent and growing interest in ‘mindfulness’ and ‘mindfulness meditation’ across disciplines in the West presents us with a unique opportunity to reconsider whether Buddhism has anything to offer our contemporary psychological investigations. I argue that the Buddhist-inspired practice of mindfulness has potentially profound implications for the ways in which we conduct our investigations as psychologists, and that, as a style of experiential inquiry, it has at least one Western antecedent in the early introspectionist method of William James. Both are practices of becoming aware of experience; and paradoxically becoming intimately distant with our experience. I present a non-dualistic approach in which introspection and mindfulness are seen not only as psychological but also as social practices, operating simultaneously at the boundary of the individual/inner and social/outer, collapsing such distinctions in practice, and radically undermining the distinction between self and other. While there are similarities between James’ practice of introspection and mindfulness, there are also differences, and I suggest that they should not be easily conflated. Clarifying their relationship should be helpful, not only in distinguishing them from one another, but also in pointing to how mindfulness might allow a broader application than James’ introspection once did.     

Understanding unfamiliar words in young, young-old, and old-old adults: inferential processing and the abstraction-deficit hypothesis

Previous research suggests that older adults derive interpretations of unfamiliar words that are less precise than those of young adults (D. McGinnis & E. M. Zelinski, 2000). Thirty-one adults aged 18-37, 27 aged 65-74, and 28 aged 75-87 read passages containing unfamiliar words (1 per passage) and were asked to think aloud during reading. After reading each passage, participants selected meaning-relevant cues and rated the quality of 4 definition options. Compared with the 2 younger groups, the oldest group rated thematic and irrelevant definitions significantly higher, and their think-aloud protocols included more generalized inferences. Results pertaining to cue selection were not significant. Taken together, these results suggest that age differences in meaning derivation may be related to inferential processing that is overgeneralized, providing support for the abstraction-deficit hypothesis.     

The influence of articulation,self‐monitoring ability,and sensitivity to others on creativity

Although it is often recommended to think aloud to solve problems and to become more creative, cognitive and social psychological research suggests thinking aloud may actually produce less creative ideas than thinking in silence. The results of two experiments indeed showed that thinking aloud hinders creativity—although people produced the same amount of new uses for an object, these were judged to be less original in the thinking aloud condition. Experiment 2 further showed that this effect was particularly pronounced for individuals with high sensitivity to what other's think of them and low ability to adapt to these expectations. From this, we conclude that the felt presence of an actual or implied audience when thinking aloud, reduces creative idea generation especially among those having difficulty adapting to others. Implications for creativity research, and for promoting creativity in applied settings such as organizational teams are discussed. Copyright © 2006 John Wiley & Sons, Ltd.     

Visual impairment, verbal humor, and conservation

Through a cerebral hemispheric specialization model, I speculated that the reported discrepancy between visually impaired children's competency in language development and lags in cognition may be explained by the use of visual-spatial measurements of cognition. The performance of 51 visually impaired and 51 sighted children, ages 7 to 15 years, on a task of joke comprehension and conservation of weight scores was analyzed. I hypothesized that the ability to comprehend verbal humor would reflect the function of the left hemisphere of the brain and the ability to perform a Piagetian visual-spatial task would reflect the function of the right hemisphere. The findings indicate that although age and IQ contributed significantly to predicting joke comprehension, the ability to conserve did not have a unique contribution. In conclusion, I suggest that the cerebral hemispheric specialization model be considered when investigating language and cognition of visually impaired children.     

Web usability evaluation with screen reader users: implementation of the partial concurrent thinking aloud technique

A verbal protocol technique, adopted for a web usability evaluation, requires that the users are able to perform a double task: surfing and talking. Nevertheless, when blind users surf by using a screen reader and talk about the way they interact with the computer, the evaluation is influenced by a structural interference: users are forced to think aloud and listen to the screen reader at the same time. The aim of this study is to build up a verbal protocol technique for samples of visual impaired users in order to overcome the limits of concurrent and retrospective protocols. The technique we improved, called partial concurrent thinking aloud (PCTA), integrates a modified set of concurrent verbalization and retrospective analysis. One group of 6 blind users and another group of 6 sighted users evaluated the usability of a website using PCTA. By estimating the number of necessary users by the means of an asymptotic test, it was found out that the two groups had an equivalent ability of identifying usability problems, both over 80%. The result suggests that PCTA, while respecting the properties of classic verbal protocols, also allows to overcome the structural interference and the limits of concurrent and retrospective protocols when used with screen reader users. In this way, PCTA reduces the efficiency difference of usability evaluation between blind and sighted users.     

On the development of knowledge during problem solving

A person might solve a problem progressing very quickly after an initial period of very little progress, as if new insight was suddenly gained, without that person possessing anything that merits being called insight. This is demonstrated in a series of studies as an extension of the experiments of Kotovsky and Simon (1990). A total of 68 undergraduate students solved an isomorph of the Chinese Ring Puzzle. The first study showed that the verbal protocols of participants thinking aloud contained accumulating explicit knowledge. In the second study, the participants' move records were analysed in a novel way, and it was shown that the choices of moves gradually improved. In the third study the knowledge about the puzzle was tested after the puzzle was solved. The participants demonstrated that they knew, although imperfectly, which move was possible in a given situation. The fourth study showed that verbalising did not alter problem solving profoundly. Together, the four studies demonstrated that solving the puzzle took place after a gradual development of knowledge, instead of a flash of insight.     

Shoemaker, Self–Blindness and Moore's Paradox

I show how the 'inner–sense' (quasi–perceptual) view of introspection can be defended against Shoemaker's influential 'argument from self–blindness'. If introspection and perception are analogous, the relationship between beliefs and introspective knowledge of them is merely contingent. Shoemaker argues that this implies the possibility that agents could be self–blind, i.e., could lack any introspective awareness of their own mental states. By invoking Moore's paradox, he rejects this possibility. But because Shoemaker's discussion conflates introspective awareness and self–knowledge, he cannot establish his conclusion. There is third–person evidence available to the self–blind which Shoemaker ignores, and it can account for the considerations from Moore's paradox that he raises.     

Context, Cognition, and Common Method Variance: Psychometric and Verbal Protocol Evidence

Researchers continue to debate the importance of (item) context effects, which are often thought to produce inflated percept-percept correlations in organizational self-reports. Using Feldman and Lynch's (1988) theory of self-generated validity, we propose five conditions under which such context effects are most likely to occur and to have an impact on substantive conclusions. The proposed effects are tested with psychometric and verbal protocol data from 208 subjects responding to an organizational justice questionnaire, using a 3 (types of context) by 2 ("think aloud" versus "silent") experimental design. Psychometric results revealed context effects on scale means, reliabilities, and some of the relations between constructs. Respondents' concurrent verbal protocols from the "think aloud" condition provided evidence for the cognitive basis of these effects.     

It's Time We Brought Introspection Out of the Closet

ABSTRACT— I identify the main reasons why introspection unofficially has been banned from the field for some 100 years, though it has been partly smuggled back in through the use of "verbal (self-) reports." I then show why introspection is indispensable to psychology. I argue that this ban has retarded the progress of psychology by taking attention away from the problem of the accuracy of self-reports, limiting our understanding of concepts like emotion, discouraging psychologists from using introspection themselves, and ignoring the study of how to develop introspective skill. I conclude by suggesting some approaches to validating introspective reports.     

Gambling: Relationship Between the Frequency of Wins and Irrational Thinking

In this study, we evaluated the relationship between the number of wins and the number of irrational verbalizations on the part of individuals playing American roulette. We predicted that subjects who win frequently would make more irrational verbalizations than rational verbalizations compared to subjects who win infrequently. Twenty Canadian subjects (8 men, 12 women) were randomly divided into two experimental groups (frequent vs. infrequent wins). The subjects used the thinking aloud procedure to verbalize their thoughts, which were then classified according to degree of rationality or irrationality. The results showed that irrational verbalizations outnumbered rational verbalizations in both groups. A few wins were sufficient to evoke an illusory perception of control.     

Individual differences in strategies for syllogistic reasoning

Current theories of reasoning such as mental models or mental logic assume a universal cognitive mechanism that underlies human reasoning performance. However, there is evidence that this is not the case, for example, the work of Ford (1995), who found that some people adopted predominantly spatial and some verbal strategies in a syllogistic reasoning task. Using written and think-aloud protocols, the present study confirmed the existence of these individual differences. However, in sharp contrast to Ford, the present study found few differences in reasoning performance between the two groups, in terms of accuracy or type of conclusion generated. Hence, reasoners present an outward appearance of ubiquity, despite underlying differences in reasoning processes. These findings have implications for theoretical accounts of reasoning, and for attempts to model reasoning data. Any comprehensive account needs to account for strategic differences and how these may develop in logically untrained individuals.