Complex Communication Dynamics: Exploring the Structure of an Academic Talk

Communication is a multimodal phenomenon. The cognitive mechanisms supporting it are still understudied. We explored a natural dataset of academic lectures to determine how communication modalities are used and coordinated during the presentation of complex information. Using automated and semi‐automated techniques, we extracted and analyzed, from the videos of 30 speakers, measures capturing the dynamics of their body movement, their slide change rate, and various aspects of their speech (speech rate, articulation rate, fundamental frequency, and intensity). There were consistent but statistically subtle patterns in the use of speech rate, articulation rate, intensity, and body motion across the presentation. Principal component analysis also revealed patterns of system‐like covariation among modalities. These findings, although tentative, do suggest that the cognitive system is integrating body, slides, and speech in a coordinated manner during natural language use. Further research is needed to clarify the specific coordination patterns that occur between the different modalities.     

A facilitative effect of punishment on unpunished behavior

The key pecking of two pigeons was reinforced on a variable-interval schedule of reinforcement during the presentation of each of two stimuli. In various phases of the experiment, punishment followed every response emitted in the presence of one of the stimuli. In general, when the rate of punished responding changed during the presentation of one stimulus, the rate of unpunished responding during the other stimulus changed in the opposite direction. This sort of change in rate is an example of behavioral contrast. When punishment was introduced, the rate of punished responding decreased and the rate of unpunished responding increased as functions of shock intensity. When the rate of previously punished responding increased after the termination of the shock, the rate of the always unpunished responding decreased. When the procedure correlated with a red key was changed from variable-interval reinforcement and punishment for each response to extinction and no punishment, the rate of reinforced responding during presentations of a green key decreased and then increased while the rate of the previously punished responding during red first increased and then decreased during extinction.     

Propositions, Sets, and Worlds

If we agree with Michael Jubien that propositions do not exist, while accepting the existence of abstract sets in a realist mathematical ontology, then the combined effect of these ontological commitments has surprising implications for the metaphysics of modal logic, the ontology of logically possible worlds, and the controversy over modal realism versus actualism. Logically possible worlds as maximally consistent proposition sets exist if sets generally exist, but are equivalently expressed as maximally consistent conjunctions of the same propositions in corresponding sets. A conjunction of propositions, even if infinite in extent, is nevertheless itself a proposition. If sets and hence proposition sets exist but propositions do not exist, then whether or not modal realism is true depends on which of two apparently equivalent methods of identifying, representing, or characterizing logically possible worlds we choose to adopt. I consider a number of reactions to the problem, concluding that the best solution may be to reject the conventional model set theoretical concept of logically possible worlds as maximally consistent proposition sets, and distinguishing between the actual world alone as maximally consistent and interpreting all nonactual merely logically possible worlds as submaximal. I am grateful to the Netherlands Institute for Advanced Study in the Humanities and Social Sciences (NIAS), Royal Netherlands Academy of Arts and Sciences (KNAW), for supporting this among related research projects in philosophical logic and philosophy of mathematics during my Resident Research Fellowship in 2005-2006.     

The Bursts and Lulls of Multimodal Interaction: Temporal Distributions of Behavior Reveal Differences Between Verbal and Non‐Verbal Communication

Recent studies of naturalistic face‐to‐face communication have demonstrated coordination patterns such as the temporal matching of verbal and non‐verbal behavior, which provides evidence for the proposal that verbal and non‐verbal communicative control derives from one system. In this study, we argue that the observed relationship between verbal and non‐verbal behaviors depends on the level of analysis. In a reanalysis of a corpus of naturalistic multimodal communication (Louwerse, Dale, Bard, & Jeuniaux, 2012 ), we focus on measuring the temporal patterns of specific communicative behaviors in terms of their burstiness. We examined burstiness estimates across different roles of the speaker and different communicative modalities. We observed more burstiness for verbal versus non‐verbal channels, and for more versus less informative language subchannels. Using this new method for analyzing temporal patterns in communicative behaviors, we show that there is a complex relationship between verbal and non‐verbal channels. We propose a “temporal heterogeneity” hypothesis to explain how the language system adapts to the demands of dialog.     

Disequilibrium in the mind,disharmony in the body

Although it is generally acknowledged that experiences of frustration, confusion, and anxiety are embodied phenomena, very little is known about how these processes modulate presumably unconscious, but constantly present, subtle bodily movement. We addressed this problem by tracking the low-level dynamics of body movement, using 1/f noise, pink noise, or “fractal scaling”, during naturalistic experiences of affect in two studies involving deep learning and effortful problem-solving. Our results indicate that body movement fluctuations of individuals experiencing cognitive equilibrium was characteristic of correlated pink noise, but there was a whitening of the signal when participants experienced states that are diagnostic of cognitive distress such as anxiety, confusion, and frustration. We orient our findings within theories that emphasise the embodied nature of cognition and affect and with perspectives that view affective and cognitive processes as emergent products of a self-organising dynamical system (the brain) that is inextricably coupled to the body.     

Dealing With Complexity Differently: From Interaction-Dominant Dynamics to Theoretical Plurality

We describe how the work of Guy Van Orden has been deeply influential in a variety of ways and focus on 2 important features: measurement and context. The centrality of these variables in understanding how psychological regularities emerge in our investigative contexts, and evolve into theories, recommends a different way of dealing with complexity. We argue that the Van Orden approach has, as one possible consequence, a plural approach to psychological phenomena. We end by describing what this means for cognitive science.     

Object-based representations govern both the storage of information in visual short-term memory and the retrieval of information from it

Interest is growing in how information is retained in visual short-term memory (VSTM). We describe an experiment that assesses VSTM within the context of multidimensional signal detection theory. On every trial, participants were presented with a 250-ms display containing four colored shapes. They were then probed 900 ms later with a colored shape and made separate old/new judgments about the color and the shape. In any particular trial, one, both, or neither of the probed features had been presented. Performance differed according to whether both probed features belonged to a single object or to two different objects. When both probed features belonged to the same object, featural retrieval was better than would be predicted by independent feature storage. When both probed features belonged to two different objects, featural retrieval was worse than would be predicted by independent feature storage. We conclude that storage in and retrieval from VSTM operate on the basis of object-based representations.     

The structure of language abilities at 4 years: a twin study

Normal language development was studied in 310 pairs of 4-year-old twins born in the United Kingdom in 1994. Twins were assessed individually in their homes on a diverse battery of language and nonverbal measures. Rotated factor analyses indicated the presence of a general Language factor (L) as well as a general Nonverbal (NV) factor. Moderate genetic influence was found for both L and NV abilities. Bivariate genetic analysis estimated a genetic correlation of .63 between L and NV abilities, implying that over half of the genetic influence on L overlaps with genetic influence on NV. These results suggest that at age 4, genetic influences on individual differences in language overlap substantially with genetic influences on individual differences in other cognitive abilities, although perhaps less so than later in development.