Keeping the Result in Sight and Mind: General Cognitive Principles and Language‐Specific Influences in the Perception and Memory of Resultative Events

We study how people attend to and memorize endings of events that differ in the degree to which objects in them are affected by an action: Resultative events show objects that undergo a visually salient change in state during the course of the event (peeling a potato), and non‐resultative events involve objects that undergo no, or only partial state change (stirring in a pan). We investigate general cognitive principles, and potential language‐specific influences, in verbal and nonverbal event encoding and memory, across two experiments with Dutch and Estonian participants. Estonian marks a viewer's perspective on an event's result obligatorily via grammatical case on direct object nouns: Objects undergoing a partial/full change in state in an event are marked with partitive/accusative case, respectively. Therefore, we hypothesized increased saliency of object states and event results in Estonian speakers, as compared to speakers of Dutch. Findings show (a) a general cognitive principle of attending carefully to endings of resultative events, implying cognitive saliency of object states in event processing; (b) a language‐specific boost on attention and memory of event results under verbal task demands in Estonian speakers. Results are discussed in relation to theories of event cognition, linguistic relativity, and thinking for speaking.     

Introspective access to implicit shifts of attention

Literature in metacognition has systematically rejected the possibility of introspective access to complex cognitive processes. This situation derives from the difficulty of experimentally manipulating cognitive processes while abiding by the two contradictory constraints. First, participants must not be aware of the experimental manipulation, otherwise they run the risk of incorporating their knowledge of the experimental manipulation in some rational elaboration. Second, we need an external, third person perspective evidence that the experimental manipulation did impact some relevant cognitive processes. Here, we study introspection during visual searches, and we try to overcome the above dilemma, by presenting a barely visible, “pre-conscious” cue just before the search array. We aim at influencing the attentional guidance of the search processes, while participants would not notice that fact. Results show that introspection of the complexity of a search process is driven in part by subjective access to its attentional guidance.     

Degree of Language Experience Modulates Visual Attention to Visible Speech and Iconic Gestures During Clear and Degraded Speech Comprehension

Visual information conveyed by iconic hand gestures and visible speech can enhance speech comprehension under adverse listening conditions for both native and non‐native listeners. However, how a listener allocates visual attention to these articulators during speech comprehension is unknown. We used eye‐tracking to investigate whether and how native and highly proficient non‐native listeners of Dutch allocated overt eye gaze to visible speech and gestures during clear and degraded speech comprehension. Participants watched video clips of an actress uttering a clear or degraded (6‐band noise‐vocoded) action verb while performing a gesture or not, and were asked to indicate the word they heard in a cued‐recall task. Gestural enhancement was the largest (i.e., a relative reduction in reaction time cost) when speech was degraded for all listeners, but it was stronger for native listeners. Both native and non‐native listeners mostly gazed at the face during comprehension, but non‐native listeners gazed more often at gestures than native listeners. However, only native but not non‐native listeners' gaze allocation to gestures predicted gestural benefit during degraded speech comprehension. We conclude that non‐native listeners might gaze at gesture more as it might be more challenging for non‐native listeners to resolve the degraded auditory cues and couple those cues to phonological information that is conveyed by visible speech. This diminished phonological knowledge might hinder the use of semantic information that is conveyed by gestures for non‐native compared to native listeners. Our results demonstrate that the degree of language experience impacts overt visual attention to visual articulators, resulting in different visual benefits for native versus non‐native listeners.     

Cognitive Task Analysis for Implicit Knowledge About Visual Representations With Similarity Learning Methods

Visual representations are prevalent in STEM instruction. To benefit from visuals, students need representational competencies that enable them to see meaningful information. Most research has focused on explicit conceptual representational competencies, but implicit perceptual competencies might also allow students to efficiently see meaningful information in visuals. Most common methods to assess students’ representational competencies rely on verbal explanations or assume explicit attention. However, because perceptual competencies are implicit and not necessarily verbally accessible, these methods are ill‐equipped to assess them. We address these shortcomings with a method that draws on similarity learning, a machine learning technique that detects visual features that account for participants’ responses to triplet comparisons of visuals. In Experiment 1, 614 chemistry students judged the similarity of Lewis structures and in Experiment 2, 489 students judged the similarity of ball‐and‐stick models. Our results showed that our method can detect visual features that drive students’ perception and suggested that students’ conceptual knowledge about molecules informed perceptual competencies through top‐down processes. Furthermore, Experiment 2 tested whether we can improve the efficiency of the method with active sampling. Results showed that random sampling yielded higher accuracy than active sampling for small sample sizes. Together, the experiments provide the first method to assess students’ perceptual competencies implicitly, without requiring verbalization or assuming explicit visual attention. These findings have implications for the design of instructional interventions that help students acquire perceptual representational competencies.     

“Seeing Rain”: Integrating phenomenological and Bayesian predictive coding approaches to visual hallucinations and self-disturbances (Ichstörungen) in schizophrenia

We present a schizophrenia patient who reports “seeing rain” with attendant somatosensory features which separate him from his surroundings. Because visual/multimodal hallucinations are understudied in schizophrenia, we examine a case history to determine the role of these hallucinations in self-disturbances (Ichstörungen). Developed by the early Heidelberg School, self-disturbances comprise two components: 1. The self experiences its own automatic processing as alien to self in a split-off, “doubled-I.” 2. In “I-paralysis,” the disruption to automatic processing is now outside the self in omnipotent agents. Self-disturbances (as indicated by visual/multimodal hallucinations) involve impairment in the ability to predict moment-to-moment experiences in the ongoing perception-action cycle. The phenomenological approach to subjective experience of self-disturbances complements efforts to model psychosis using the computational framework of hierarchical predictive coding. We conclude that self-disturbances play an adaptive, compensatory role following the uncoupling of perception and action, and possibly, other low-level perceptual anomalies.     

The implications of low quality bicycle paths on gaze behavior of cyclists: A field test

Unlike for car driving and walking, the visual behavior during cycling is poorly documented. The aim of this experiment was to explore the visual behavior of adult bicycle users ‘in situ’ and to investigate to what extent the surface quality affects this behavior. Therefore cycling speed, gaze distribution and gaze location of five participants were analyzed on a high and a low quality bicycle track. Although there was no difference in cycling speed between the low and the high quality cycling path, there was an apparent shift of attention from distant environmental regions to more proximate road properties on the low quality track. These findings suggest that low quality bicycle tracks may affect the alertness and responsiveness of cyclists to environmental hazards.     

Use of experimenter-given cues in dogs

Since the observations of O. Pfungst the use of human-provided cues by animals has been well-known in the behavioural sciences (“Clever Hans effect”). It has recently been shown that rhesus monkeys (Macaca mulatta) are unable to use the direction of gazing by the experimenter as a cue for finding food, although after some training they learned to respond to pointing by hand. Direction of gaze is used by chimpanzees, however. Dogs (Canis familiaris) are believed to be sensitive to human gestural communication but their ability has never been formally tested. In three experiments we examined whether dogs can respond to cues given by humans. We found that dogs are able to utilize pointing, bowing, nodding, head-turning and glancing gestures of humans as cues for finding hidden food. Dogs were also able to generalize from one person (owner) to another familiar person (experimenter) in using the same gestures as cues. Baseline trials were run to test the possibility that odour cues alone could be responsible for the dogs’ performance. During training individual performance showed limited variability, probably because some dogs already “knew” some of the cues from their earlier experiences with humans. We suggest that the phenomenon of dogs responding to cues given by humans is better analysed as a case of interspecific communication than in terms of discrimination learning. Received: 30 May 1998 / Accepted after revision: 6 September 1998     

Response preferences of monkeys (Macaca mulatta) within wavelength and line-tilt dimensions

Four rhesus monkeys were tested for preferences within the wavelength and line-tilt dimensions. In the case of wavelength, the response panel was back-illuminated by light of one of the following wavelengths, presented in a random manner: 470, 525, 580, and 635 nm. Similarly, the line-tilt dimension was studied, by presenting a 5 cm by 0.3 cm black bar tilted at 0, 30, 60, or 90 degrees. No preferences were found within this latter dimension; in contrast, marked wavelength preferences existed, the order of preference being 470 (most preferred), 525, 580, and 635 nm (least preferred). These response preferences were resistant to behavioral manipulation; the number of responses to blue and to red in extinction was about equal when red was used as the training stimulus, but vastly different following training on blue. These results indicate that such response preferences must be taken into account in the design of a wide variety of experiments.     

Artificial consciousness and the consciousness-attention dissociation

Artificial Intelligence is at a turning point, with a substantial increase in projects aiming to implement sophisticated forms of human intelligence in machines. This research attempts to model specific forms of intelligence through brute-force search heuristics and also reproduce features of human perception and cognition, including emotions. Such goals have implications for artificial consciousness, with some arguing that it will be achievable once we overcome short-term engineering challenges. We believe, however, that phenomenal consciousness cannot be implemented in machines. This becomes clear when considering emotions and examining the dissociation between consciousness and attention in humans. While we may be able to program ethical behavior based on rules and machine learning, we will never be able to reproduce emotions or empathy by programming such control systems—these will be merely simulations. Arguments in favor of this claim include considerations about evolution, the neuropsychological aspects of emotions, and the dissociation between attention and consciousness found in humans. Ultimately, we are far from achieving artificial consciousness.