Perceiving images and telling tales: A visual and verbal analysis of the meaning of the internet

This paper uses visual and verbal analysis to delve into the multi-faceted ways in which individuals construct their own meanings and shape their own experiences with the Internet. We build on the Zaltman Metaphor Elicitation Technique, and the principles of visual rhetoric to show how perceptual processes affect picture choices, and how these choices contribute to the narrative imagination. Numerous perceptual principles [abstraction, concept formation, perceptual problem solving, constancy, closure, symmetry and balance]are identified in the choice and organization of visual images. The argument we make is that images and words (visual and textual processes) provide deeper insights into our understanding of consumer online experiences.     

Number versus continuous quantity in numerosity judgments by fish

A fundamental question in human cognition is how people reason about space. We use a computational model to explore cross-cultural commonalities and differences in spatial cognition. Our model is based upon two hypotheses: (1) the structure-mapping model of analogy can explain the visual comparisons used in spatial reasoning; and (2) qualitative, structural representations are computed by people’s visual systems and used in these comparisons. We apply our model to a visual oddity task, in which individuals are shown an array of two-dimensional images and asked to the pick the one that does not belong. This task was previously used to evaluate understanding of geometric concepts in two disparate populations: North Americans, and the Mundurukú, a South American indigenous group. Our model automatically generates representations of each hand-segmented image and compares them to solve the task. The model achieves human-level performance on this task, and problems that are hard for the model are also difficult for people in both cultures. Furthermore, ablation studies on the model suggest explanations for cross-cultural differences in terms of differences in spatial representations.     

A computational framework for attentional object discovery in RGB-D videos

We present a computational framework for attention-guided visual scene exploration in sequences of RGB-D data. For this, we propose a visual object candidate generation method to produce object hypotheses about the objects in the scene. An attention system is used to prioritise the processing of visual information by (1) localising candidate objects, and (2) integrating an inhibition of return (IOR) mechanism grounded in spatial coordinates. This spatial IOR mechanism naturally copes with camera motions and inhibits objects that have already been the target of attention. Our approach provides object candidates which can be processed by higher cognitive modules such as object recognition. Since objects are basic elements for many higher level tasks, our architecture can be used as a first layer in any cognitive system that aims at interpreting a stream of images. We show in the evaluation how our framework finds most of the objects in challenging real-world scenes.     

The secret is in the sound: from unsegmented speech to lexical categories

When learning language, young children are faced with many seemingly formidable challenges, including discovering words embedded in a continuous stream of sounds and determining what role these words play in syntactic constructions. We suggest that knowledge of phoneme distributions may play a crucial part in helping children segment words and determine their lexical category, and we propose an integrated model of how children might go from unsegmented speech to lexical categories. We corroborated this theoretical model using a two‐stage computational analysis of a large corpus of English child‐directed speech. First, we used transition probabilities between phonemes to find words in unsegmented speech. Second, we used distributional information about word edges – the beginning and ending phonemes of words – to predict whether the segmented words from the first stage were nouns, verbs, or something else. The results indicate that discovering lexical units and their associated syntactic category in child‐directed speech is possible by attending to the statistics of single phoneme transitions and word‐initial and final phonemes. Thus, we suggest that a core computational principle in language acquisition is that the same source of information is used to learn about different aspects of linguistic structure.     

Image Interpretation: Bridging the Gap from Mechanically Produced Image to Representation

There is currently a gap in our understanding of how figures produced by mechanical imaging techniques play evidential roles: several studies based on close examination of scientific practice show that imaging techniques do not yield data whose significance can simply be read off the image. If image-making technology is not a simple matter of nature re-presenting itself to us in a legible way, just how do the images produced provide support for scientific claims? In this article I will first show that there is a distinct question about the semiotics of mechanically produced images that has not yet been answered. I show that my account of visual representations can do so, and I argue that the role of convention involved in my account is compatible with the view that visual representations produced through mechanized imaging techniques can play genuine evidential roles in scientific reasoning.     

Measuring the size of mental images

Mental images seem to have a size; the experimental problem was to map that image size onto a scale of physical measurement. To this end, two experiments were conducted to measure the size of mental images in degrees of visual angle. In Experiment 1, college students employed light pointers to indicate the horizontal extent of projected mental images of words (the letter string, not the referent). Imagined words covered about 1.0 degress of visual angle per letter. In Experiment 2, a more objective eye-movement response was used to measure the visual angle size of imagined letter strings. Visual angle of eye movement was found to increase regularly as the letter distance between the fixation point and a probed letter position increased. Each letter occupied about 2.5 degrees of visual angle for the four-letter strings in the control/default size condition. Possible relations between eye movements and images are discussed.     

Robust representations for face recognition: the power of averages

We are able to recognise familiar faces easily across large variations in image quality, though our ability to match unfamiliar faces is strikingly poor. Here we ask how the representation of a face changes as we become familiar with it. We use a simple image-averaging technique to derive abstract representations of known faces. Using Principal Components Analysis, we show that computational systems based on these averages consistently outperform systems based on collections of instances. Furthermore, the quality of the average improves as more images are used to derive it. These simulations are carried out with famous faces, over which we had no control of superficial image characteristics. We then present data from three experiments demonstrating that image averaging can also improve recognition by human observers. Finally, we describe how PCA on image averages appears to preserve identity-specific face information, while eliminating non-diagnostic pictorial information. We therefore suggest that this is a good candidate for a robust face representation.     

The embodied nature of medical concepts: image schemas and language for pain

Cognitive linguistics assumes that knowledge is both embodied and situated as far as it is acquired through our bodily interaction with the world in a specific environment (e.g. Barsalou in Lang Cogn Process 18:513–562, 2003; Connell et al. in PLoS One 7:3, 2012). Therefore, embodiment provides an explanation to the mental representation and linguistic expression of concepts. Among the first, we find multimodal conceptual structures, like image schemas, which are schematic representations of embodied experiences resulting from our conceptualization of the surrounding environment (Tercedor Sánchez et al. in J Spec Transl 18:187–205, 2012). Furthermore, the way we interact with the environment and its objects is dynamic and configures how we refer to concepts both by means of images and lexicalizations. In this article, we investigate how image schemas underlie verbal and visual representations. They both evoke concepts based on exteroception, interoception and proprioception which can be lexicalized through language. More specifically, we study (1) a multimodal corpus of medical texts to examine how image schemas lexicalize in the language of medicine to represent specialized concepts and (2) medical pictures to explore the depiction of image-schematic concepts, in order to account for the verbal and visual representation of embodied concepts. We explore the concept pain, a sensory and emotional experience associated with actual or potential tissue damage, using corpus analysis tools (Sketch Engine) to extract information about the lexicalization of underlying image schemas in definitions and defining contexts. Then, we use the image schemas behind medical concepts to consistently select images which depict our experience of pain and the way we understand it. Finally, such lexicalizations and visualizations will help us assess how we refer to pain both verbally and visually.     

The Cognitive Architecture of Perceived Animacy: Intention,Attention, and Memory

Human vision supports social perception by efficiently detecting agents and extracting rich information about their actions, goals, and intentions. Here, we explore the cognitive architecture of perceived animacy by constructing Bayesian models that integrate domain‐specific hypotheses of social agency with domain‐general cognitive constraints on sensory, memory, and attentional processing. Our model posits that perceived animacy combines a bottom–up, feature‐based, parallel search for goal‐directed movements with a top–down selection process for intent inference. The interaction of these architecturally distinct processes makes perceived animacy fast, flexible, and yet cognitively efficient. In the context of chasing, in which a predator (the “wolf”) pursues a prey (the “sheep”), our model addresses the computational challenge of identifying target agents among varying numbers of distractor objects, despite a quadratic increase in the number of possible interactions as more objects appear in a scene. By comparing modeling results with human psychophysics in several studies, we show that the effectiveness and efficiency of human perceived animacy can be explained by a Bayesian ideal observer model with realistic cognitive constraints. These results provide an understanding of perceived animacy at the algorithmic level—how it is achieved by cognitive mechanisms such as attention and working memory, and how it can be integrated with higher‐level reasoning about social agency.     

A perceptually based comparison of image similarity metrics

The assessment of how well one image matches another forms a critical component both of models of human visual processing and of many image analysis systems. Two of the most commonly used norms for quantifying image similarity are L1 and L2, which are specific instances of the Minkowski metric. However, there is often not a principled reason for selecting one norm over the other. One way to address this problem is by examining whether one metric, better than the other, captures the perceptual notion of image similarity. This can be used to derive inferences regarding similarity criteria the human visual system uses, as well as to evaluate and design metrics for use in image-analysis applications. With this goal, we examined perceptual preferences for images retrieved on the basis of the L1 versus the L2 norm. These images were either small fragments without recognizable content, or larger patterns with recognizable content created by vector quantization. In both conditions the participants showed a small but consistent preference for images matched with the L1 metric. These results suggest that, in the domain of natural images of the kind we have used, the L1 metric may better capture human notions of image similarity.     

The organization of words and environmental sounds in the second year: Behavioral and electrophysiological evidence

The majority of research examining early auditory‐semantic processing and organization is based on studies of meaningful relations between words and referents. However, a thorough investigation into the fundamental relation between acoustic signals and meaning requires an understanding of how meaning is associated with both lexical and non‐lexical sounds. Indeed, it is unknown how meaningful auditory information that is not lexical (e.g., environmental sounds) is processed and organized in the young brain. To capture the structure of semantic organization for words and environmental sounds, we record event‐related potentials as 20‐month‐olds view images of common nouns (e.g., dog) while hearing words or environmental sounds that match the picture (e.g., “dog” or barking), that are within‐category violations (e.g., “cat” or meowing), or that are between‐category violations (e.g., “pen” or scribbling). Results show both words and environmental sounds exhibit larger negative amplitudes to between‐category violations relative to matches. Unlike words, which show a greater negative response early and consistently to within‐category violations, such an effect for environmental sounds occurs late in semantic processing. Thus, as in adults, the young brain represents semantic relations between words and between environmental sounds, though it more readily differentiates semantically similar words compared to environmental sounds.