The Epistemological Status of Vision and its Implications for Design

Computational theories of vision typically rely on the analysis of two aspects of human visual function: (1) object and shape recognition (2) co-calibration of sensory measurements. Both these approaches are usually based on an inverse-optics model, where visual perception is viewed as a process of inference from a 2D retinal projection to a 3D percept within a Euclidean space schema. This paradigm has had great success in certain areas of vision science, but has been relatively less successful in understanding perceptual representation, namely, the nature of the perceptual encoding. One of the drawbacks of inverse-optics approaches has been the difficulty in defining the constraints needed to make the inference computationally tractable (e.g. regularity assumptions, Bayesian priors, etc.). These constraints, thought to be learned assumptions about the nature of the physical and optical structures of the external world, have to be incorporated into any workable computational model in the inverse-optics paradigm. But inference models that employ an inverse optics plus structural assumptions approach inevitably result in a naïve realist theory of perceptual representation. Another drawback of inference models for theories of perceptual representation is their inability to explain central features of the visual experience. The one most evident in the process and visual understanding of design is the fact that some visual configurations appear, often spontaneously, as perceptually more coherent than others. The epistemological consequences of inferential approaches to vision indicate that they fail to capture enduring aspects of our visual experience. Therefore they may not be suited to a theory of perceptual representation, or useful for an understanding of the role of perception in the design process and product.     

A sensorimotor account of vision and visual consciousness

Many current neurophysiological, psychophysical, and psychological approaches to vision rest on the idea that when we see, the brain produces an internal representation of the world. The activation of this internal representation is assumed to give rise to the experience of seeing. The problem with this kind of approach is that it leaves unexplained how the existence of such a detailed internal representation might produce visual consciousness. An alternative proposal is made here. We propose that seeing is a way of acting. It is a particular way of exploring the environment. Activity in internal representations does not generate the experience of seeing. The outside world serves as its own, external, representation. The experience of seeing occurs when the organism masters what we call the governing laws of sensorimotor contingency. The advantage of this approach is that it provides a natural and principled way of accounting for visual consciousness, and for the differences in the perceived quality of sensory experience in the different sensory modalities. Several lines of empirical evidence are brought forward in support of the theory, in particular: evidence from experiments in sensorimotor adaptation, visual "filling in," visual stability despite eye movements, change blindness, sensory substitution, and color perception.     

Solving the "real" mysteries of visual perception: the world as an outside memory.

Visual science is currently a highly active domain, with much progress being made in fields such as colour vision, stereo vision, perception of brightness and contrast, visual illusions, etc. But the "real" mystery of visual perception remains comparatively unfathomed, or at least relegated to philosophical status: Why it is that we can see so well with what is apparently such a badly constructed visual apparatus? In this paper I will discuss several defects of vision and the classical theories of how they are overcome. I will criticize these theories and suggest an alternative approach, in which the outside world is considered as a kind of external memory store which can be accessed instantaneously by casting one's eyes (or one's attention) to some location. The feeling of the presence and extreme richness of the visual world is, under this view, a kind of illusion, created by the immediate availability of the information in this external store.     

Classification objects, ideal observers & generative models

A successful vision system must solve the problem of deriving geometrical information about three-dimensional objects from two-dimensional photometric input. The human visual system solves this problem with remarkable efficiency, and one challenge in vision research is to understand how neural representations of objects are formed and what visual information is used to form these representations. Ideal observer analysis has demonstrated the advantages of studying vision from the perspective of explicit generative models and a specified visual task, which divides the causes of image variations into the separate categories of signal and noise. Classification image techniques estimate the visual information used in a task from the properties of “noise” images that interact most strongly with the task. Both ideal observer analysis and classification image techniques rely on the assumption of a generative model. We show here how the ability of the classification image approach to understand how an observer uses visual information can be improved by matching the type and dimensionality of the model to that of the neural representation or internal template being studied. Because image variation in real world object tasks can arise from both geometrical shape and photometric (illumination or material) changes, a realistic image generation process should model geometry as well as intensity. A simple example is used to demonstrate what we refer to as a “classification object” approach to studying three-dimensional object representations.     

Visual enhancement of touch and the bodily self

We experience our own body through both touch and vision. We further see that others’ bodies are similar to our own body, but we have no direct experience of touch on others’ bodies. Therefore, relations between vision and touch are important for the sense of self and for mental representation of one’s own body. For example, seeing the hand improves tactile acuity on the hand, compared to seeing a non-hand object. While several studies have demonstrated this visual enhancement of touch (VET) effect, its relation to the ‘bodily self’, or mental representation of one’s own body remains unclear. We examined whether VET is an effect of seeing a hand, or of seeing my hand, using the rubber hand illusion. In this illusion, a prosthetic hand which is brushed synchronously—but not asynchronously—with one’s own hand is felt to actually be one’s hand. Thus, we manipulated whether or not participants felt like they were looking directly at their hand, while holding the actual stimulus they viewed constant. Tactile acuity was measured by having participants judge the orientation of square-wave gratings. Two characteristic effects of VET were observed: (1) cross-modal enhancement from seeing the hand was inversely related to overall tactile acuity, and (2) participants near sensory threshold showed significant improvement following synchronous stroking, compared to asynchronous stroking or no stroking at all. These results demonstrate a clear functional relation between the bodily self and basic tactile perception.     

Sensorimotor expectations and the visual field

Sensorimotor expectations concern how visual experience covaries with bodily movement. Sensorimotor theorists argue from such expectations to the conclusion that the phenomenology of vision is constitutively embodied: objects within the visual field are experienced as 3-D because sensorimotor expectations partially constitute our experience of such objects. Critics argue that there are (at least) two ways to block the above inference: to explain how we visually experience objects as 3-D, one may appeal to such non-bodily factors as (1) expectations about movements of objects, not the perceiver, or to (2) the role of mental imagery in visual experience. But instead of using sensorimotor expectations to explain how objects are experienced within the visual field, we can instead use them to explain our experience of the visual field itself and, in particular, our experience of its limits; that is, our ever-present visual sense of there being more to see, beyond what’s currently within the visual field. Crucially, this inference from sensorimotor expectations to the constitutive embodiment of visual phenomenology is not threatened by the above two challenges. I thus present here a sensorimotor theory of the phenomenology of the visual field, that is, our experience of our visual fields as such.

     

The influence of uncertainty and premovement visual information on manual aiming

Target-aiming studies in which premovement visual information is manipulated suggest that when vision is occluded, a brief visual representation of the target environment may be used to guide movement. The purpose of this work was to determine if the internal representation contains information about the whole movement environment or just specific information about the position of a single target goal. Two experiments were conducted in which we manipulated both target uncertainty and the visual information available before and during a target-aiming movement. Radial error differences between visual conditions and the independence of the vision and uncertainty manipulations support the hypothesis that subjects form a representation of the overall movement environment.     

Words,shape, visual search and visual working memory in 3‐year‐old children

Do words cue children's visual attention, and if so, what are the relevant mechanisms? Across four experiments, 3‐year‐old children (N = 163) were tested in visual search tasks in which targets were cued with only a visual preview versus a visual preview and a spoken name. The experiments were designed to determine whether labels facilitated search times and to examine one route through which labels could have their effect: By influencing the visual working memory representation of the target. The targets and distractors were pictures of instances of basic‐level known categories and the labels were the common name for the target category. We predicted that the label would enhance the visual working memory representation of the target object, guiding attention to objects that better matched the target representation. Experiments 1 and 2 used conjunctive search tasks, and Experiment 3 varied shape discriminability between targets and distractors. Experiment 4 compared the effects of labels to repeated presentations of the visual target, which should also influence the working memory representation of the target. The overall pattern fits contemporary theories of how the contents of visual working memory interact with visual search and attention, and shows that even in very young children heard words affect the processing of visual information.     

Evolutionary Constraints on Human Object Perception

Language and culture endow humans with access to conceptual information that far exceeds any which could be accessed by a non‐human animal. Yet, it is possible that, even without language or specific experiences, non‐human animals represent and infer some aspects of similarity relations between objects in the same way as humans. Here, we show that monkeys’ discrimination sensitivity when identifying images of animals is predicted by established measures of semantic similarity derived from human conceptual judgments. We used metrics from computer vision and computational neuroscience to show that monkeys’ and humans’ performance cannot be explained by low‐level visual similarity alone. The results demonstrate that at least some of the underlying structure of object representations in humans is shared with non‐human primates, at an abstract level that extends beyond low‐level visual similarity. Because the monkeys had no experience with the objects we tested, the results suggest that monkeys and humans share a primitive representation of object similarity that is independent of formal knowledge and cultural experience, and likely derived from common evolutionary constraints on object representation.     

Spatial representation by blind and sighted children

Spatial representation by 72 blind and blindfolded sighted children between the ages of 6 and 11 was tested in two experiments by mental rotation of a raised line under conditions of clockwise varied directions.Experiment 1 showed that the two groups were well matched on tactual recognition and scored equally badly on matching displays to their own mentally rotated position.Experiment 2 found the sighted superior in recall tests. There was a highly significant interaction between sighted status and degree of rotation. Degree of rotation affected only the blind. Their scores were significantly lower for rotating to oblique and to the far orthogonal directions than to near orthogonal test positions. On near orthogonals the blind did not differ from the sighted.Age was a main effect, but it did not interact with any other variable. Older blind children whose visual experience dated from before the age of 6 were superior to congenitally blind subjects, but not differentially more so on oblique directions.The results were discussed in relation to hypotheses about the nature of spatial representation and strategies by children whose prior experience derived from vision or from touch and movement.     

On the mental representations originating during the interaction between language and vision

The interaction between vision and language processing is clearly of interest to both cognitive psychologists and psycholinguists. Recent research has begun to create understanding of the interaction between vision and language in terms of the representational issues involved. In this paper, we first review some of the theoretical and methodological issues in the current vision–language interaction debate. Later, we develop a model that attempts to account for effects of affordances and visual context on language-scene interaction as well as the role of sensorimotor simulation. The paper addresses theoretical issues related to the mental representations that arise when visual and linguistic systems interact.     

See what you’ve done! Active touch affects the number of perceived visual objects

Previous research has shown that visual perception is affected by sensory information from other modalities. For example, sound can alter the visual intensity or the number of visual objects perceived. However, when touch and vision are combined, vision normally dominates—a phenomenon known asvisual capture. Here we report a cross-modal interaction between active touch and vision: The perceived number of brief visual events (flashes) is affected by the number of concurrently performed finger movements (keypresses). This sensorimotor illusion occurred despite little ambiguity in the visual stimuli themselves and depended on a close temporal proximity between movement execution and vision.     

The topography of training-induced visual field recovery: Perimetric maps and subjective representations

The cognitive representation of blind regions varies considerably between patients with vision loss and may influence compensatory behaviour and treatment motivation. We therefore measured “objective” visual field topography (perimetry) in 19 patients with postgeniculate visual system lesions and related this to the subjective scotoma representation as expressed by patients’ drawings of the defect and monitored changes of these measures during training-induced recovery of function. Blind regions were mostly adequately represented; however, central regions were overestimated and peripheral areas underestimated in size. Perimetric and subjective defect size decreased significantly during training. Again, training-induced visual field border shifts in central regions were larger in subjective than in perimetric maps but vice versa in the peripheral field. Thus, vision restoration therapy improves “objective” visual field size along with its cognitive representation. The subjective topography is shaped by the functional importance of visual field regions and is a function of cortical magnification, thus resembling the neural representation in visual cortex.     

3-D vision and figure-ground separation by visual cortex

A neural network theory of three-dimensional (3-D) vision, called FACADE theory, is described. The theory proposes a solution of the classical figure-ground problem for biological vision. It does so by suggesting how boundary representations and surface representations are formed within a boundary contour system (BCS) and a feature contour system (FCS). The BCS and FCS interact reciprocally to form 3-D boundary and surface representations that are mutually consistent. Their interactions generate 3-D percepts wherein occluding and occluded object parts are separated, completed, and grouped. The theory clarifies how preattentive processes of 3-D perception and figure-ground separation interact reciprocally with attentive processes of spatial localization, object recognition, and visual search. A new theory of stereopsis is proposed that predicts how cells sensitive to multiple spatial frequencies, disparities, and orientations are combined by context-sensitive filtering, competition, and cooperation to form coherent BCS boundary segmentations. Several factors contribute to figure-ground pop-out, including: boundary contrast between spatially contiguous boundaries, whether due to scenic differences in luminance, color, spatial frequency, or disparity-partially ordered interactions from larger spatial scales and disparities to smaller scales and disparities; and surface filling-in restricted to regions surrounded by a connected boundary. Phenomena such as 3-D pop-out from a 2-D picture, Da Vinci stereopsis, 3-D neon color spreading, completion of partially occluded objects, and figure-ground reversals are analyzed. The BCS and FCS subsystems model aspects of how the two parvocellular cortical processing streams that join the lateral geniculate nucleus to prestriate cortical area V4 interact to generate a multiplexed representation of Form-And-Color-And-DEpth, orfacade, within area V4. Area V4 is suggested to support figure-ground separation and to interact with cortical mechanisms of spatial attention, attentive object learning, and visual search. Adaptive resonance theory (ART) mechanisms model aspects of how prestriate visual cortex interacts reciprocally with a visual object recognition system in inferotemporal (IT) cortex for purposes of attentive object learning and categorization. Object attention mechanisms of the What cortical processing stream through IT cortex are distinguished from spatial attention mechanisms of the Where cortical processing stream through parietal cortex. Parvocellular BCS and FCS signals interact with the model What stream. Parvocellular FCS and magnocellular motion BCS signals interact with the model Where stream. Reciprocal interactions between these visual, What, and Where mechanisms are used to discuss data about visual search and saccadic eye movements, including fast search of conjunctive targets, search of 3-D surfaces, selective search of like-colored targets, attentive tracking of multielement groupings, and recursive search of simultaneously presented targets.     

Postattentive vision

Much research has examined preattentive vision: visual representation prior to the arrival of attention. Most vision research concerns attended visual stimuli; very little research has considered postattentive vision. What is the visual representation of a previously attended object once attention is deployed elsewhere? The authors argue that perceptual effects of attention vanish once attention is redeployed. Experiments 1-6 were visual search studies. In standard search, participants looked for a target item among distractor items. On each trial, a new search display was presented. These tasks were compared to repeated search tasks in which the search display was not changed. On successive trials, participants searched the same display for new targets. Results showed that if search was inefficient when participants searched a display the first time, it was inefficient when the same, unchanging display was searched the second, fifth, or 350th time. Experiments 7 and 8 made a similar point with a curve tracing paradigm. The results have implications for an understanding of scene perception, change detection, and the relationship of vision to memory.     

The representation and perception of 3D space: Interactions between 2D location and depth

ABSTRACT

We live in a 3D world, and yet the majority of vision research is restricted to 2D phenomena, with depth research typically treated as a separate field. Here we ask whether 2D spatial information and depth information interact to form neural representations of 3D space, and if so, what are the perceptual implications? Using fMRI and behavioural methods, we reveal that human visual cortex gradually transitions from 2D to 3D spatial representations, with depth information emerging later along the visual hierarchy, and demonstrate that 2D location holds a fundamentally special place in early visual processing.     

Development of interactions between sensorimotor representations in school-aged children

Reliable sensory-motor integration is a pre-requisite for optimal movement control; the functionality of this integration changes during development. Previous research has shown that motor performance of school-age children is characterized by higher variability, particularly under conditions where vision is not available, and movement planning and control is largely based on kinesthetic input. The purpose of the current study was to determine the characteristics of how kinesthetic-motor internal representations interact with visuo-motor representations during development. To this end, we induced a visuo-motor adaptation in 59 children, ranging from 5 to 12 years of age, as well as in a group of adults, and measured initial directional error (IDE) and endpoint error (EPE) during a subsequent condition where visual feedback was not available, and participants had to rely on kinesthetic input. Our results show that older children (age range 9–12 years) de-adapted significantly more than younger children (age range 5–8 years) over the course of 36 trials in the absence of vision, suggesting that the kinesthetic-motor internal representation in the older children was utilized more efficiently to guide hand movements, and was comparable to the performance of the adults.     

Through the inverting glass: first-person observations on spatial vision and imagery

Experience with inverting glasses reveals key factors of spatial vision. Interpretations of the literature based on the metaphor of a “visual image” have raised the question whether visual experience with inverting glasses remains inverted or whether it may turn back to normal after adaptation to the glasses. Here, I report on my experience with left/right inverting glasses and argue that a more fine-grained sensorimotor analysis can resolve the issue. Crucially, inverting glasses introduce a conflict at the very heart of spatial vision. At first, the experience of visual direction grounded in head movements differs from visual experience grounded in eye movements. During adaptation, this difference disappears, and one may learn to see without conflict where objects are located (this took me 123 h of practice). The momentary experience became once again integrated within the larger flow of visual exploration involving head movements, a change of experience that was abrupt and comparable to a Gestalt switch. The resulting experience remains different from normal vision, and I argue that this difference can be understood in sensorimotor terms. I describe how adaptation to inverting glasses is further reflected in mental imagery, supporting the idea that imagery is grounded in sensorimotor engagement with the environment as well.     

Varieties of cognitive penetration in visual perception

Is our perceptual experience a veridical representation of the world or is it a product of our beliefs and past experiences? Cognitive penetration describes the influence of higher level cognitive factors on perceptual experience and has been a debated topic in philosophy of mind and cognitive science. Here, we focus on visual perception, particularly early vision, and how it is affected by contextual expectations and memorized cognitive contents. We argue for cognitive penetration based on recent empirical evidence demonstrating contextual and top-down influences on early visual processes. On the basis of a perceptual model, we propose different types of cognitive penetration depending on the processing level on which the penetration happens and depending on where the penetrating influence comes from. Our proposal has two consequences: (1) the traditional controversy on whether cognitive penetration occurs or not is ill posed, and (2) a clear-cut perception–cognition boundary cannot be maintained.     

Smelling lessons

Much of the philosophical work on perception has focused on vision. Recently, however, philosophers have begun to correct this ‘tunnel vision’ by considering other modalities. Nevertheless, relatively little has been written about the chemical senses—olfaction and gustation. The focus of this paper is olfaction. In this paper, I consider the question: does human olfactory experience represents objects as thus and so? If we take visual experience as the paradigm of how experience can achieve object representation, we might think that the answer to this question is no. I argue that olfactory experience does indeed represent objects—just not in a way that is easily read from the dominant visual case.