A sensorimotor account of vision and visual consciousness

Many current neurophysiological, psychophysical, and psychological approaches to vision rest on the idea that when we see, the brain produces an internal representation of the world. The activation of this internal representation is assumed to give rise to the experience of seeing. The problem with this kind of approach is that it leaves unexplained how the existence of such a detailed internal representation might produce visual consciousness. An alternative proposal is made here. We propose that seeing is a way of acting. It is a particular way of exploring the environment. Activity in internal representations does not generate the experience of seeing. The outside world serves as its own, external, representation. The experience of seeing occurs when the organism masters what we call the governing laws of sensorimotor contingency. The advantage of this approach is that it provides a natural and principled way of accounting for visual consciousness, and for the differences in the perceived quality of sensory experience in the different sensory modalities. Several lines of empirical evidence are brought forward in support of the theory, in particular: evidence from experiments in sensorimotor adaptation, visual "filling in," visual stability despite eye movements, change blindness, sensory substitution, and color perception.     

Top-down control of visual perception: attention in natural vision

Top-down perceptual influences can bias (or pre-empt) perception. In natural scenes, the receptive fields of neurons in the inferior temporal visual cortex (IT) shrink to become close to the size of objects. This facilitates the read-out of information from the ventral visual system, because the information is primarily about the object at the fovea. Top-down attentional influences are much less evident in natural scenes than when objects are shown against blank backgrounds, though are still present. It is suggested that the reduced receptive-field size in natural scenes, and the effects of top-down attention contribute to change blindness. The receptive fields of IT neurons in complex scenes, though including the fovea, are frequently asymmetric around the fovea, and it is proposed that this is the solution the IT uses to represent multiple objects and their relative spatial positions in a scene. Networks that implement probabilistic decision-making are described, and it is suggested that, when in perceptual systems they take decisions (or 'test hypotheses'), they influence lower-level networks to bias visual perception. Finally, it is shown that similar processes extend to systems involved in the processing of emotion-provoking sensory stimuli, in that word-level cognitive states provide top-down biasing that reaches as far down as the orbitofrontal cortex, where, at the first stage of affective representations, olfactory, taste, flavour, and touch processing is biased (or pre-empted) in humans.     

Position of code and code for position: From isomorphism to a sensorimotor account of space perception

The paper starts with a discussion of the assumption that positions of the outer world are coded by the anatomical locations of firing neurons within retinotopic maps (“coding position by position”). This “code position theory” explains space perception by some kind of structural isomorphism since it implies that the perceptual space is based on a spatial structure within the brain. The axiom of structural isomorphism is rejected. Subsequently, a sensorimotor account of space perception is outlined according to which the spatial structure of the outer world is coded by the temporal structure of cortical processing. The basis is that action changes the perceiver's relationship to the outer world and, therefore, changes the representation of the outer world coded by the sensory responses of the brain. According to this view the code for position is not a spatial but a temporal structure resulting from action (“coding position by action”). The sensorimotor account offers a possible solution to the binding problem. The paper ends with some remarks on the possible origin and function of retinotopic representations.     

Unawareness of visual and sensorimotor defects: A hypothesis

A theory is proposed to account for unawareness of blindness, hemianopsia, and hemiplegia, and for phantom limb after amputation. It is assumed that interruption of a sensory pathway at any level--from peripheral nerve to primary sensory cortex--is not associated with any immediate sensory experience that uniquely specifies the defect. Instead the sensory loss must be discovered by a process of self-observation and inference. Discovery is easy for defects that create major functional disability, such as total blindness. Hence unawareness of total blindness occurs only in association with severe intellectual impairment, precluding the required self-observation and inference. In contrast, hemianopsia is difficult to discover because several mechanisms automatically compensate the defect effectively. Thus unawareness of hemianopsia is common, even in intellectually normal individuals. Insensate fields are often the source of suggested (false) percepts, because no information from such a field specifies the absence of a sensory stimulus. The most powerful source of suggestion is sensory activity in uninvolved portions of the affected sensory field. Thus hemianopsics may perceive complete geometric forms when only incomplete forms are shown and the missing portion falls in the hemianopsic fields. Such perceptual completion also occurs in hemianesthetic hemiplegics, creating the illusion that there are normally functioning limbs on the affected side. This perceptual completion increases the difficulty of discovery of hemianesthetic hemiplegia, but the disability is still sufficiently obvious that some additional cognitive impairment is invariably present in patients with lasting unawareness of hemiplegia. Phantom limb after amputation is the product of perceptual completion without associated cognitive impairment. The patient with phantom limb is thus aware of the illusory quality of his phantom. Some insight into the neural basis of perceptual completion and of unawareness of sensory loss may derive from considering sensory systems and associative cortex as parallel-distributed processing mechanisms.     

Sound enhances visual perception: cross-modal effects of auditory organization on vision

Six experiments demonstrated cross-modal influences from the auditory modality on the visual modality at an early level of perceptual organization. Participants had to detect a visual target in a rapidly changing sequence of visual distractors. A high tone embedded in a sequence of low tones improved detection of a synchronously presented visual target (Experiment 1), but the effect disappeared when the high tone was presented before the target (Experiment 2). Rhythmically based or order-based anticipation was unlikely to account for the effect because the improvement was unaffected by whether there was jitter (Experiment 3) or a random number of distractors between successive targets (Experiment 4). The facilitatory effect was greatly reduced when the tone was less abrupt and part of a melody (Experiments 5 and 6). These results show that perceptual organization in the auditory modality can have an effect on perceptibility in the visual modality.     

When sound affects vision: effects of auditory grouping on visual motion perception

Two identical visual targets moving across each other can be perceived either to bounce off or to stream through each other. A brief sound at the moment the targets coincide biases perception toward bouncing. We found that this bounce-inducing effect was attenuated when other identical sounds (auditory flankers) were presented 300 ms before and after the simultaneous sound. The attenuation occurred only when the simultaneous sound and auditory flankers had similar acoustic characteristics and the simultaneous sound was not salient. These results suggest that there is an aspect of auditory-grouping (saliency-assigning) processes that is context-sensitive and can be utilized by the visual system for solving ambiguity. Furthermore, control experiments revealed that such auditory context did not affect the perceptual qualities of the simultaneous sound. Because the attenuation effect is not manifest in the perception of acoustic characteristics of individual sound elements, we conclude that it is a genuine cross-modal effect.     

A visual display system for reading and visual perception research

The selection of a computer visual display system suitable for word recognition and reading research is described. The software character generation routines permit flexible definition of character sets. The display software permits control of size scaling and point density of characters being displayed as well as control over the temporal microstructure of presenting and refreshing the displayed text.     

A matching-to-sample procedure for the study of visual perception and concept formation in infrahumans: A test in the pigeon

An apparatus suitable for conducting simultaneous or delayed matching-to-sample experiments with as many as eight comparison stimuli is described. Data are presented which indicate that pigeons are capable of performing such a task. It is concluded that the simultaneous matching-to-sample procedure with more than two comparison stimuli is an efficient means of obtaining information about discrimination capacity and that this procedure may be suitable for studying visual perception in infrahuman Ss.     

The visual perception coordinate system uses axes defined by the earth, trunk, and vision

Eight young adults adjusted a line located on one side of a computer display parallel to internally specified Earth-fixed vertical (display in frontal plane), to the horizontal trunk-fixed anterior-posterior axis (display in horizontal plane), and to an oblique line (display in horizontal and vertical planes). All tasks were completed in a dark room with the head and trunk in both a standard erect posture and varied postures. Errors were lowest when setting the line to internally specified vertical in the frontal plane and to an oblique line in the horizontal plane when head and trunk orientations were varied. Constant errors for setting one line parallel to a second line were in opposite directions when the second line was located on the left versus right side of the display, but did not differ in direction when setting the line parallel to internally specified axes. Also, the oblique effect was preserved when the head and trunk were tilted to various orientations, suggesting that it results from integration of an internally specified gravitational reference with visual input. We conclude that the visual perceptual coordinate system uses internally specified vertical and, when available, a visually specified horizontal reference axis to define object orientation.     

A unified account of categorical effects in phonetic perception

Categorical effects are found across speech sound categories, with the degree of these effects ranging from extremely strong categorical perception in consonants to nearly continuous perception in vowels. We show that both strong and weak categorical effects can be captured by a unified model. We treat speech perception as a statistical inference problem, assuming that listeners use their knowledge of categories as well as the acoustics of the signal to infer the intended productions of the speaker. Simulations show that the model provides close fits to empirical data, unifying past findings of categorical effects in consonants and vowels and capturing differences in the degree of categorical effects through a single parameter.     

Processing spatial layout by perception and sensorimotor interaction

Everyone has the feeling that perception is usually accurate - we apprehend the layout of the world without significant error, and therefore we can interact with it effectively. Several lines of experimentation, however, show that perceived layout is seldom accurate enough to account for the success of visually guided behaviour. A visual world that has more texture on one side, for example, induces a shift of the body's straight ahead to that side and a mislocalization of a small target to the opposite side. Motor interaction with the target remains accurate, however, as measured by a jab with the finger. Slopes of hills are overestimated, even while matching the slopes of the same hills with the forearm is more accurate. The discrepancy shrinks as the estimated range is reduced, until the two estimates are hardly discrepant for a segment of a slope within arm's reach. From an evolutionary standpoint, the function of perception is not to provide an accurate physical layout of the world, but to inform the planning of future behaviour. Illusions - inaccuracies in perception - are perceived as such only when they can be verified by objective means, such as measuring the slope of a hill, the range of a landmark, or the location of a target. Normally such illusions are not checked and are accepted as reality without contradiction.     

Impressions of force in visual perception of collision events: A test of the causal asymmetry hypothesis

When two objects interact they exert equal and opposite forces on each other. According to the causal asymmetry hypothesis, however, when one object has been identified as causal and the other as that in which the effect occurs, the causal object is perceived as exerting greater force on the effect object than the latter is perceived as exerting on the former. An example of this is a stimulus in which one object moves toward another stationary one, and when contact occurs the former stops and the latter moves away. In this situation the initially moving object is identified as causal, so the causal asymmetry hypothesis predicts that more force will be judged to be exerted by the moving object on the stationary one than by the stationary one on the moving one. Participants’ judgments consistently supported this hypothesis for a variety of stimuli in which kinematic parameters were varied, even when the initially moving object reversed direction after contact.     

A multi-stage account of binding in vision: Neuropsychological evidence

I review neuropsychological evidence on the problems patients can have in binding together the attributes of visual stimuli, following brain damage. The evidence indicates that there can be several kinds of binding deficit in patients. Damage to early visual processing within the ventral visual stream can disrupt the binding of contours into shapes, though the binding of form elements into contours can still operate. This suggests that the process of binding elements into contour is distinct from the process of binding contours into shapes. The latter form of binding seems to operate within the ventral visual system. In addition, damage to the parietal lobe can disrupt the binding of shape to surface information about objects, even when the binding of elements into contours, and contours into shapes, seems to be preserved. These findings are consistent with a multi-stage account of binding in vision, which distinguishes between the processes involved in binding shape information (in the ventral visual stream) and the processes involved in binding shape and surface detail (involving interactions between the ventral and dorsal streams). In addition, I present evidence indicating that a further, transient form of binding can take place, based on stimuli having common visual onsets. I discuss the relations between these different forms of binding.     

Eye movements and the integration of visual memory and visual perception

Because visual perception has temporal extent, temporally discontinuous input must be linked in memory. Recent research has suggested that this may be accomplished by integrating the active contents of visual short-term memory (VSTM) with subsequently perceived information. In the present experiments, we explored the relationship between VSTM consolidation and maintenance and eye movements, in order to discover how attention selects the information that is to be integrated. Specifically, we addressed whether stimuli needed to be overtly attended in order to be included in the memory representation or whether covert attention was sufficient. Results demonstrated that in static displays in which the to-be-integrated information was presented in the same spatial location, VSTM consolidation proceeded independently of the eyes, since subjects made few eye movements. In dynamic displays, however, in which the to-be-integrated information was presented in different spatial locations, eye movements were directly related to task performance. We conclude that these differences are related to different encoding strategies. In the static display case, VSTM was maintained in the same spatial location as that in which it was generated. This could apparently be accomplished with covert deployments of attention. In the dynamic case, however, VSTM was generated in a location that did not overlap with one of the to-be-integrated percepts. In order to "move" the memory trace, overt shifts of attention were required.     

A comparison of tactile and blurred visual form perception

Two experiments were performed to assess the similarity of tactile form perception and visual form perception under conditions of extreme visual blurring. In the first experiment, resolution and relative localization tasks were performed by five subjects under both tactile and blurred visual presentation. The results obtained here were the same for the two modalities. In the second experiment, all 26 block letters were presented to each of four subjects in two distinct methods of presentation, using both tactile and blurred visual displays. In one method, the full-field letter was flashed for 1.0 sec; in the other, a vertical slit scanned the letter from left to right. For all method comparisons, a strong similarity was found between the patterns of correct responses as a function of letter. In addition, there was a definite similarity between the two modalities in terms of which letters improved in recognizability in the change from the full-field to the slit mode of presentation. However, the overall superiority of the slit method found for tactile recognition was not obtained visually. The two experiments indicate that recognition with blurred vision is similar to recognition using the intact cutaneous sense, although some differences remain.