Object interpolation in three dimensions

Perception of objects in ordinary scenes requires interpolation processes connecting visible areas across spatial gaps. Most research has focused on 2-D displays, and models have been based on 2-D, orientation-sensitive units. The authors present a view of interpolation processes as intrinsically 3-D and producing representations of contours and surfaces spanning all 3 spatial dimensions. The authors propose a theory of 3-D relatability that indicates for a given edge which orientations and positions of other edges in 3 dimensions may be connected to it, and they summarize the empirical evidence for 3-D relatability. The theory unifies and illuminates a number of fundamental issues in object formation, including the identity hypothesis in visual completion, the relations of contour and surface processes, and the separation of local and global processing. The authors suggest that 3-D interpolation and 3-D relatability have major implications for computational and neural models of object perception.     

Interpolation processes in object perception: reply to Anderson (2007)

P. J. Kellman, P. Garrigan, & T. F. Shipley presented a theory of 3-D interpolation in object perception. Along with results from many researchers, this work supports an emerging picture of how the visual system connects separate visible fragments to form objects. In his commentary, B. L. Anderson challenges parts of that view, especially the idea of a common underlying interpolation component in modal and amodal completion (the identity hypothesis). Here the authors analyze Anderson's evidence and argue that he neither provides any reason to abandon the identity hypothesis nor offers a viable alternative theory. The authors offer demonstrations and analyses indicating that interpolated contours can appear modally despite absence of the luminance relations, occlusion geometry, and surface attachment that Anderson claims to be necessary. The authors elaborate crossing interpolations as key cases in which modal and amodal appearance must be consequences of interpolation. Finally, the authors dispute Anderson's assertion that vision researchers are misguided in using objective performance methods, and they argue that his challenges to relatability fail because contour and surface processes, as well as local and global influences, have been distinguished experimentally.     

Mom’s shadow: structure-from-motion in newly hatched chicks as revealed by an imprinting procedure

The ability to recognize three-dimensional objects from two-dimensional (2-D) displays was investigated in domestic chicks, focusing on the role of the object’s motion. In Experiment 1 newly hatched chicks, imprinted on a three-dimensional (3-D) object, were allowed to choose between the shadows of the familiar object and of an object never seen before. In Experiments 2 and 3 random-dot displays were used to produce the perception of a solid shape only when set in motion. Overall, the results showed that domestic chicks were able to recognize familiar shapes from 2-D motion stimuli. It is likely that similar general mechanisms underlying the perception of structure-from-motion and the extraction of 3-D information are shared by humans and animals. The present data shows that they occur similarly in birds as known for mammals, two separate vertebrate classes; this possibly indicates a common phylogenetic origin of these processes.     

3-D shape perception

In this paper, we analyze and test three theories of 3-D shape perception: (1) Helmholtzian theory, which assumes that perception of the shape of an object involves reconstructing Euclidean structure of the object (up to size scaling) from the object’s retinal image after taking into account the object’s orientation relative to the observer, (2) Gibsonian theory, which assumes that shape perception involves invariants (projective or affine) computed directly from the object’s retinal image, and (3) perspective invariants theory, which assumes that shape perception involves a new kind of invariants of perspective transformation. Predictions of these three theories were tested in four experiments. In the first experiment, we showed that reliable discrimination between a perspective and nonperspective image of a random polygon is possible even when information only about the contour of the image is present. In the second experiment, we showed that discrimination performance did not benefit from the presence of a textured surface, providing information about the 3-D orientation of the polygon, and that the subjects could not reliably discriminate between the 3-D orientation of the textured surface and that of a shape. In the third experiment, we compared discrimination for solid shapes that either had flat contours (cuboids) or did not have visible flat contours (cylinders). The discrimination was very reliable in the case of cuboids but not in the case of cylinders. In the fourth experiment, we tested the effectiveness of planar motion in perception of distances and showed that the discrimination threshold was large and similar to thresholds when other cues to 3-D orientation were used. All these results support perspective invariants as a model of 3-D shape perception.     

Does face recognition rely on encoding of 3-D surface? Examining the role of shape-from-shading and shape-from-stereo

It is now well known that processing of shading information in face recognition is susceptible to bottom lighting and contrast reversal, an effect that may be due to a disruption of 3-D shape processing. The question then is whether the disruption can be rectified by other sources of 3-D information, such as shape-from-stereo. We examined this issue by comparing identification performance either with or without stereo information using top-lit and bottom-lit face stimuli in both photographic positive and negative conditions. The results show that none of the shading effects was reduced by the presence of stereo information. This finding supports the notion that shape-from-shading overrides shape-from-stereo in face perception. Although shape-from-stereo did produce some signs of facilitation for face identification, this effect was negligible. Together, our results support the view that 3-D shape processing plays only a minor role in face recognition. Our data are best accounted for by a weighted function of 2-D processing of shading pattern and 3-D processing of shapes, with a much greater weight assigned to 2-D pattern processing.     

Necessity of spatial pooling for the perception of heading in nonrigid environments

This study examined whether the perception of heading is determined by spatially pooling velocity information. Observers were presented displays simulating observer motion through a volume of 3-D objects. To test the importance of spatial pooling, the authors systematically varied the nonrigidity of the flow field using two types of object motion: adding a unique rotation or translation to each object. Calculations of the signal-to-noise (observer velocity-to-object motion) ratio indicated no decrements in performance when the ratio was .39 for object rotation and .45 for object translation. Performance also increased with the number of objects in the scene. These results suggest that heading is determined by mechanisms that use spatial pooling over large regions.     

3-D vision and figure-ground separation by visual cortex

A neural network theory of three-dimensional (3-D) vision, called FACADE theory, is described. The theory proposes a solution of the classical figure-ground problem for biological vision. It does so by suggesting how boundary representations and surface representations are formed within a boundary contour system (BCS) and a feature contour system (FCS). The BCS and FCS interact reciprocally to form 3-D boundary and surface representations that are mutually consistent. Their interactions generate 3-D percepts wherein occluding and occluded object parts are separated, completed, and grouped. The theory clarifies how preattentive processes of 3-D perception and figure-ground separation interact reciprocally with attentive processes of spatial localization, object recognition, and visual search. A new theory of stereopsis is proposed that predicts how cells sensitive to multiple spatial frequencies, disparities, and orientations are combined by context-sensitive filtering, competition, and cooperation to form coherent BCS boundary segmentations. Several factors contribute to figure-ground pop-out, including: boundary contrast between spatially contiguous boundaries, whether due to scenic differences in luminance, color, spatial frequency, or disparity-partially ordered interactions from larger spatial scales and disparities to smaller scales and disparities; and surface filling-in restricted to regions surrounded by a connected boundary. Phenomena such as 3-D pop-out from a 2-D picture, Da Vinci stereopsis, 3-D neon color spreading, completion of partially occluded objects, and figure-ground reversals are analyzed. The BCS and FCS subsystems model aspects of how the two parvocellular cortical processing streams that join the lateral geniculate nucleus to prestriate cortical area V4 interact to generate a multiplexed representation of Form-And-Color-And-DEpth, orfacade, within area V4. Area V4 is suggested to support figure-ground separation and to interact with cortical mechanisms of spatial attention, attentive object learning, and visual search. Adaptive resonance theory (ART) mechanisms model aspects of how prestriate visual cortex interacts reciprocally with a visual object recognition system in inferotemporal (IT) cortex for purposes of attentive object learning and categorization. Object attention mechanisms of the What cortical processing stream through IT cortex are distinguished from spatial attention mechanisms of the Where cortical processing stream through parietal cortex. Parvocellular BCS and FCS signals interact with the model What stream. Parvocellular FCS and magnocellular motion BCS signals interact with the model Where stream. Reciprocal interactions between these visual, What, and Where mechanisms are used to discuss data about visual search and saccadic eye movements, including fast search of conjunctive targets, search of 3-D surfaces, selective search of like-colored targets, attentive tracking of multielement groupings, and recursive search of simultaneously presented targets.     

Mutuality Relations,Observation, and Intentional Constraints

This commentary focuses on the implications of Stoffregen's (target article, this issue) theory, as they apply to current research on human biological motion. We take up his suggestion that affordances, not events, are perceived and that data generated within event-perception research may reflect conversion of affordance-based perception to "event-based scales." Research on point-light walkers has been classed with event perception; however, results from our current research on perception of point-light sports displays suggest that accurate detection of humans and their actions in these displays may be controlled by complex relations better explained within an affordance-based account. We report the results of an experiment that controlled the presence and absence of relations between biological motion and a discrete environmental object. Detection was best when these affordance-relevant relations were available. Finally, we consider the utility of Stoffregen's ontological distinction as it may inform our understanding of past, current, and future research on perception of point-light walker displays.     

Recovery of 3-D shape from binocular disparity and structure from motion

Four experiments were conducted to examine the integration of depth information from binocular stereopsis and structure from motion (SFM), using stereograms simulating transparent cylindrical objects. We found that the judged depth increased when either rotational or translational motion was added to a display, but the increase was greater for rotating (SFM) displays. Judged depth decreased as texture element density increased for static and translating stereo displays, but it stayed relatively constant for rotating displays. This result indicates that SFM may facilitate stereo processing by helping to resolve the stereo correspondence problem. Overall, the results from these experiments provide evidence for a cooperative relationship between. SFM and binocular disparity in the recovery of 3-D relationships from 2-D images. These findings indicate that the processing of depth information from SFM and binocular disparity is not strictly modular, and thus theories of combining visual information that assume strong modularity-or-independence cannot accurately characterize all instances of depth perception from multiple sources.     

Surface formation and depth in monocular scene perception

The visual perception of monocular stimuli perceived as 3-D objects has received considerable attention from researchers in human and machine vision. However, most previous research has focused on how individual 3-D objects are perceived. Here this is extended to a study of how the structure of 3-D scenes containing multiple, possibly disconnected objects and features is perceived. Da Vinci stereopsis, stereo capture, and other surface formation and interpolation phenomena in stereopsis and structure-from-motion suggest that small features having ambiguous depth may be assigned depth by interpolation with features having unambiguous depth. I investigated whether vision may use similar mechanisms to assign relative depth to multiple objects and features in sparse monocular images, such as line drawings, especially when other depth cues are absent. I propose that vision tends to organize disconnected objects and features into common surfaces to construct 3-D-scene interpretations. Interpolations that are too weak to generate a visible surface percept may still be strong enough to assign relative depth to objects within a scene. When there exists more than one possible surface interpolation in a scene, the visual system's preference for one interpolation over another seems to be influenced by a number of factors, including: (i) proximity, (ii) smoothness, (iii) a preference for roughly frontoparallel surfaces and 'ground' surfaces, (iv) attention and fixation, and (v) higher-level factors. I present a variety of demonstrations and an experiment to support this surface-formation hypothesis.     

Development of three-dimensional form perception

In three experiments with infants and one with adults we explored the generality, limitations, and informational bases of early form perception. In the infant studies we used a habituation-of-looking-time procedure and the method of Kellman (1984), in which responses to three-dimensional (3-D) form were isolated by habituating 16-week-old subjects to a single object in two different axes of rotation in depth, and testing afterward for dishabituation to the same object and to a different object in a novel axis of rotation. In Experiment 1, continuous optical transformations given by moving 16-week-old observers around a stationary 3-D object specified 3-D form to infants. In Experiment 2 we found no evidence of 3-D form perception from multiple, stationary, binocular views of objects by 16- and 24-week-olds. Experiment 3A indicated that perspective transformations of the bounding contours of an object, apart from surface information, can specify form at 16 weeks. Experiment 3B provided a methodological check, showing that adult subjects could neither perceive 3-D forms from the static views of the objects in Experiment 3A nor match views of either object across different rotations by proximal stimulus similarities. The results identify continuous perspective transformations, given by object or observer movement, as the informational bases of early 3-D form perception. Detecting form in stationary views appears to be a later developmental acquisition.     

Discrimination of 3-D shape and 3-D curvature from motion in active vision

We examined the ability of human observers to discriminate between different 3-D quadratic surfaces defined by motion, and with head position fed back to the stimulus to provide an up-to-date dynamical perspective view. We tested whether 3-D shape or 3-D curvature would affect discrimination performance. It appeared that discrimination of 3-D quadratic shape clearly depended on shape but not on the amount of curvature. Even when the amount of curvature was randomized, subjects’ performance was not altered. On the other hand, the discrimination of 3-D curvature clearly depended linearly on curvature with Weber fractions of 20% on the average and, to a small degree, on 3-D shape. The experiment shows that observers can easily separate 3-D shape and 3-D curvature, and that Koenderink’s shape index and curvedness provide a convenient way to specify shape. These results warn us against using just any arbitrary 3-D shape in 3-D shape perception tasks and indicate, for example, that emphasizing 3-D shape in computer displays by exaggerating curvature does not have any effect.     

Object-based warping: an illusory distortion of space within objects

Visual objects are high-level primitives that are fundamental to numerous perceptual functions, such as guidance of attention. We report that objects warp visual perception of space in such a way that spatial distances within objects appear to be larger than spatial distances in ground regions. When two dots were placed inside a rectangular object, they appeared farther apart from one another than two dots with identical spacing outside of the object. To investigate whether this effect was object based, we measured the distortion while manipulating the structure surrounding the dots. Object displays were constructed with a single object, multiple objects, a partially occluded object, and an illusory object. Nonobject displays were constructed to be comparable to object displays in low-level visual attributes. In all cases, the object displays resulted in a more powerful distortion of spatial perception than comparable non-object-based displays. These results suggest that perception of space within objects is warped.     

A theory of visual interpolation in object perception

We describe a new theory explaining the perception of partly occluded objects and illusory figures, from both static and kinematic information, in a unified framework. Three ideas guide our approach. First, perception of partly occluded objects, perception of illusory figures, and some other object perception phenomena derive from a single boundary interpolation process. These phenomena differ only in respects that are not part of the unit formation process, such as the depth placement of units formed. Second, unit formation from static and kinematic information can be treated in the same general framework. Third, spatial and spatiotemporal discontinuities in the boundaries of optically projected areas are fundamental to the unit formation process. Consistent with these ideas, we develop a detailed theory of unit formation that accounts for most cases of boundary perception in the absence of local physical specification. According to this theory, discontinuities in the first derivative of projected edges are initiating conditions for unit formation. A formal notion of relatability is defined, specifying which physically given edges leading into discontinuities can be connected to others by interpolated edges. Intuitively, relatability requires that two edges be connectable by a smooth, monotonic curve. The roots of the discontinuity and relatability notions in ecological constraints on object perception are discussed. Finally, we elaborate our approach by discussing related issues, some new phenomena, connections to other approaches, and issues for future research.     

The perception of lightness in 3-D curved objects

Lightness constancy in complex scenes requires that the visual system take account of information concerning variations of illumination falling on visible surfaces. Three experiments on the perception of lightness for three-dimensional (3-D) curved objects show that human observers are better able to perform this accounting for certain scenes than for others. The experiments investigate the effect of object curvature, illumination direction, and object shape on lightness perception. Lightness constancy was quite good when a rich local gray-level context was provided. Deviations occurred when both illumination and reflectance changed along the surface of the objects. Does the perception of a 3-D surface and illuminant layout help calibrate lightness judgments? Our results showed a small but consistent improvement between lightness matches on ellipsoid shapes, relative to flat rectangle shapes, under illumination conditions that produce similar image gradients. Illumination change over 3-D forms is therefore taken into account in lightness perception.     

Effects of 3-D complexity on the perception of 2-D depictions of objects

In a recent study, Pelli (1999 Science 285 844-846) performed a set of perceptual experiments using portrait paintings by Chuck Close. Close's work is similar to the 'Lincoln' portraits of Harmon and Julesz (1973 Science 180 1194-1197) in that they are composite images consisting of coarsely sampled, individually painted, mostly homogeneous cells. Pelli showed that perceived shape was dependent on size, refuting findings that perception of this type is scale-invariant. In an attempt to broaden this finding we designed a series of experiments to investigate the interaction of 2-D scale and 3-D structure on our perception of 3-D shape. We present a series of experiments where field of view, 3-D object complexity, 2-D image resolution, viewing orientation, and subject matter of the stimulus are manipulated. On each trial, observers indicated if the depicted objects appeared to be 2-D or 3-D. Results for face stimuli are similar to Pelli's, while more geometrically complex stimuli show a further interaction of the 3-D information with distance and image information. Complex objects need more image information to be seen as 3-D when close; however, as they are moved further away from the observer, there is a bias for seeing them as 3-D objects rather than 2-D images. Finally, image orientation, relative to the observer, shows little effect, suggesting the participation of higher-level processes in the determination of the 'solidness' of the depicted object. Thus, we show that the critical image resolution depends systematically on the geometric complexity of the object depicted.     

Wayfinding: The effects of large displays and 3-D perception

Large displays and stereopsis have been shown to improve performance in several virtual navigation tasks. In the present research, we sought to determine whether wayfinding could benefit from these factors. Participants were tested in a virtual town. There were three viewing conditions: a desktop, a large screen, and a large screen on which the virtual environment was viewed in three dimensions (3-D) using polarized glasses. Participants explored the town and had to remember the location of several landmarks. Their memory of the layout of the town was tested by asking them to navigate from one landmark to another, taking the shortest route possible. All groups performed equally well in terms of the distance traveled to target locations. From this result, we concluded that large displays and 3-D perception do not significantly contribute to wayfinding. Thus, experimental paradigms and training programs that utilize wayfinding are as valuable when administered on standard desktops as on more sophisticated and costly equipment and do not induce simulator sickness as large displays tend to do.     

Strength of visual interpolation depends on the ratio of physically specified to total edge length.

We report four experiments in which the strength of edge interpolation in illusory figure displays was tested. In Experiment 1, we investigated the relative contributions of the lengths of luminance-specified edges and the gaps between them to perceived boundary clarity as measured by using a magnitude estimation procedure. The contributions of these variables were found to be best characterized by a ratio of the length of luminance-specified contour to the length of the entire edge (specified plus interpolated edge). Experiment 2 showed that this ratio predicts boundary clarity for a wide range of ratio values and display sizes. There was no evidence that illusory figure boundaries are clearer in displays with small gaps than they are in displays with larger gaps and equivalent ratios. In Experiment 3, using a more sensitive pairwise comparison paradigm, we again found no such effect. Implications for boundary interpolation in general, including perception of partially occluded objects, are discussed. The dependence of interpolation on the ratio of physically specified edges to total edge length has the desirable ecological consequence that unit formation will not change with variations in viewing distance.     

Overestimation of heights in virtual reality is influenced more by perceived distal size than by the 2-D versus 3-D dimensionality of the display

One important aspect of the pictorial representation of a scene is the depiction of object proportions. Yang, Dixon, and Proffitt (1999 Perception 28 445-467) recently reported that the magnitude of the vertical-horizontal illusion was greater for vertical extents presented in three-dimensional (3-D) environments compared to two-dimensional (2-D) displays. However, because all of the 3-D environments were large and all of the 2-D displays were small, the question remains whether the observed magnitude differences were due solely to the dimensionality of the displays (2-D versus 3-D) or to the perceived distal size of the extents (small versus large). We investigated this question by comparing observers' judgments of vertical relative to horizontal extents on a large but 2-D display compared to the large 3-D and the small 2-D displays used by Yang et al (1999). The results confirmed that the magnitude differences for vertical overestimation between display media are influenced more by the perceived distal object size rather than by the dimensionality of the display.     

Strength of visual interpolation depends on the ratio of physically specified to total edge length

We report four experiments in which the strength ofedge-interpoiat-ion in illusory figure displays was tested. In Experiment 1, we investigated the relative contributions of the lengths of luminance-specified edges and the gaps between them to perceived boundary clarity as measured by using a magnitude estimation procedure. The contributionaoLthese variables were found to be best characterized by a ratio of the length of luminance-specified contour to the length of the entire edge (specified plus interpolated edge). Experiment 2 showed that this ratio predicts boundary clarity for a wide range of ratio values and display sizes.There was no evidence that illusory figure boundaries are clearer in displays with small gaps than they are in displays with larger gaps and equivalent ratios. In Experiment 3, using a more sensitive pairwise comparison paradigm, we again found no such effect. Implications for boundary interpolation in general, including perception of partially occluded objects, are discussed. The dependence of interpolation on the ratio of physically specified edges to total edgelength has thedesirable eeological consequence that unit formation will not change with variations in viewing distance.