Incidental visual memory for objects in scenes

Three experiments were conducted to investigate the existence of incidentally acquired, long-term, detailed visual memory for objects embedded in previously viewed scenes. Participants performed intentional memorization and incidental visual search learning tasks while viewing photographs of real-world scenes. A visual memory test for previously viewed objects from these scenes then followed. Participants were not aware that they would be tested on the scenes following incidental learning in the visual search task. In two types of memory tests for visually specific object information (token discrimination and mirror-image discrimination), performance following both the memorization and visual search conditions was reliably above chance. These results indicate that recent demonstrations of good visual memory during scene viewing are not due to intentional scene memorization. Instead, long-term visual representations are incidentally generated as a natural product of scene perception.     

Eye movement control during scene viewing: Immediate degradation and enhancement effects of spatial frequency filtering

What controls how long the eyes remain fixated during scene perception? We investigated whether fixation durations are under the immediate control of the quality of the current scene image. Subjects freely viewed photographs of scenes in preparation for a later memory test while their eye movements were recorded. Using the saccade-contingent display change method, scenes were degraded (Experiment 1) or enhanced (Experiment 2) via blurring (low-pass filtering) during predefined saccades. Results showed that fixation durations immediately after a display change were influenced by the degree of blur, with a monotonic relationship between degree of blur and fixation duration. The results also demonstrated that fixation durations can be both increased and decreased by changes in the degree of blur. The results suggest that fixation durations in scene viewing are influenced by the ease of processing of the image currently in view. The results are consistent with models of saccade generation in scenes in which moment-to-moment difficulty in visual and cognitive processing modulates fixation durations.     

Meaning guides attention during scene viewing,even when it is irrelevant

During real-world scene viewing, humans must prioritize scene regions for attention. What are the roles of low-level image salience and high-level semantic meaning in attentional prioritization? A previous study suggested that when salience and meaning are directly contrasted in scene memorization and preference tasks, attentional priority is assigned by meaning (Henderson & Hayes in Nature Human Behavior, 1, 743–747, 2017). Here we examined the role of meaning in attentional guidance using two tasks in which meaning was irrelevant and salience was relevant: a brightness rating task and a brightness search task. Meaning was represented by meaning maps that captured the spatial distribution of semantic features. Meaning was contrasted with image salience, represented by saliency maps. Critically, both maps were represented similarly, allowing us to directly compare how meaning and salience influenced the spatial distribution of attention, as measured by fixation density maps. Our findings suggest that even in tasks for which meaning is irrelevant and salience is relevant, meaningful scene regions are prioritized for attention over salient scene regions. These results support theories in which scene semantics play a dominant role in attentional guidance in scenes.     

ICAT: a computational model for the adaptive control of fixation durations

Eye movements depend on cognitive processes related to visual information processing. Much has been learned about the spatial selection of fixation locations, while the principles governing the temporal control (fixation durations) are less clear. Here, we review current theories for the control of fixation durations in tasks like visual search, scanning, scene perception, and reading and propose a new model for the control of fixation durations. We distinguish two local principles from one global principle of control. First, an autonomous saccade timer initiates saccades after random time intervals (local-I). Second, foveal inhibition permits immediate prolongation of fixation durations by ongoing processing (local-II). Third, saccade timing is adaptive, so that the mean timer value depends on task requirements and fixation history (Global). We demonstrate by numerical simulations that our model qualitatively reproduces patterns of mean fixation durations and fixation duration distributions observed in typical experiments. When combined with assumptions of saccade target selection and oculomotor control, the model accounts for both temporal and spatial aspects of eye movement control in two versions of a visual search task. We conclude that the model provides a promising framework for the control of fixation durations in saccadic tasks.     

The limits of visual resolution in natural scene viewing

We examined the limits of visual resolution in natural scene viewing, using a gaze-contingent multiresolutional display having a gaze-centred area-of-interest and decreasing resolution with eccentricity. Twelve participants viewed high-resolution scenes in which gaze-contingent multiresolutional versions occasionally appeared for single fixations. Both detection of image degradation (five filtering levels plus a no-area-of-interest control) in the gaze-contingent multiresolutional display, and eye fixation durations, were well predicted by a model of eccentricity-dependent contrast sensitivity. The results also illuminate the time course of detecting image filtering. Detection did not occur for fixations below 100 ms, and reached asymptote for fixations above 200 ms. Detectable filtering lengthened fixation durations by 160 ms, and interference from an imminent manual response occurred by 400-450 ms, often lengthening the next fixation. We provide an estimate of the limits of visual resolution in natural scene viewing useful for theories of scene perception, and help bridge the literature on spatial vision and eye movement control.     

Spatial frequency filtering and the direct control of fixation durations during scene viewing

The present study employed a saccade-contingent change paradigm to investigate the effect of spatial frequency filtering on fixation durations during scene viewing. Subjects viewed grayscale scenes while encoding them for a later memory test. During randomly chosen saccades, the scene was replaced with an alternate version that remained throughout the critical fixation that followed. In Experiment 1, during the critical fixation, the scene could be changed to high-pass and low-pass spatial frequency filtered versions. Under both conditions, fixation durations increased, and the low-pass condition produced a greater effect than the high-pass condition. In subsequent experiments, we manipulated the familiarity of scene information during the critical fixation by flipping the filtered scenes upside down or horizontally. Under these conditions, we observed lengthening of fixation durations but no difference between the high-pass and low-pass conditions, suggesting that the filtering effect is related to the mismatch between information extracted within the critical fixation and the ongoing scene representation in memory. We also conducted control experiments that tested the effect of changes to scene orientation (Experiment 2a) and the addition of color to a grayscale scene (Experiment 2b). Fixation distribution analysis suggested two effects on the distribution fixation durations: a fast-acting effect that was sensitive to all transsaccadic changes tested and a later effect in the tail of the distribution that was likely tied to the processing of scene information. These findings are discussed in the context of theories of oculomotor control during scene viewing.     

Visual search of driving situations: danger and experience

Previous research on visual search in driving suffers from a number of problems: small sample sizes, a concentration on mundane situations, and a failure to link results to more general psychological theory. The study reported in this paper addresses these issues by recording the eye movements of a large sample of drivers while they watched films of dangerous driving situations and comparing the findings with those from more general studies on scene perception. Stimuli were classified according to the types of road shown and the degree of danger present in the scenes. Two groups of subjects took part, fifty-one young novice drivers who had just gained a full driving licence and twenty-six older more experienced drivers. Dangerous situations were characterised by a narrowing of visual search, shown by an increase in fixation durations, a decrease in saccade angular distances, and a reduction in the variance of fixation locations. These effects are similar to the concept of 'attention focusing' in traumatic situations as it is described in the literature on eyewitness memory. When road types are compared, the least visually complex rural roads attracted the longest fixation durations and the shortest angular saccade distances, while the most visually complex urban roads attracted the greatest spread of search but the shortest fixation durations. Differences between the groups of subjects were also present. Novices had longer fixation durations than experienced drivers, particularly in dangerous situations. Experienced drivers also fixated lower down and had less vertical variance in fixation locations than novices.     

Interrupted visual searches reveal volatile search memory

This study investigated memory from interrupted visual searches. Participants conducted a change detection search task on polygons overlaid on scenes. Search was interrupted by various disruptions, including unfilled delay, passive viewing of other scenes, and additional search on new displays. Results showed that performance was unaffected by short intervals of unfilled delay or passive viewing, but it was impaired by additional search tasks. Across delays, memory for the spatial layout of the polygons was retained for future use, but memory for polygon shapes, background scene, and absolute polygon locations was not. The authors suggest that spatial memory aids interrupted visual searches, but the use of this memory is easily disrupted by additional searches.     

The eye gaze direction of an observed person can bias perception,memory, and attention in adolescents with and without autism spectrum disorder

The reported experiments aimed to investigate whether a person and his or her gaze direction presented in the context of a naturalistic scene cause perception, memory, and attention to be biased in typically developing adolescents and high-functioning adolescents with autism spectrum disorder (ASD). A novel computerized image manipulation program presented a series of photographic scenes, each containing a person. The program enabled participants to laterally maneuver the scenes behind a static window, the borders of which partially occluded the scenes. The gaze direction of the person in the scenes spontaneously cued attention of both groups in the direction of gaze, affecting judgments of preference (Experiment 1a) and causing memory biases (Experiment 1b). Experiment 2 showed that the gaze direction of a person cues visual search accurately to the exact location of gaze in both groups. These findings suggest that biases in preference, memory, and attention are caused by another person’s gaze direction when viewed in a complex scene in adolescents with and without ASD     

Direct control of fixation times in scene viewing: Evidence from analysis of the distribution of first fixation duration

Participants' eye movements were monitored in two scene viewing experiments that manipulated the task-relevance of scene stimuli and their availability for extrafoveal processing. In both experiments, participants viewed arrays containing eight scenes drawn from two categories. The arrays of scenes were either viewed freely (Free Viewing) or in a gaze-contingent viewing mode where extrafoveal preview of the scenes was restricted (No Preview). In Experiment 1a, participants memorized the scenes from one category that was designated as relevant, and in Experiment 1b, participants chose their preferred scene from within the relevant category. We examined first fixations on scenes from the relevant category compared to the irrelevant category (Experiments 1a and 1b), and those on the chosen scene compared to other scenes not chosen within the relevant category (Experiment 1b). A survival analysis was used to estimate the first discernible influence of the task-relevance on the distribution of first-fixation durations. In the free viewing condition in Experiment 1a, the influence of task relevance occurred as early as 81 ms from the start of fixation. In contrast, the corresponding value in the no preview condition was 254 ms, demonstrating the crucial role of extrafoveal processing in enabling direct control of fixation durations in scene viewing. First fixation durations were also influenced by whether or not the scene was eventually chosen (Experiment 1b), but this effect occurred later and affected fewer fixations than the effect of scene category, indicating that the time course of scene processing is an important variable mediating direct control of fixation durations.     

Linked linear mixed models: A joint analysis of fixation locations and fixation durations in natural reading

The complexity of eye-movement control during reading allows measurement of many dependent variables, the most prominent ones being fixation durations and their locations in words. In current practice, either variable may serve as dependent variable or covariate for the other in linear mixed models (LMMs) featuring also psycholinguistic covariates of word recognition and sentence comprehension. Rather than analyzing fixation location and duration with separate LMMs, we propose linking the two according to their sequential dependency. Specifically, we include predicted fixation location (estimated in the first LMM from psycholinguistic covariates) and its associated residual fixation location as covariates in the second, fixation-duration LMM. This linked LMM affords a distinction between direct and indirect effects (mediated through fixation location) of psycholinguistic covariates on fixation durations. Results confirm the robustness of distributed processing in the perceptual span. They also offer a resolution of the paradox of the inverted optimal viewing position (IOVP) effect (i.e., longer fixation durations in the center than at the beginning and end of words) although the opposite (i.e., an OVP effect) is predicted from default assumptions of psycholinguistic processing efficiency: The IOVP effect in fixation durations is due to the residual fixation-location covariate, presumably driven primarily by saccadic error, and the OVP effect (at least the left part of it) is uncovered with the predicted fixation-location covariate, capturing the indirect effects of psycholinguistic covariates. We expect that linked LMMs will be useful for the analysis of other dynamically related multiple outcomes, a conundrum of most psychonomic research.     

Rapid detection of person information in a naturalistic scene

A preferential-looking paradigm was used to investigate how gaze is distributed in naturalistic scenes. Two scenes were presented side by side: one contained a single person (person-present) and one did not (person-absent). Eye movements were recorded, the principal measures being the time spent looking at each region of the scenes, and the latency and location of the first fixation within each trial. We studied gaze patterns during free viewing, and also in a task requiring gender discrimination of the human figure depicted. Results indicated a strong bias towards looking to the person-present scene. This bias was present on the first fixation after image presentation, confirming previous findings of ultra-rapid processing of complex information. Faces attracted disproportionately many fixations, the preference emerging in the first fixation and becoming stronger in the following ones. These biases were exaggerated in the gender-discrimination task. A tendency to look at the object being fixated by the person in the scene was shown to be strongest at a slightly later point in the gaze sequence. We conclude that human bodies and faces are subject to special perceptual processing when presented as part of a naturalistic scene.     

Eye scanpaths during visual imagery reenact those of perception of the same visual scene

Eye movements during mental imagery are not epiphenomenal but assist the process of image generation. Commands to the eyes for each fixation are stored along with the visual representation and are used as spatial index in a motor‐based coordinate system for the proper arrangement of parts of an image. In two experiments, subjects viewed an irregular checkerboard or color pictures of fish and were subsequently asked to form mental images of these stimuli while keeping their eyes open. During the perceptual phase, a group of subjects was requested to maintain fixation onto the screen's center, whereas another group was free to inspect the stimuli. During the imagery phase, all of these subjects were free to move their eyes. A third group of subjects (in Experiment 2) was free to explore the pattern but was requested to maintain central fixation during imagery. For subjects free to explore the pattern, the percentage of time spent fixating a specific location during perception was highly correlated with the time spent on the same (empty) locations during imagery. The order of scanning of these locations during imagery was correlated to the original order during perception. The strength of relatedness of these scanpaths and the vividness of each image predicted performance accuracy. Subjects who fixed their gaze centrally during perception did the same spontaneously during imagery. Subjects free to explore during perception, but maintaining central fixation during imagery, showed decreased ability to recall the pattern. We conclude that the eye scanpaths during visual imagery reenact those of perception of the same visual scene and that they play a functional role.     

Not fixating at the line of text comes at a cost

Previous research on eye guidance in reading has investigated systematic tendencies with respect to horizontal fixation locations on letters within words and the relationship between fixation location in a word and the duration of the fixation. The present study investigates where readers place their eyes vertically on the line of text and how vertical fixation location is related to fixation duration. Analyses were based on a large corpus of eye movement recordings from single-sentence reading. The vertical preferred viewing location was found to be within the vertical extent of the font, but fixations beyond the vertical boundaries of the text also frequently occurred. Analyzing fixation duration as a function of vertical fixation location revealed a vertical optimal viewing position (vOVP) effect: Fixations were shortest when placed optimally on the line of text, and fixation duration gradually increased for fixations that fell above or below the line of text. The vOVP effect can be explained by the limits of visual resolution along the vertical meridian. It is concluded that vertical and horizontal landing positions in single-sentence reading are associated with differences in fixation durations in opposite ways.     

Effect of contextual knowledge on spatial layout extrapolation

Boundary extension (BE) refers to the tendency to remember a previously perceived scene with a greater spatial expanse. This phenomenon is described as resulting from different sources of information: external (i.e., visual) and internally driven (i.e., amodal, conceptual, and contextual) information. Although the literature has emphasized the role of top-down expectations to account for layout extrapolation, their effect has rarely been tested experimentally. In this research, we attempted to determine how visual context affects BE, as a function of scene exposure duration (long, short). To induce knowledge about visual context, the memorization phase of the camera distance paradigm was preceded by a preexposure phase, during which each of the to-be-memorized scenes was presented in a larger spatial framework. In an initial experiment, we examined the effect of contextual knowledge with presentation duration, allowing for in-depth processing of visual information during encoding (i.e., 15 s). The results indicated that participants exposed to the preexposure showed decreased BE, and displayed no directional memory error in some conditions. Because the effect of context is known to occur at an early stage of scene perception, in a second experiment we sought to determine whether the effect of a preview occurs during the first fixation on a visual scene. The results indicated that BE seems not to be modulated by this factor at very brief presentation durations. These results are discussed in light of current visual scene representation theories.     

How does the purpose of inspection influence the potency of visual salience in scene perception?

Salience-map models have been taken to suggest that the locations of eye fixations are determined by the extent of the low-level discontinuities in an image. While such models have found some support, an increasing emphasis on the task viewers are performing implies that these models must combine with cognitive demands to describe how the eyes are guided efficiently. An experiment is reported in which eye movements to objects in photographs were examined while viewers performed a memory-encoding task or one of two search tasks. The objects depicted in the scenes had known salience ranks according to a popular model. Participants fixated higher-salience objects sooner and more often than lower-salience objects, but only when memorising scenes. This difference shows that salience-map models provide useful predictions even in complex scenes and late in viewing. However, salience had no effects when searching for a target defined by category or exemplar. The results suggest that salience maps are not used to guide the eyes in these tasks, that cognitive override by task demands can be total, and that modelling top-down search is important but may not be easily accomplished within a salience-map framework.     

Attentional allocation during the perception of scenes

Semantic influences on attention during the 1st fixation on a scene were explored in 3 experiments. Subjects viewed briefly presented scenes; following scene presentation, a spatial probe was presented at the location of an object whose identity was consistent or inconsistent with the scene category. Responses to the probe served as an index of attention. The results of Experiment 1 suggest that within approximately 150 ms of scene onset, subjects attend preferentially to inconsistent objects. The results of Experiment 2, in which additional scene durations were used, confirm the presence of an inconsistent-object advantage that emerges within approximately 150 ms. Finally, the results of Experiment 3 demonstrate that the inconsistent-object advantage does not reflect strategic allocation of attention to likely probe locations. Implications of the results for scene perception and exploration are discussed.     

Scrambled eyes? Disrupting scene structure impedes focal processing and increases bottom-up guidance

Previous research has demonstrated that search and memory for items within natural scenes can be disrupted by "scrambling" the images. In the present study, we asked how disrupting the structure of a scene through scrambling might affect the control of eye fixations in either a search task (Experiment 1) or a memory task (Experiment 2). We found that the search decrement in scrambled scenes was associated with poorer guidance of the eyes to the target. Across both tasks, scrambling led to shorter fixations and longer saccades, and more distributed, less selective overt attention, perhaps corresponding to an ambient mode of processing. These results confirm that scene structure has widespread effects on the guidance of eye movements in scenes. Furthermore, the results demonstrate the trade-off between scene structure and visual saliency, with saliency having more of an effect on eye guidance in scrambled scenes.     

Does oculomotor inhibition of return influence fixation probability during scene search?

Oculomotor inhibition of return (IOR) is believed to facilitate scene scanning by decreasing the probability that gaze will return to a previously fixated location. This "foraging" hypothesis was tested during scene search and in response to sudden-onset probes at the immediately previous (one-back) fixation location. The latencies of saccades landing within 1o of the previous fixation location were elevated, consistent with oculomotor IOR. However, there was no decrease in the likelihood that the previous location would be fixated relative to distance-matched controls or an a priori baseline. Saccades exhibit an overall forward bias, but this is due to a general bias to move in the same direction and for the same distance as the last saccade (saccadic momentum) rather than to a spatially specific tendency to avoid previously fixated locations. We find no evidence that oculomotor IOR has a significant impact on return probability during scene search.     

Waldo reveals cultural differences in return fixations

ABSTRACT

Humans routinely perform visual search towards targets to adapt to the environment. These sequences of ballistic eye movements are shaped by a combination of top–down and bottom–up factors. Recent research documented that human observers display cultural-specific fixation patterns in a range of visual processing tasks. In particular, eye movement strategies extracting information from faces clearly differs between Western Caucasian (WC) and East Asian (EA) observers. However, whether such cultural differences are also present for visual scene processing remains debated. To this aim, we recorded the eye movements of WC and EA observers while they were solving visual search problems parametrically varying in difficulty: Where’s Waldo. Both groups had a comparable familiarity with the Waldo books reaching a comparable level of accuracy in target detection. Both cultural groups also showed a comparable temporal effect on inhibition of return, with longer fixation durations when saccades were performed to a return location compared to other locations. Westerners, however, located Waldo faster than Easterners. Interestingly, this modulation of speed was likely related to differences occurring on the low-level mechanisms of spatial inhibition of return, with EA observers returning more often to previously visited locations than the WC observers. This suboptimal eye movement strategy in the Easterners might be engendered by their cultural perceptual bias consisting in? a greater use of extra-foveal information. Overall, our data point towards the existence of a subtle, but significant difference in the processing of visual scenes across observers from different cultures during active visual search.