Capturing and evaluating blinks from video-based eyetrackers

Blinks are related to several emotional states, and the present report describes a simple, reliable way to measure blinks from the video stream of an eye obtained during eyetracking, where the source of the eye video is a video camera attached to a head-mounted eyetracker. Computer vision techniques are employed to determine the moments that a blink starts and ends, for the purpose of calculating blink frequency and duration. The video is first processed to show blocks of eyelid and pupil movements, and is then analyzed for blink starts and ends. The moment of a blink start is reported when the eyelid starts to move quickly, exceeding a predetermined threshold. The end of a blink arises when the pupil size increases by less than a separate threshold. We observed several different blink patterns from different subjects, and our algorithm was designed to work for all of these patterns. We evaluated our algorithm by manually measuring the true blinks of five different subjects while they were eyetracked. To test the sensitivity and specificity of the algorithm, we employed a series of threshold values to plot the receiver operating characteristic curves. Using the best thresholds, we achieved excellent sensitivity (>90 %) and specificity (>99 %) over the five subjects. Potential applications of this research include real-time, nonintrusive, continuous and automated measurements of mental workload and other emotional states related to blink rates and durations.     

Bayesian inference for psychology,part III: Parameter estimation in nonstandard models

We demonstrate the use of three popular Bayesian software packages that enable researchers to estimate parameters in a broad class of models that are commonly used in psychological research. We focus on WinBUGS, JAGS, and Stan, and show how they can be interfaced from R and MATLAB. We illustrate the use of the packages through two fully worked examples; the examples involve a simple univariate linear regression and fitting a multinomial processing tree model to data from a classic false-memory experiment. We conclude with a comparison of the strengths and weaknesses of the packages. Our example code, data, and this text are available via https://osf.io/ucmaz/.     

Introduction to Bayesian Inference for Psychology

We introduce the fundamental tenets of Bayesian inference, which derive from two basic laws of probability theory. We cover the interpretation of probabilities, discrete and continuous versions of Bayes’ rule, parameter estimation, and model comparison. Using seven worked examples, we illustrate these principles and set up some of the technical background for the rest of this special issue of Psychonomic Bulletin & Review. Supplemental material is available via https://osf.io/wskex/.     

A data-driven algorithm for offline pupil signal preprocessing and eyeblink detection in low-speed eye-tracking protocols

Event detection is the conversion of raw eye-tracking data into events—such as fixations, saccades, glissades, blinks, and so forth—that are relevant for researchers. In eye-tracking studies, event detection algorithms can have a serious impact on higher level analyses, although most studies do not accurately report their settings. We developed a data-driven eyeblink detection algorithm (Identification-Artifact Correction [I-AC]) for 50-Hz eye-tracking protocols. I-AC works by first correcting blink-related artifacts within pupil diameter values and then estimating blink onset and offset. Artifact correction is achieved with data-driven thresholds, and more reliable pupil data are output. Blink parameters are defined according to previous studies on blink-related visual suppression. Blink detection performance was tested with experimental data by visually checking the actual correspondence between I-AC output and participants’ eye images, recorded by the eyetracker simultaneously with gaze data. Results showed a 97% correct detection percentage.     

Dutch taboo norms

This article provides norms for general taboo, personal taboo, insult, valence, and arousal for 672 Dutch words, including 202 taboo words. Norms were collected using a 7-point Likert scale and based on ratings by psychology students from the Erasmus University Rotterdam in The Netherlands. The sample consisted of 87 psychology students (58 females, 29 males). We obtained high reliability based on split-half analyses. Our norms show high correlations with arousal and valence ratings collected by another Dutch word-norms study (Moors et al.,, Behavior Research Methods, 45, 169–177, 2013). Our results show that the previously found quadratic relation (i.e., U-shaped pattern) between valence and arousal also holds when only taboo words are considered. Additionally, words rated high on taboo tended to be rated low on valence, but some words related to sex rated high on both taboo and valence. Words that rated high on taboo rated high on insult, again with the exception of words related to sex many of which rated low on insult. Finally, words rated high on taboo and insult rated high on arousal. The Dutch Taboo Norms (DTN) database is a useful tool for researchers interested in the effects of taboo words on cognitive processing. The data associated with this paper can be accessed via the Open Science Framework (https://osf.io/vk782/).     

Visual marking across eye blinks

Visual search for a conjunction target can be made efficient by presenting one initial set of distractors as a preview, prior to the onset of the other items in the search display Watson & Humphreys (Psychological Review 104:90–122, 1997). However, this “preview advantage” is lost if the initial items are offset for a brief period before onsetting again with the search display Kunar, Humphreys, & Smith (Psychological Science 14:181–185, 2003). Researchers have long disputed whether the preview advantage reflects a process of internally coding and suppressing the old items or of the onset of the new items capturing attention Donk & Theeuwes (Perception & Psychophysics 63:891–900, 2001). In this study, we assessed whether an internally driven blink (in which participants close their eyes) acts in the same manner as an external blink produced by offsetting and then onsetting the preview. In the novel blink conditions, participants searched feature, conjunction, and preview displays after being cued to blink their eyes. The search displays were presented during the eye blink, and so were immediately available once participants opened their eyes. Having participants make an eye blink generally slowed search but had no effect on the search slopes. In contrast, imposing an externally driven blink disrupted preview search. The data indicated that visual attention can compensate for internally driven blinks, and this does not lead to the loss of the representations of distractors across time. Moreover, efficient preview search occurred when the search items had no abrupt onsets, demonstrating that onsets of new search items are not critical for the preview benefit.     

Blinking predicts enhanced cognitive control

Recent models have suggested an important role for neuromodulation in explaining trial-to-trial adaptations in cognitive control. The adaptation-by-binding model (Verguts & Notebaert, Psychological review, 115(2), 518–525, 2008), for instance, suggests that increased cognitive control in response to conflict (e.g., incongruent flanker stimulus) is the result of stronger binding of stimulus, action, and context representations, mediated by neuromodulators like dopamine (DA) and/or norepinephrine (NE). We presented a flanker task and used the Gratton effect (smaller congruency effect following incongruent trials) as an index of cognitive control. We investigated the Gratton effect in relation to eye blinks (DA related) and pupil dilation (NE related). The results for pupil dilation were not unequivocal, but eye blinks clearly modulated the Gratton effect: The Gratton effect was enhanced after a blink trial, relative to after a no-blink trial, even when controlling for correlated variables. The latter suggests an important role for DA in cognitive control on a trial-to-trial basis.     

Safe and sensible preprocessing and baseline correction of pupil-size data

Measurement of pupil size (pupillometry) has recently gained renewed interest from psychologists, but there is little agreement on how pupil-size data is best analyzed. Here we focus on one aspect of pupillometric analyses: baseline correction, i.e., analyzing changes in pupil size relative to a baseline period. Baseline correction is useful in experiments that investigate the effect of some experimental manipulation on pupil size. In such experiments, baseline correction improves statistical power by taking into account random fluctuations in pupil size over time. However, we show that baseline correction can also distort data if unrealistically small pupil sizes are recorded during the baseline period, which can easily occur due to eye blinks, data loss, or other distortions. Divisive baseline correction (corrected pupil size = pupil size/baseline) is affected more strongly by such distortions than subtractive baseline correction (corrected pupil size = pupil size ? baseline). We discuss the role of baseline correction as a part of preprocessing of pupillometric data, and make five recommendations: (1) before baseline correction, perform data preprocessing to mark missing and invalid data, but assume that some distortions will remain in the data; (2) use subtractive baseline correction; (3) visually compare your corrected and uncorrected data; (4) be wary of pupil-size effects that emerge faster than the latency of the pupillary response allows (within ±220 ms after the manipulation that induces the effect); and (5) remove trials on which baseline pupil size is unrealistically small (indicative of blinks and other distortions).     

A stimulus set of words and pictures matched for visual and semantic similarity

Researchers in different fields of psychology have been interested in how vision and language interact, and what type of representations are involved in such interactions. We introduce a stimulus set that facilitates such research (available online). The set consists of 100 words each of which is paired with four pictures of objects: One semantically similar object (but visually dissimilar), one visually similar object (but semantically dissimilar), and two unrelated objects. Visual and semantic similarity ratings between corresponding items are provided for every picture for Dutch and for English. In addition, visual and linguistic parameters of each picture are reported. We thus present a stimulus set from which researchers can select, on the basis of various parameters, the items most optimal for their research question.     

Examining reproducibility in psychology: A hybrid method for combining a statistically significant original study and a replication

The unrealistically high rate of positive results within psychology has increased the attention to replication research. However, researchers who conduct a replication and want to statistically combine the results of their replication with a statistically significant original study encounter problems when using traditional meta-analysis techniques. The original study’s effect size is most probably overestimated because it is statistically significant, and this bias is not taken into consideration in traditional meta-analysis. We have developed a hybrid method that does take the statistical significance of an original study into account and enables (a) accurate effect size estimation, (b) estimation of a confidence interval, and (c) testing of the null hypothesis of no effect. We analytically approximate the performance of the hybrid method and describe its statistical properties. By applying the hybrid method to data from the Reproducibility Project: Psychology (Open Science Collaboration, 2015), we demonstrate that the conclusions based on the hybrid method are often in line with those of the replication, suggesting that many published psychological studies have smaller effect sizes than those reported in the original study, and that some effects may even be absent. We offer hands-on guidelines for how to statistically combine an original study and replication, and have developed a Web-based application (https://rvanaert.shinyapps.io/hybrid) for applying the hybrid method.     

The effect of blinks and saccadic eye movements on visual reaction times

Vision is suppressed during blinks and saccadic eye movements. We hypothesized that visual reaction times (RTs) in a vigilance test would be significantly increased when a blink or a saccade happened to coincide with the stimulus onset. Thirty healthy volunteers each performed a visual RT test for 15 min while their eye and eyelid movements were monitored by a system of infrared reflectance oculography. RTs increased significantly, many by more than 200 msec, when a blink occurred between 75 msec before and up to 150 msec after the stimulus onset. A similar result was observed with saccades that started 75 to 150 msec after the stimulus. Vision or attention was evidently inhibited before each blink and for longer than the saccades lasted. We suggest that visual suppression is involved in this process, which could explain some of the normal variability in RTs over periods of seconds that has not been adequately explained before.     

More than meets the eye: The attentional blink in multisensory environments. Commentary on Kranczioch and Thorne

Temporal fluctuations of attention can influence performance of cognitive tasks substantially. A common paradigm to investigate temporal fluctuations of attention is the attentional blink paradigm. Kranczioch and Thorne (2013) report new evidence for the impact of auditory stimuli on the visual attentional blink in the current issue of Advances in Cognitive Psychology.     

Reflex eyeblinks and visual suppression

Voluntary blinks are known to be accompanied by visual suppression that arises, at least in part, through nonretinal processes. We wanted to test the hypothesis that this suppression must arise as a corollary to the voluntary command to blink. We therefore examined visual sensitivity at the time of a blink that does not involve a voluntary command, namely the corneal reflex elicited by a puff of air. We found that sensitivity was reduced during these reflex blinks; the magnitude and time course of suppression were similar to those of voluntary blinks. We therefore adopt the view that visual suppression stems from signals from the common, efferent, blink pathway.     

SwarmSight: Measuring the temporal progression of animal group activity levels from natural-scene and laboratory videos

We describe SwarmSight (available at https://github.com/justasb/SwarmSight), a novel, open-source, Microsoft Windows software tool for quantitative assessment of the temporal progression of animal group activity levels from recorded videos. The tool utilizes a background subtraction machine vision algorithm and provides an activity metric that can be used to quantitatively assess and compare animal group behavior. Here we demonstrate the tool’s utility by analyzing defensive bee behavior as modulated by alarm pheromones, wild-bird feeding onset and interruption, and cockroach nest-finding activity. Although more sophisticated, commercial software packages are available, SwarmSight provides a low-cost, open-source, and easy-to-use alternative that is suitable for a wide range of users, including minimally trained research technicians and behavioral science undergraduate students in classroom laboratory settings.     

Ocular signatures of proactive versus reactive cognitive control in young adults

During the execution of a cognitive task, the brain maintains contextual information to guide behavior and achieve desired goals. The AX-Continuous Performance Task is used to study proactive versus reactive cognitive control. Young adults tend to behave proactively in standard testing conditions. However, it remains unclear how interindividual variability (e.g., in cognitive and motivational factors) may drive people into more reactive or proactive control under the same task demands. We investigated the use of control strategies in a large population of healthy young adults. We computed the proactive behavioral index and consequently divided participants into proactive, reactive, and intermediate groups. We found that reactive participants were generally slower, presented lower context sensitivity, and larger response variability. Pupillary changes and blink rate index cognitive effort allocation. We measured, concomitantly to the task, the pupil size and frequency of blinks associated with the cue maintenance and response intervals. During the cue period, nonfrequent, nontarget cues led to increased pupil dilation and number of blinks in all participants. During the response interval, we found more errors and increased pupil dilation to the probe when all participants had to overcome a response bias generated by the frequent cue. Only reactive participants showed larger response-related pupil when they had to overcome a response bias related to the frequent probe. Contrary to expectations, groups did not differ in ocular measures in the cue period. In conclusion, interindividual differences in cognitive control between healthy adults can be mapped onto different patterns of effort allocation indexed by the pupil.     

Detecting and identifying offset gaze

A number of experiments have demonstrated that observers can accurately identify stimuli that they fail to detect (Rollman and Nachmias, 1972; Harris and Fahle, 1995; Allik et al. 1982, 2014). Using a 2x2AFC double judgements procedure, we demonstrated an analogous pattern of performance in making judgements about the direction of eye gaze. Participants were shown two faces in succession: one with direct gaze and one with gaze offset to the left or right. We found that they could identify the direction of gaze offset (left/right) better than they could detect which face contained the offset gaze. A simple Thurstonian model, under which the detection judgement is shown to be more computationally complex, was found to explain the empirical data. A further experiment incorporated metacognitive ratings into the double judgements procedure to measure observers’ metacognitive awareness (Meta-d’) across the two judgements and to assess whether observers were aware of the evidence for offset gaze when detection performance was at and below threshold. Results suggest that metacognitive awareness is tied to performance, with approximately equal Meta-d’ across the two judgements, when sensitivity is taken into account. These results show that both performance and metacognitive awareness rely not only on the strength of sensory evidence but also on the computational complexity of the decision, which determines the relative distance of that evidence from the decision axes.     

Voluntary eyeblinks disrupt iconic memory

In the present research, we investigated whether eyeblinks interfere with cognitive processing. In Experiment 1, the participants performed a partial-report iconic memory task in which a letter array was presented for 106 msec, followed 50, 150, or 750 msec later by a tone that cued recall of onerow of the array. At a cue delay of 50 msec between array offset and cue onset, letter report accuracy was lower when the participants blinked following array presentation than under no-blink conditions; the participants made more mislocation errors under blink conditions. This result suggests that blinking interferes with the binding of object identity and object position in iconic memory. Experiment 2 demonstrated that interference due to blinks was not due merely to changes in light intensity. Experiments 3 and 4 demonstrated that other motor responses did not interfere with iconic memory. We propose a new phenomenon, cognitive blink suppression, in which blinking inhibits cognitive processing. This phenomenon may be due to neural interference. Blinks reduce activation in area V1, which may interfere with the representation of information in iconic memory.     

Spontaneous eye blinks during creative task correlate with divergent processing

Creativity consists of divergent and convergent thinking, with both related to individual eye blinks at rest. To assess underlying mechanisms between eye blinks and traditional creativity tasks, we investigated the relationship between creativity performance and eye blinks at rest and during tasks. Participants performed an alternative uses and remote association task while eye blinks were recorded. Results showed that the relationship between eye blinks at rest and creativity performance was compatible with those of previous research. Interestingly, we found that the generation of ideas increased as a function of eye blink number during the alternative uses task. On the other hand, during the remote association task, accuracy was independent of eye blink number during the task, but response time increased with it. Moreover, eye blink changes in participants who responded quickly during the remote association task were different depending on their resting state eye blinks; that is, participants with many eye blinks during rest showed little increasing eye blinks and achieved solutions quickly. Positive correlations between eye blinks during creative tasks and yielding ideas on the alternative uses task and response time on the remote association task suggest that eye blinks during creativity tasks relate to divergent thinking processes such as conceptual reorganization.     

Online psychophysics: reaction time effects in cognitive experiments

Using the Internet to acquire behavioral data is currently on the rise. However, very basic questions regarding the feasibility of online psychophysics are still open. Here, we aimed to replicate five well-known paradigms in experimental psychology (Stroop, Flanker, visual search, masked priming, attentional blink) in three settings (classical “lab”, “web-in-lab”, “web”) to account for possible changes in technology and environment. Lab and web-in-lab data were both acquired in an in-lab setting with lab using “Gold Standard” methods, while web-in-lab used web technology. This allowed for a direct comparison of potential differences in acquisition software. To account for additional environmental differences, the web technology experiments were published online to participate from home (setting web), thereby keeping the software and experimental design identical and only changing the environmental setting. Our main results are: First, we found an expected fixed additive timing offset when using web technology (M = 37 ms, SD = 8.14) and recording online (M = 87 ms, SD = 16.04) in comparison to lab data. Second, all task-specific effects were reproduced except for the priming paradigm, which couldn’t be replicated in any setting. Third, there were no differences in error rates, which are independent of the timing offset. This finding further supports the assumption of data equality over all settings. Fourth, we found that browser type might be influencing absolute reaction times. Together, these results contribute to the slowly but steadily growing literature that online psychophysics is a suitable complement – or even substitute – to lab data acquisition.