Facial Video-Based Remote Eye Movement Detection using Electrooculography and Deep Neural Network

726 

Abstract Submission 

Jin Soo Jeon¹, Jong-Hwan Lee¹

¹Department of Brain and Cognitive Engineering, Korea University, Seoul, Korea, Republic of

Jin Soo Jeon  
Department of Brain and Cognitive Engineering, Korea University
Seoul, Korea, Republic of

Jong-Hwan Lee  
Department of Brain and Cognitive Engineering, Korea University
Seoul, Korea, Republic of

Blink duration and rate have been widely recognized as consistent and reliable parameters for assessing drowsiness and arousal [1]. Electrooculography (EOG) has precisely measured eye movement features, such as eyeblink rate, closing phase, opening phase, and blink duration [2]. Despite the established validity and reliability of the EOG signal in capturing eye movements, the necessity of electrode attachment near eyeballs for EOG measurement significantly limits its utility. Consequently, our study addresses this limitation by leveraging facial videos and deep neural network (DNN).

We employed the facial video and EOG data available in the DEAP dataset [3], a widely recognized multimodal dataset for human emotion analysis. To ensure synchronization with the video recordings at 50 frames per second, we downsampled the EOG signal to 50Hz. Subsequently, we re-referenced the vertical EOG signal to its corresponding electrode and applied bandpass filtering in the range 0.01–25Hz. Epochs were extracted from –5s to 60s with baseline removal using –5s to 0s signals. We utilized the discrete wavelet transform method [4] to mitigate baseline drift and further reduced residual noise via a moving median filter [5].

For blink peak detection from the preprocessed EOG, we implemented min-max scaling within subjects and employed a local maxima algorithm [4]. Blink phase detection, pinpointing the precise time points for initial closure, complete closure, and reopening of eyes were obtained using the derivative value of the vertical EOG signal [6]. We categorized target labels for all video frames into three distinct states of eye movements (i.e., normal, closing, and opening) based on the blink phase detection from EOG. A Pearson's correlation analysis using the identified EOG labels and self-reports from subjects was conducted to evaluate the validity of the extracted labels.

We trained a DNN model using recorded facial videos for eye movement detection. Our DNN model consists of (a) an Inception-ResNetV1 module to extract hierarchical visual features from each video frame, (b) a Long Short-Term Memory (LSTM) module to analyze the sequence of video frames, and (c) fully connected layers for the classification of the target labels (Fig. 2a). The performance of the model was evaluated on both the DEAP dataset and the DROZY database [7] based on the five-folds cross-validation (CV). The face region was detected and cropped by the OpenCV face detection model [8]. Six consecutive video frames were used as DNN input and model parameters were optimized using a random search method [9].

Our adopted EOG preprocessing pipeline effectively denoised EOG signals while maintaining important temporal features of eye movements (Fig. 1a-d). Among the 16 subjects in the DEAP dataset with valid EOG signals, seven showed a meaningful negative correlation between blink duration and arousal (Fig. 1e). From the DROZY Database, data from all 14 subjects revealed a substantially high correlation of 0.59 (p < 0.01) between blink duration and Karolinska Sleepiness Scale Score.

Our trained DNN yielded a 98.2% average test accuracy across the 5-fold CV using the DEAP dataset. The proposed DNN effectively predicted eye movements in facial videos, demonstrating that the average duration error, compared to the EOG label, was approximately 60ms (3 frames) with a minimal delay of 40ms (2 frames) (Fig. 2c).

We developed a DNN model to extract vertical eye blink movements from facial videos robustly. Notably, our facial video-based blink detection model often outperformed the EOG label in predicting the observable eye movement. This would be because the EOG, which captures electric signals originating from inner muscle movement, may inherently introduce minor differences from overt eye movements. The estimated eye blink movements can potentially estimate affective states and arousal, including drowsiness, remotely using a camera without EOG electrodes.

Emotion and Motivation Other ¹

Classification and Predictive Modeling

Methods Development

Multivariate Approaches

Other Methods ²

Computing

Emotions

Machine Learning

Modeling

Other - Deep neural networks; Electrooculography; Eye movement detection; Facial video