Accurate Detection of Mental Fatigue with LSTM Network using fNIRS
Poster No:
1393
Submission Type:
Abstract Submission
Authors:
Mengshan Li1,2, Wenchao Zhang1,2, Jiayu Xu1,2, Zaichen La1,2, Huiling Zhou1,2, Zhao Yan1,2, Yonghuan Feng1,2, Guanya Li1,2, Yang Hu1,2, Yi Zhang1,2
Institutions:
1Center for Brain Imaging, School of Life Science and Technology, Xidian University, Xi'an, Shaanxi 710126, China, 2International Joint Research Center for Advanced Medical Imaging and Intelligent Diagnosis and Treatment & Xi'an Key Laboratory of Intelligent Sensing and Regulation of trans-Scale Life Information, School of Life Science and Technology, Xidian University, Xi'an, Shaanxi 710126, China
First Author:
Mengshan Li
Center for Brain Imaging, School of Life Science and Technology, Xidian University|International Joint Research Center for Advanced Medical Imaging and Intelligent Diagnosis and Treatment & Xi'an Key Laboratory of Intelligent Sensing and Regulation of trans-Scale Life Information, School of Life Science and Technology, Xidian University
Xi'an, Shaanxi 710126, China|Xi'an, Shaanxi 710126, China
Center for Brain Imaging, School of Life Science and Technology, Xidian University|International Joint Research Center for Advanced Medical Imaging and Intelligent Diagnosis and Treatment & Xi'an Key Laboratory of Intelligent Sensing and Regulation of trans-Scale Life Information, School of Life Science and Technology, Xidian University
Xi'an, Shaanxi 710126, China|Xi'an, Shaanxi 710126, China
Co-Author(s):
Wenchao Zhang
Center for Brain Imaging, School of Life Science and Technology, Xidian University|International Joint Research Center for Advanced Medical Imaging and Intelligent Diagnosis and Treatment & Xi'an Key Laboratory of Intelligent Sensing and Regulation of trans-Scale Life Information, School of Life Science and Technology, Xidian University
Xi'an, Shaanxi 710126, China|Xi'an, Shaanxi 710126, China
Center for Brain Imaging, School of Life Science and Technology, Xidian University|International Joint Research Center for Advanced Medical Imaging and Intelligent Diagnosis and Treatment & Xi'an Key Laboratory of Intelligent Sensing and Regulation of trans-Scale Life Information, School of Life Science and Technology, Xidian University
Xi'an, Shaanxi 710126, China|Xi'an, Shaanxi 710126, China
Jiayu Xu
Center for Brain Imaging, School of Life Science and Technology, Xidian University|International Joint Research Center for Advanced Medical Imaging and Intelligent Diagnosis and Treatment & Xi'an Key Laboratory of Intelligent Sensing and Regulation of trans-Scale Life Information, School of Life Science and Technology, Xidian University
Xi'an, Shaanxi 710126, China|Xi'an, Shaanxi 710126, China
Center for Brain Imaging, School of Life Science and Technology, Xidian University|International Joint Research Center for Advanced Medical Imaging and Intelligent Diagnosis and Treatment & Xi'an Key Laboratory of Intelligent Sensing and Regulation of trans-Scale Life Information, School of Life Science and Technology, Xidian University
Xi'an, Shaanxi 710126, China|Xi'an, Shaanxi 710126, China
Zaichen La
Center for Brain Imaging, School of Life Science and Technology, Xidian University|International Joint Research Center for Advanced Medical Imaging and Intelligent Diagnosis and Treatment & Xi'an Key Laboratory of Intelligent Sensing and Regulation of trans-Scale Life Information, School of Life Science and Technology, Xidian University
Xi'an, Shaanxi 710126, China|Xi'an, Shaanxi 710126, China
Center for Brain Imaging, School of Life Science and Technology, Xidian University|International Joint Research Center for Advanced Medical Imaging and Intelligent Diagnosis and Treatment & Xi'an Key Laboratory of Intelligent Sensing and Regulation of trans-Scale Life Information, School of Life Science and Technology, Xidian University
Xi'an, Shaanxi 710126, China|Xi'an, Shaanxi 710126, China
Huiling Zhou
Center for Brain Imaging, School of Life Science and Technology, Xidian University|International Joint Research Center for Advanced Medical Imaging and Intelligent Diagnosis and Treatment & Xi'an Key Laboratory of Intelligent Sensing and Regulation of trans-Scale Life Information, School of Life Science and Technology, Xidian University
Xi'an, Shaanxi 710126, China|Xi'an, Shaanxi 710126, China
Center for Brain Imaging, School of Life Science and Technology, Xidian University|International Joint Research Center for Advanced Medical Imaging and Intelligent Diagnosis and Treatment & Xi'an Key Laboratory of Intelligent Sensing and Regulation of trans-Scale Life Information, School of Life Science and Technology, Xidian University
Xi'an, Shaanxi 710126, China|Xi'an, Shaanxi 710126, China
Zhao Yan
Center for Brain Imaging, School of Life Science and Technology, Xidian University|International Joint Research Center for Advanced Medical Imaging and Intelligent Diagnosis and Treatment & Xi'an Key Laboratory of Intelligent Sensing and Regulation of trans-Scale Life Information, School of Life Science and Technology, Xidian University
Xi'an, Shaanxi 710126, China|Xi'an, Shaanxi 710126, China
Center for Brain Imaging, School of Life Science and Technology, Xidian University|International Joint Research Center for Advanced Medical Imaging and Intelligent Diagnosis and Treatment & Xi'an Key Laboratory of Intelligent Sensing and Regulation of trans-Scale Life Information, School of Life Science and Technology, Xidian University
Xi'an, Shaanxi 710126, China|Xi'an, Shaanxi 710126, China
Yonghuan Feng
Center for Brain Imaging, School of Life Science and Technology, Xidian University|International Joint Research Center for Advanced Medical Imaging and Intelligent Diagnosis and Treatment & Xi'an Key Laboratory of Intelligent Sensing and Regulation of trans-Scale Life Information, School of Life Science and Technology, Xidian University
Xi'an, Shaanxi 710126, China|Xi'an, Shaanxi 710126, China
Center for Brain Imaging, School of Life Science and Technology, Xidian University|International Joint Research Center for Advanced Medical Imaging and Intelligent Diagnosis and Treatment & Xi'an Key Laboratory of Intelligent Sensing and Regulation of trans-Scale Life Information, School of Life Science and Technology, Xidian University
Xi'an, Shaanxi 710126, China|Xi'an, Shaanxi 710126, China
Guanya Li
Center for Brain Imaging, School of Life Science and Technology, Xidian University|International Joint Research Center for Advanced Medical Imaging and Intelligent Diagnosis and Treatment & Xi'an Key Laboratory of Intelligent Sensing and Regulation of trans-Scale Life Information, School of Life Science and Technology, Xidian University
Xi'an, Shaanxi 710126, China|Xi'an, Shaanxi 710126, China
Center for Brain Imaging, School of Life Science and Technology, Xidian University|International Joint Research Center for Advanced Medical Imaging and Intelligent Diagnosis and Treatment & Xi'an Key Laboratory of Intelligent Sensing and Regulation of trans-Scale Life Information, School of Life Science and Technology, Xidian University
Xi'an, Shaanxi 710126, China|Xi'an, Shaanxi 710126, China
Yang Hu
Center for Brain Imaging, School of Life Science and Technology, Xidian University|International Joint Research Center for Advanced Medical Imaging and Intelligent Diagnosis and Treatment & Xi'an Key Laboratory of Intelligent Sensing and Regulation of trans-Scale Life Information, School of Life Science and Technology, Xidian University
Xi'an, Shaanxi 710126, China|Xi'an, Shaanxi 710126, China
Center for Brain Imaging, School of Life Science and Technology, Xidian University|International Joint Research Center for Advanced Medical Imaging and Intelligent Diagnosis and Treatment & Xi'an Key Laboratory of Intelligent Sensing and Regulation of trans-Scale Life Information, School of Life Science and Technology, Xidian University
Xi'an, Shaanxi 710126, China|Xi'an, Shaanxi 710126, China
Yi Zhang
Center for Brain Imaging, School of Life Science and Technology, Xidian University|International Joint Research Center for Advanced Medical Imaging and Intelligent Diagnosis and Treatment & Xi'an Key Laboratory of Intelligent Sensing and Regulation of trans-Scale Life Information, School of Life Science and Technology, Xidian University
Xi'an, Shaanxi 710126, China|Xi'an, Shaanxi 710126, China
Center for Brain Imaging, School of Life Science and Technology, Xidian University|International Joint Research Center for Advanced Medical Imaging and Intelligent Diagnosis and Treatment & Xi'an Key Laboratory of Intelligent Sensing and Regulation of trans-Scale Life Information, School of Life Science and Technology, Xidian University
Xi'an, Shaanxi 710126, China|Xi'an, Shaanxi 710126, China
Introduction:
Long-term cognitive tasks easily lead to mental fatigue, usually manifested in decreased attention, slower reaction time, and increased aversion to tasks, resulting in increased accident incidence [1]. To cope with these adverse but preventable consequences caused by mental fatigue, accurate detection of mental fatigue is required. Several ways including electroencephalography and functional Near-infrared Spectroscopy (fNIRS) have been employed, however, they used multiple channels covering the whole brain [2-4], which is less inefficient and inconvenient in applications [5]. Thus, it is necessary to identify the specific regions that are closely related to mental fatigue. In the current study, multi-channel fNIRS signals covering the frontal and parietal brain regions were measured on participants with normal and mentally fatigued states, respectively. Then, the individualized mental fatigue detection model was constructed and the contribution channels for fatigue detection were determined.
Methods:
Seventeen healthy students were recruited from Xidian University to participate in the study. The Shimadzu LABNIRS device was employed to measure the fNIRS data of participants during the sustained cognitive tasks inducing mental fatigue (Figure 1A), and optrodes were arranged on the frontal-parietal brain regions according to the international 10-20 system, with a total of 46 channels (Figure 1B). Resting - state fNIRS data were measured for each participant before and after the sustained cognitive task, and their mental fatigue levels were assessed using the Karolinska Sleepiness Scale (KSS, Figure 2A). Each subject participated in the experiment twice.
The two five-minute resting-state datasets before and after the sustained cognitive task were categorized into normal and mental fatigue states. A sliding window with a length of 6s and a shift of 1.5s was applied to segment the fNIRS data into 196 normal and mental fatigue samples for each participant in each experiment respectively. For each participant, a detection model based on the Long Short-Term Memory (LSTM) network was trained with the samples from the first experiment, and the samples from the second experiment were used to test the model (Model 1). The shap interpreter was employed to evaluate the weights of channels [6], and the top 20% of channels contributing most to the classification were selected as mental fatigue-related channels. Then, the detection model was retrained and tested with the selected channels (Model 2). In addition, paired t-tests were performed to test the difference in Amplitude of Low Frequency Fluctuations (ALFF) between normal and mental fatigue states for each channel, and channels with significant differences were chosen to retrain the detection models again for each participant (Model 3, Figure 1C).
The two five-minute resting-state datasets before and after the sustained cognitive task were categorized into normal and mental fatigue states. A sliding window with a length of 6s and a shift of 1.5s was applied to segment the fNIRS data into 196 normal and mental fatigue samples for each participant in each experiment respectively. For each participant, a detection model based on the Long Short-Term Memory (LSTM) network was trained with the samples from the first experiment, and the samples from the second experiment were used to test the model (Model 1). The shap interpreter was employed to evaluate the weights of channels [6], and the top 20% of channels contributing most to the classification were selected as mental fatigue-related channels. Then, the detection model was retrained and tested with the selected channels (Model 2). In addition, paired t-tests were performed to test the difference in Amplitude of Low Frequency Fluctuations (ALFF) between normal and mental fatigue states for each channel, and channels with significant differences were chosen to retrain the detection models again for each participant (Model 3, Figure 1C).
Results:
The KSS score was significantly increased along with the sustained task progress (P < 0.001, Figure 2A). The average accuracy of the Model 1 was 91.93%. Channels with high contributions were located in Brodmann's area 5 (BA5) and BA7 (parietal area and somatosensory association cortex), BA6 and 9 (frontal area), and BA40 (superior temporal gyrus, Figure 2B). Model 2 used the above mental fatigue-related channels and achieved an average accuracy of 91.06%, with no significant decrease compared with Model 1. The group-level statistic method identified channels corresponding to BA8, BA9, and BA40. However, the detection model with these channels (Model 3) only yielded an accuracy of 50.64%, which was significantly lower than that of Model 1 (P < 0.001, Figure 2C).
Conclusions:
This study identified mental fatigue-related channels within the frontal-parietal network based on individualized mental fatigue detection models, reducing the number of required optrodes without compromising detection accuracy. This study is helpful to the practical application of mental fatigue detection based on fNIRS.
Modeling and Analysis Methods:
Classification and Predictive Modeling 1
Novel Imaging Acquisition Methods:
NIRS 2
Keywords:
Other - mental fatigue; fNIRS; LSTM; frontal-parietal network.
1|2Indicates the priority used for review
Provide references using author date format
[2] Zhang, C., Sun, L., Cong, F., Kujala, T., Ristaniemi, T., and Parviainen, T.(2020). ‘Optimal imaging of multi-channel EEG features based on a novel clustering technique for driver fatigue detection’, Biomed. Signal Process and Control, vol 62, pp 1-9.
[3] Peng Y, Li C, Chen Q, Zhu Y, Sun L. (2022). ‘Functional Connectivity Analysis and Detection of Mental Fatigue Induced by Different Tasks Using Functional Near-Infrared Spectroscopy’, Frontiers in neuroscience, vol. 15.
[4] Lin, C. T., King, J. T., Chuang, C. H., Ding, W., Chuang, W. Y., Liao, L. D., & Wang, Y. K. (2020). ‘Exploring the Brain Responses to Driving Fatigue Through Simultaneous EEG and fNIRS Measurements’, International journal of neural systems, vol. 30, no. 2.
[5] Li, R. Yang, D. Fang, F, Hong, K.-S, Reiss, A.L, Zhang, Y. (2022). ‘Concurrent fNIRS and EEG for Brain Function Investigation: A Systematic, Methodology-Focused Review’, Sensors (Basel, Switzerland), vol. 22, no. 15.
[6] Lundberg, Scott M., and Su-In Lee.(2017). ‘A unified approach to interpreting model predictions’, Advances in neural information processing systems, pp. 4765-4774