Leveraging periodic activation functions in deep neural network to predict fMRI signals from EEG
Poster No:
1455
Submission Type:
Abstract Submission
Authors:
Yamin Li1, Ange Lou1, Shiyu Wang1, Catie Chang1
Institutions:
1Vanderbilt University, Nashville, TN
First Author:
Co-Author(s):
Introduction:
fMRI has been an irreplaceable neuroimaging modality but is limited by hemodynamic blurring, high cost, and incompatibility with metal implants [1]. Complementary to fMRI, EEG directly records cortical electrical activity at high temporal resolution, but has limited spatial resolution and limited sensitivity to deep subcortical brain structures. The ability to obtain fMRI information from EEG would enable cost-effective, naturalistic imaging across a wider set of brain regions. Further, beyond augmenting the capabilities of EEG, cross-modality models would facilitate the interpretation of fMRI signals. However, as both EEG and fMRI are high-dimensional and prone to noise and artifacts, it is currently still challenging to model fMRI from EEG. To address this challenge, here we aimed to construct a novel deep learning framework to reconstruct fMRI signals directly from EEG data.
Methods:
Simultaneous "Eyes Open – Eyes Closed" EEG-fMRI data from 8 subjects were used in this analysis (EEG: 30channels; fMRI: TR=1.95s for first 4 subjects and TR = 2s for the last 4; resolution=3mm isotropic; duration=5min per subject, data and preprocessing details see in [2]). fMRI ROI signals were extracted using the Harvard-Oxford structural atlas [3] and interpolated to 100 Hz. EEG was downsampled to the same sampling rate, and shifted by 6 seconds to approximate the time delay of the HRF [4]. In our analysis, we trained subject-specific models given the potentially unique response properties of individuals. The preprocessed data for each subject were divided into training and testing sets in a ratio of 4:1. The training samples with a window length of 20.48 seconds were randomly sampled from the 4-minute training time course.
Our model comprises two main components: 1) sinusoidal representation network (SIREN) blocks and 2) feature encoder and decoder blocks. Inspired by [5], the input is first passed to SIREN, a framework that leverages the periodic activation function in each layer of a multilayer perceptron for spatial filtering and frequency-related feature extraction. The output of SIREN is sent into a subsequent encoder-decoder to recover the fMRI signals. Each encoder block has a down-sampling operation to increase the receptive field while retaining important information. The decoder comprises the same symmetric building blocks and up-samples the latent space features to produce the fMRI signal of a certain ROI. The model optimizes the linear combination of the mean squared error loss and the correlation loss: Loss=L_mse+αL_corr. The prediction performance is evaluated by calculating the Pearson correlation between predicted ROI signals and the ground truth.
Our model comprises two main components: 1) sinusoidal representation network (SIREN) blocks and 2) feature encoder and decoder blocks. Inspired by [5], the input is first passed to SIREN, a framework that leverages the periodic activation function in each layer of a multilayer perceptron for spatial filtering and frequency-related feature extraction. The output of SIREN is sent into a subsequent encoder-decoder to recover the fMRI signals. Each encoder block has a down-sampling operation to increase the receptive field while retaining important information. The decoder comprises the same symmetric building blocks and up-samples the latent space features to produce the fMRI signal of a certain ROI. The model optimizes the linear combination of the mean squared error loss and the correlation loss: Loss=L_mse+αL_corr. The prediction performance is evaluated by calculating the Pearson correlation between predicted ROI signals and the ground truth.
·Fig. 1 Research pipeline and model architecture
Results:
We observe a reasonable agreement between the actual and predicted ROI fMRI traces in subcortical regions. The corresponding attribution topographies mainly emphasize the motor cortex which is consistent with the functional role of basal ganglia and the presence of corresponding interconnections leading to motor cortical areas [6]. Overall, as shown in Fig. 2(C), our model outperforms the current state-of-the-art deep learning model that was designed for the same task [4].
·Fig. 2 The fMRI prediction results
Conclusions:
Our proposed model successfully reconstructs fMRI signals from EEG time series without explicit feature engineering and improves the prediction accuracy compared with existing models. This work contributes a novel framework that leverages periodic activation functions in deep neural networks to learn representations of functional neuroimaging data. As we only train our model on the "eyes-open-eyes-closed" data on healthy control and the prediction performances might vary on different datasets, future work would try to assess performance on different task conditions and patient populations.
Modeling and Analysis Methods:
Activation (eg. BOLD task-fMRI)
Classification and Predictive Modeling 1
EEG/MEG Modeling and Analysis 2
Methods Development
Keywords:
Data analysis
Electroencephaolography (EEG)
FUNCTIONAL MRI
Machine Learning
Other - EEG to fMRI; cross-modal prediction
1|2Indicates the priority used for review
Provide references using author date format
[2] van der Meer, J. et al. (2016), 'Eyes Open - Eyes Closed’ EEG/fMRI data set including dedicated ‘Carbon Wire Loop’ motion detection channels,' Data in brief, vol. 7, pp. 990–994.
[3] Desikan, R. S. et al.(2006), 'An automated labeling system for subdividing the human cerebral cortex on MRI scans into gyral based regions of interest,' Neuroimage, vol. 31, no. 3, pp. 968–980.
[4] Kovalev, A., Mikheev, I., and Ossadtchi, A.(2022), 'fMRI from EEG is only Deep Learning away: the use of interpretable DL to unravel EEG-fMRI relationships,', preprint.
[5] Sitzmann, V. et al. (2020), “Implicit neural representations with periodic activation functions,” Advances in neural information processing systems, vol. 2020-Decem.
[6] Purves, D. et al. (2001), Neuroscience 2nd edition. glutamate receptors in chapter 7.