Data-driven latent alignment in simultaneous EEG-fMRI to fuse and generalize brain states

2398 

Abstract Submission 

Reinmar Kobler^1,2, Toshikazu Kuroda¹, Takeshi Ogawa¹, Ce Ju³, Motoaki Kawanabe^1,2

¹Advanced Telecommunications Research Institute International, Kyoto, Japan, ²RIKEN AIP, Kyoto, Japan, ³Nanyang Technological University, Singapore, Singapore

Reinmar Kobler  
Advanced Telecommunications Research Institute International|RIKEN AIP
Kyoto, Japan|Kyoto, Japan

Toshikazu Kuroda  
Advanced Telecommunications Research Institute International
Kyoto, Japan

Takeshi Ogawa  
Advanced Telecommunications Research Institute International
Kyoto, Japan

Ce Ju  
Nanyang Technological University
Singapore, Singapore

Motoaki Kawanabe  
Advanced Telecommunications Research Institute International|RIKEN AIP
Kyoto, Japan|Kyoto, Japan

Modalities like electroencephalography (EEG) and functional magnetic resonance imaging (fMRI) provide distinct views to human whole-brain dynamics owing to their contrasting spatiotemporal sensitivities and underlying neurophysiological coupling mechanisms. Advances in simultaneous EEG-fMRI instrumentation lays the groundwork for fusing these distinct views to recover whole-brain dynamics at high spatiotemporal resolution. However, fusing EEG and fMRI data poses a considerable challenge (Phil.+2021) due to low signal-to-noise ratios, non-stationarities, outliers, and a poorly understood non-linear coupling mechanism (Warb. 2022). To address these challenges, past methods relied on extensive domain-knowledge-based preprocessing and feature extraction before fusing both views (Warb. 2022, Sada.+2020).
Our take on this problem is fundamentally different. Using minimally pre-processed data, we propose a data-driven approach to establish a latent representation space where EEG and fMRI dynamics vary consistently. To do so, we leverage cutting-edge EEG (Kobl.+2022) and fMRI (Thom.+2022) decoder architectures and train them to maximize geodesic correlation (Ju+in review) between their latent representations (Fig.1A).

In this study, we applied our proposed approach to two simultaneous EEG-fMRI datasets. The first (DS1), publicly available (v. d. Me.+2016), comprised resting subjects (n=8, 2 runs) in eyes closed vs. open conditions. We considered 6 subjects with clear alpha power changes between conditions. The second dataset (DS2), collected in-house, involved n-back task data (n=31, 2 days) in 0-back vs. 2-back conditions. For both datasets, fMRI data underwent preprocessing using fmriprep, followed by regressing out confounds and extracting ROI activity between 0.01 and 0.2 Hz. EEG data were preprocessed using a custom pipeline to mitigate artifacts and extract brain activity in channel-space between 1 and 36 Hz. Each run was segmented into short segments of paired EEG-fMRI data (10 s duration), which were robustly z-scored and concatenated. We generated train/test splits using cross-validation (CV), assessing generalization across subjects, days, and runs. Additionally, in DS2, we used unimodal EEG recordings to test generalization to data recorded outside the fMRI scanner.

Fig.1B showcases the training convergence for DS 2, while Fig.1C displays t-SNE projections of learned representations for a representative CV split. Maximizing geodesic correlation between latent EEG and fMRI representations consistently preserved task structure. Employing explainable AI techniques (Fig.1D), we confirmed that the representations reflect task-relevant brain activity. The EEG decoder utilized power modulations in the alpha band within a large-scale network spanning frontal and parietal regions. For the fMRI decoder, we report self-attention maps that summarize the averaged self-attention layer output of the Transformer network per condition. They indicate engagement of executive (2-back) and saliency (0-back) networks. For DS1 (eyes open vs. closed), the latent representations reflect activity of occipital networks.
Fig.2 lists CV test set scores. Moderate to high correlation scores indicate a robust coupling between the latent EEG and fMRI representations that generalizes to unseen data. Unimodal classification performance for distinguishing task conditions confirm that our end-to-end learning approach utilized task information, encoded in both EEG and fMRI, to fuse the latent representations.

We introduced a data-driven approach to fuse EEG-fMRI data and studied two datasets involving cognitive and resting conditions. The approach effectively preserved task-related information in the latent EEG and fMRI representations with the underlying neural patterns linked to specific cognitive processes. The approach exhibited promise in generalizing to simultaneous EEG-fMRI data, and even to EEG data of unseen subjects recorded outside the fMRI scanner.

Activation (eg. BOLD task-fMRI)

EEG/MEG Modeling and Analysis

Methods Development ²

Multivariate Approaches

Multi-Modal Imaging ¹

Data analysis

Electroencephaolography (EEG)

FUNCTIONAL MRI

Machine Learning

Multivariate