Whole-brain fMRI Pattern Extractor: 3D Variational Autoencoder for Sensorimotor Classification
Poster No:
1442
Submission Type:
Abstract Submission
Authors:
Ji-Hye Lim1, Jong-Hwan Lee2, Hyun-Chul Kim1
Institutions:
1Kyungpook National University, Daegu, Korea, Republic of, 2Korea University, Seoul, Korea, Republic of
First Author:
Co-Author(s):
Introduction:
By extracting representation features from whole-brain fMRI datasets and inputting them into machine learning models, it is possible to alleviate an overfitting issue in fMRI classification tasks. Since the complex brain patterns occurring in the brain are not derived from a single cause but from multiple factors, it is desirable to use 3D Variational Autoencoder (3D-VAE) that can well explain non-linearity [1]. We hypothesize that Gaussian sampling in the latent space of 3D-VAE can produce a diverse array of plausible brain activity patterns, potentially improving classification task performance. Thus, this study aims to explore if features derived from 3D-VAE improve performance in four sensorimotor classification tasks.
Methods:
Twelve right-handed male individuals (age = 25.0 ± 2.0 years) participated in four sensorimotor tasks: auditory attention, left-hand clenching, right-hand clenching, and visual stimulation. Each activity began with a 30-second period of focusing on a cross, followed by three task blocks of 20 seconds each, interspersed with 20-second periods of cross-fixation. The fMRI data were processed using a standardized protocol in SPM [2]. Task-specific volumes (repetition time = 2 s; 30 volumes for each task; totaling 1,440 volumes across all participants) were subsequently prepared for a four-category classification task.
For extracting representation features from the whole-brain fMRI data, we employed a 3D-VAE model. Subsequently, we carried out four sensorimotor classification tasks to verify the robustness of these representations using a Multi-Layer Perceptron (MLP) classifier (including two hidden layers). The classification performance was assessed for leave-one-subject-out CV (LOOCV). More specifically, the training dataset was used to develop a 3D-VAE model featuring a 3D Convolutional Neural Network (3D-CNN), included in both encoder and decoder components. This model was designed to extract latent representations in lower-dimensional space. In particular, the representations used the z latent calculated through the reparameterization trick for backpropagation. The representation features derived from the training dataset were used to train a MLP classifier, while features from the test dataset were used to evaluate the classification performance. Also, we visualized the reconstructed fMRI pattern using the Decoder of the 3D-VAE to explore features related to the task, further enhancing implementation of the pattern captured by the model.
For extracting representation features from the whole-brain fMRI data, we employed a 3D-VAE model. Subsequently, we carried out four sensorimotor classification tasks to verify the robustness of these representations using a Multi-Layer Perceptron (MLP) classifier (including two hidden layers). The classification performance was assessed for leave-one-subject-out CV (LOOCV). More specifically, the training dataset was used to develop a 3D-VAE model featuring a 3D Convolutional Neural Network (3D-CNN), included in both encoder and decoder components. This model was designed to extract latent representations in lower-dimensional space. In particular, the representations used the z latent calculated through the reparameterization trick for backpropagation. The representation features derived from the training dataset were used to train a MLP classifier, while features from the test dataset were used to evaluate the classification performance. Also, we visualized the reconstructed fMRI pattern using the Decoder of the 3D-VAE to explore features related to the task, further enhancing implementation of the pattern captured by the model.
Results:
We observed that the 3D-VAE, featuring encoder network with convolution layers of 5 and a latent space dimensions of 256, achieved a mean error rate achieved 1.81±1.66 %, using LOOCV. This represents a decrease of 0.29% in the error rate compared to the LOOCV of a 3D-CNN using whole brain data without extracting representation using 3D-VAE [2]. Visualizing the representations extracted from each test subject with t-distributed stochastic neighbor embedding in LOOCV demonstrated effective subject identification and task-specific clustering within subjects. Furthermore, the decoded fMRI patterns based on the employed representations revealed brain patterns specific to the sensorimotor tasks.
Conclusions:
We discovered that the 3D-VAE can extract crucial representations in a lower latent space from the whole-brain, and these representations have demonstrated enhanced performance in the classification of four sensorimotor tasks. Future work is warranted to investigate the efficacy of the proposed approach in different classification tasks (e.g., working memory, emotion, social, language, gambling, relational) using Human Connectome Project datasets.
Modeling and Analysis Methods:
Activation (eg. BOLD task-fMRI) 2
Classification and Predictive Modeling 1
Keywords:
FUNCTIONAL MRI
Other - Classification, Sensorimotor, 3D Convolutional Neural Network, Variational Autoencoder
1|2Indicates the priority used for review
Provide references using author date format
References
[1] Kim, J. H., Zhang, Y., Han, K., Wen, Z., Choi, M., & Liu, Z. (2021). Representation learning of resting state fMRI with variational autoencoder. NeuroImage, 241, 118423.
[2] Vu, H., Kim, H. C., Jung, M., & Lee, J. H. (2020). fMRI volume classification using a 3D convolutional neural network robust to shifted and scaled neuronal activations. NeuroImage, 223, 117328.