RepL-fMRI: A Python Library for fMRI Representation Learning

1940 

Abstract Submission 

Yaoxin Li¹, Kuan Han², Owen MacKenzie², Minkyu Choi², Zhongming Liu², Scott Peltier²

¹Michigan Neuroscience Institute, University of Michigan, Ann Arbor, MI, ²University of Michigan, Ann Arbor, MI

Yaoxin Li  
Michigan Neuroscience Institute, University of Michigan
Ann Arbor, MI

Kuan Han  
University of Michigan
Ann Arbor, MI

Owen MacKenzie  
University of Michigan
Ann Arbor, MI

Minkyu Choi  
University of Michigan
Ann Arbor, MI

Zhongming Liu  
University of Michigan
Ann Arbor, MI

Scott Peltier  
University of Michigan
Ann Arbor, MI

Deep neural networks hold the unique potential for learning low-dimensional representations of high-dimensional fMRI data and supporting functional interpretation of brain activity linked to behavior (Kim 2021; Han 2019). Despite growing literature and large-scale datasets (Ortega 2023; Thomas 2022; Van Essen 2013; Littlejohns 2020; Casey 2018; Markiewicz 2021), there remains a lack of a streamlined pipeline for deep learning in fMRI analysis. To address this gap, we introduce RepL-fMRI: a Python library for building modular, interoperable, and interpretable deep neural networks for fMRI representation learning. This library encapsulates the open-source and pre-trained models for fMRI analysis developed through our prior work, whilst providing a means for extension through containerized software and a web user interface.

The RepL-fMRI package includes a pre-trained deep foundation model, which converts fMRI data into latent representations, and add-on shallow networks, which further link the representations to human behavior phenotypes. The foundation model uses an encoder-decoder architecture, including two stages of processing: 1) identifying spatial patterns and 2) extracting temporal sequences. The first stage is based on a variational autoencoder (VAE) (Kim 2021). The second stage is based on bi-directional transformer encoder (BERT) (Han 2019). Both of these stages contain paired encoder and decoder models, allowing for compression of brain data into latent representations and decompression of these vectors back to aid analysis of brain networks and dynamics. The foundation model is pre-trained with resting state fMRI data from the Human Connectome Project (HCP) (Van Essen 2013). The embeddings output by the foundation model are used to train a multi-layer perceptron (MLP) model that classifies cognitive and emotional traits from the rich and compressed tokens.

Using Python, we define classes to encapsulate these models, as well as their variations, extensions, or linear counterparts. The classes follow the convention of object-oriented programming and build upon a generic set of interfaces for using various models as plug-and-play modules in both forward (encoding) and reverse (decoding) inference. The library also encapsulates existing standards in fMRI preprocessing, including the fMRIPrep (Esteban 2019) and HCP minimal preprocessing pipelines (Glasser 2013). As a framework, RepL-fMRI provides intuitive facilities to manage application of deep learning modules in fMRI analysis.

We tested the Repl-fMRI framework for encoding 2-D (cortical) and 3-D (whole-brain) fMRI activity patterns into an evolving trajectory of latent representation. Through the foundation model, the representations of resting state fMRI activity were distinguishable across individuals (Figure 2a). The model was generalizable to task states, showing comparable encoding and decoding accuracies for both resting and task states. Representations can reveal transitions between different brain states (e.g. from rest to movie watching, Figure 2a). The pre-trained foundation model alongside MLP add-ons could predict the cognitive and emotional traits in terms of both their first principal components and individual measures (see Figure 2b).

Repl-fMRI is an open-source infrastructure to streamline the process of building and using deep learning models as interoperable modules in human fMRI analysis. Its current form and future extensions will contribute to the broader ecosystem for reliable and reproducible fMRI studies, especially for brain-behavior associations.

Methods Development ¹

Informatics Other ²

Data analysis

FUNCTIONAL MRI

Machine Learning