Multimodal connectivity-based graph transformer networks and its application to sex classification

1433 

Abstract Submission 

Hye Won Park¹, Won Hee Lee¹

¹Kyung Hee University, Yongin, Republic of Korea

Hye Won Park  
Kyung Hee University
Yongin, Republic of Korea

Won Hee Lee  
Kyung Hee University
Yongin, Republic of Korea

The integration of multimodal brain imaging data and the use of graph-based approaches have emerged as powerful tools for gaining deep insights into the complex relationships and topological characteristics of brain connectivity [1, 2]. The combination of structural connectivity (SC) derived from diffusion tensor imaging (DTI) and functional connectivity (FC) obtained from functional MRI (fMRI) provides a comprehensive representation of brain connectivity, encompassing both the physical connections between brain regions and their dynamic interactions [3]. By analyzing these modalities in conjunction, researchers can uncover brain patterns that may be obscured when examining each modality in isolation. In this context, we propose a new method that leverages a graph transformer network (GTN) to fuse multimodal connectivity information, and evaluate its performance on sex classification. We also identify important nodes to provide insights into the roles of specific brain regions in the sex classification task.

We used SC and resting-state FC data of 753 healthy individuals derived from the Human Connectome Project (HCP) [4, 5]. Cortical regions were defined based on the 400-region Schaefer parcellation [6]. SC was computed using fiber tractography and a log transformation of stream counts was applied. FC was generated using Pearson's correlation of fMRI signals between regions of interest (ROIs). Figure 1(a) shows the pipeline of the proposed model. For each subject, both SC and FC graphs were created, where rows and columns represent nodes and their features, respectively. The edges were derived from Euclidean distances between ROI pairs of MNI 152 centroid coordinates. A Gaussian kernel was employed to distances, retaining the top 1% of connections. Both graphs were trained with the graph transformer [7], followed by batch normalization and parametric ReLU. To fuse graphs, we combined the nodes of two graphs using learnable weights (θ₁ and θ₂) using weighted summation. The edges and their features in a fusion graph were trained using XGBoost (Figure 1(b)). SHAP [8] was employed to assess feature importance, retaining the top 1% of connections. Edge features were defined by concatenating the filtered SC, FC, and centroid distances. After applying adjacency dropout, the GTN updated node features considering edge features. Concatenating pooling was applied to merge features from all nodes, followed by sex classification using fully connected layers. Salient nodes were identified using GNNExplainer [9]. The model was evaluated using a 10-fold cross-validation and compared to single modality-based GTN models.

Our proposed multimodal connectivity model shows superior performance compared to the single modality model (Figure 2(a)). The average accuracy of our model reached 95.76%, outperforming both the single SC (88.71%) and the single FC model (90.04%). Figure 2(b) shows the top 10 brain regions contributing to the sex classification. The most salient regions in the fusion model included the left temporal, the left inferior parietal lobule, and the left temporal pole for SC, and the left temporal, the left lateral prefrontal cortex, and the left inferior parietal lobule for FC. These findings indicate the significance of integrating information from both SC and FC modalities, as different regions contribute uniquely to the sex classification task in each modality.

We proposed a new GTN-based multimodal fusion model that effectively integrates the FC and SC modalities to capture complex brain connectivity patterns. Our proposed model achieved high accuracy in the sex classification task and provided interpretable insights into the key brain regions contributing to the model's decisions. This study contributes to advancing our understanding of brain connectivity and lays the groundwork for future studies aiming to enhance the robustness and interpretability of models in neuroimaging research.

Classification and Predictive Modeling ¹

Connectivity (eg. functional, effective, structural) ²

fMRI Connectivity and Network Modeling

Other - Graph Transformer Networks;Brain Connectivity;Multimodal Fusion Model