DL joint-fusion model with explanation shows the interplay between sMRI and fMRI in the study of ASD
Poster No:
1452
Submission Type:
Abstract Submission
Authors:
Francesca Lizzi1, Sara Saponaro2, Giacomo Serra3, Francesca Mainas4, Piernicola Oliva4, Alessia Giuliano5, Sara Calderoni6, Alessandra Retico7
Institutions:
1Pisa Division of National Institute for Nuclear Physics (INFN), pisa, Italy, 2Pisa Division of National Institute for Nuclear Physics (INFN), Pisa, Italy, 3National Institute of Nuclear Physics (INFN), Cagliari, Italy, 4National Institute of Nuclear Physics (INFN), Cagliari , Italy, 5Unit of Medical Physics, Pisa University Hospital "Azienda Ospedaliero-Universitaria Pisana", Pisa, Italy, 6Developmental Psychiatry Unit – IRCCS Stella Maris Foundation, Pisa, Italy, 7National Institute of Nuclear Physics (INFN), Pisa, Italy
First Author:
Co-Author(s):
Alessia Giuliano
Unit of Medical Physics, Pisa University Hospital "Azienda Ospedaliero-Universitaria Pisana"
Pisa, Italy
Unit of Medical Physics, Pisa University Hospital "Azienda Ospedaliero-Universitaria Pisana"
Pisa, Italy
Introduction:
The development of multi-modal models that used data from different modalities has the potential to advance precision medicine. Combining data from multiple modalities allows the extraction of more comprehensive and complementary information, resulting in the creation of better-performing models. Additionally, this approach reveals important relationships that may cannot be detected when relying on a single modality.
In this work, we develop a joint fusion deep learning (DL) model to combine harmonized structural and functional MRI features trained to discriminate subjects with Autism Spectrum Disorders (ASD) with respect to typically developing controls (TD). Additionally, we implement the SHapley Additive exPlanations (SHAP) explainability framework to identify the most significant features contributing to the classification of ASD subjects.
In this work, we develop a joint fusion deep learning (DL) model to combine harmonized structural and functional MRI features trained to discriminate subjects with Autism Spectrum Disorders (ASD) with respect to typically developing controls (TD). Additionally, we implement the SHapley Additive exPlanations (SHAP) explainability framework to identify the most significant features contributing to the classification of ASD subjects.
Methods:
We analyzed both structural and functional MRI brain scans publicly available within the ABIDE I and II data collections [2,3]. We considered 1383 male subjects with age between 5 and 40 years, including 680 subjects with ASD and 703 TD from 35 different acquisition sites. Due to the multisite nature of the dataset, we separately harmonized the Freesurfer structural features and the functional connectivity measures using the NeuroHarmonize package [4] as reported in [5].
The ASD vs. TD classification was carried out with a DL model, consisting in a feature dimensionality reduction neural network (FR-NN) and a classification neural network (C-NN). The FR-NN generates a fixed-length feature representation for each data modality. Specifically, we implemented the joint fusion approach [1], which propagates the loss back to the FR-NN during training, allowing the creation of informative feature representations for each data modality.
Additionally, we implemented single data modality-based models to assess the potential enhancement offered by using a multimodal model. The single-modality models exclusively considered either structural or connectivity features and utilized similar neural networks for classification. The models were training for 150 epochs, applying standard deep learning techniques to mitigate overfitting.
The performance was evaluated by computing the Area under the Receiver Operating Characteristic curve (AUC) within a nested 10-fold cross-validation, preserving the matching proportions of ASD and TD diagnoses.
To identify the most significant features able to discriminate between ASD and TD subjects, we selected features with scores exceeding the 99th percentile among those determined by SHAP as important. Additionally, we quantified the effect size of the ASD vs. TD group difference using Cohen's d coefficient.
The ASD vs. TD classification was carried out with a DL model, consisting in a feature dimensionality reduction neural network (FR-NN) and a classification neural network (C-NN). The FR-NN generates a fixed-length feature representation for each data modality. Specifically, we implemented the joint fusion approach [1], which propagates the loss back to the FR-NN during training, allowing the creation of informative feature representations for each data modality.
Additionally, we implemented single data modality-based models to assess the potential enhancement offered by using a multimodal model. The single-modality models exclusively considered either structural or connectivity features and utilized similar neural networks for classification. The models were training for 150 epochs, applying standard deep learning techniques to mitigate overfitting.
The performance was evaluated by computing the Area under the Receiver Operating Characteristic curve (AUC) within a nested 10-fold cross-validation, preserving the matching proportions of ASD and TD diagnoses.
To identify the most significant features able to discriminate between ASD and TD subjects, we selected features with scores exceeding the 99th percentile among those determined by SHAP as important. Additionally, we quantified the effect size of the ASD vs. TD group difference using Cohen's d coefficient.
Results:
The AUC values of 0.66±0.05 and of 0.76±0.04 were obtained in the ASD vs. TD discrimination when only structural or functional features are considered, respectively. The joint fusion approach resulted in an AUC of 0.78±0.04.
The features identified as crucial in discriminating between ASD and TD subjects primarily originated from functional MRI data. Additionally, we observed a pattern of reduced long-range inter-hemispheric connectivity and increased intra-hemispheric connectivity in ASD subjects compared to TDs. Finally, the set of features identified as the most important for the two-class discrimination supports the idea that brain changes tend to occur in individuals with ASD in regions belonging to the Default Mode Network and to the Social Brain.
The features identified as crucial in discriminating between ASD and TD subjects primarily originated from functional MRI data. Additionally, we observed a pattern of reduced long-range inter-hemispheric connectivity and increased intra-hemispheric connectivity in ASD subjects compared to TDs. Finally, the set of features identified as the most important for the two-class discrimination supports the idea that brain changes tend to occur in individuals with ASD in regions belonging to the Default Mode Network and to the Social Brain.
Conclusions:
Our results demonstrate that the DL-based joint fusion approach outperforms the other ones as it efficiently exploits the complementary information related to the ASD diagnosis contained in sMRI and rs-fMRI images. Furthermore, this work suggests that multi-modality DL models are promising tools for identifying potential neuroimaging biomarkers of neurodevelopmental disorders.
Acknowledgments: FAIR-AIM project (POR FSE 2014-2020) and PNRR - M4C2 - PE "FAIR - Future Artificial Intelligence Research"
Acknowledgments: FAIR-AIM project (POR FSE 2014-2020) and PNRR - M4C2 - PE "FAIR - Future Artificial Intelligence Research"
Disorders of the Nervous System:
Neurodevelopmental/ Early Life (eg. ADHD, autism) 2
Modeling and Analysis Methods:
Classification and Predictive Modeling 1
Keywords:
Autism
Data analysis
FUNCTIONAL MRI
Machine Learning
STRUCTURAL MRI
Other - Deep Learning; Explainable AI
1|2Indicates the priority used for review