Generalization performance of sex classification models to multiple datasets

1447 

Abstract Submission 

Lisa Wiersch^1,2, Patrick Friedrich^1,2, Sami Hamdan^1,2, Vera Komeyer^1,2, Felix Hoffstaedter^1,2, Kaustubh Patil^1,2, Simon Eickhoff^1,2, Susanne Weis^1,2

¹Institute of Systems Neuroscience, Heinrich Heine University Düsseldorf, Düsseldorf, Germany, ²Institute of Neuroscience and Medicine (INM-7), Research Centre Jülich, Jülich, Germany

Lisa Wiersch  
Institute of Systems Neuroscience, Heinrich Heine University Düsseldorf|Institute of Neuroscience and Medicine (INM-7), Research Centre Jülich
Düsseldorf, Germany|Jülich, Germany

Patrick Friedrich  
Institute of Systems Neuroscience, Heinrich Heine University Düsseldorf|Institute of Neuroscience and Medicine (INM-7), Research Centre Jülich
Düsseldorf, Germany|Jülich, Germany

Sami Hamdan  
Institute of Systems Neuroscience, Heinrich Heine University Düsseldorf|Institute of Neuroscience and Medicine (INM-7), Research Centre Jülich
Düsseldorf, Germany|Jülich, Germany

Vera Komeyer  
Institute of Systems Neuroscience, Heinrich Heine University Düsseldorf|Institute of Neuroscience and Medicine (INM-7), Research Centre Jülich
Düsseldorf, Germany|Jülich, Germany

Felix Hoffstaedter  
Institute of Systems Neuroscience, Heinrich Heine University Düsseldorf|Institute of Neuroscience and Medicine (INM-7), Research Centre Jülich
Düsseldorf, Germany|Jülich, Germany

Kaustubh Patil  
Institute of Systems Neuroscience, Heinrich Heine University Düsseldorf|Institute of Neuroscience and Medicine (INM-7), Research Centre Jülich
Düsseldorf, Germany|Jülich, Germany

Simon Eickhoff  
Institute of Systems Neuroscience, Heinrich Heine University Düsseldorf|Institute of Neuroscience and Medicine (INM-7), Research Centre Jülich
Düsseldorf, Germany|Jülich, Germany

Susanne Weis  
Institute of Systems Neuroscience, Heinrich Heine University Düsseldorf|Institute of Neuroscience and Medicine (INM-7), Research Centre Jülich
Düsseldorf, Germany|Jülich, Germany

Machine Learning (ML) methods are a powerful tool increasingly applied for studying phenotypes based on neuroimaging data. Knowing to what extent the choice of training samples influences the generalizability of ML-models is crucial to achieve most accurate model performance in across-sample predictions. The present study investigates the influence of the choice of sample for sex classification analyses based on resting-state functional connectivity (RSFC) in a variety of cohorts differing in sample size, age range and imaging quality.

We employed samples from four different independent cohorts: HCP (N = 878, age range = 22-37, mean age: 28.49; [1]), GSP (N = 854, age range = 21-35, mean age: 22.92; [2]), eNKI (N = 190, age range = 20-83, mean age: 46.02; [3]) and 1000Brains (N = 1000, age range = 21-85, mean age: 61.18; [4]). For each sample, we included healthy subjects aged 20 years or older with a similar number of men and women who were matched for age. Sex classification models were trained on the data of each of the four cohorts individually and a compound sample comprising 75% data of all four samples (N = 2190, age range = 20-85, mean age: 40.10). Following the parcelwise approach by Weis et al. (2020; [5]), we trained sex classifiers on the parcelwise RSFC profile of 436 parcels individually resulting in five sets of parcelwise Classifiers (pwCs; [6]). Each of the classification models was trained with a Support Vector Machine Classifier implemented in Julearn [7]. All five pwCs were applied on test samples derived from the four original samples. In addition, out-of-sample performance of all pwCs was evaluated on data extracted from the AOMIC dataset (N = 370, age range: 20-26, mean age: 22.50) which was not included in training any of the pwCs.

For pwCs trained on single samples, pwC HCP demonstrated highest mean cross-validation (CV) accuracy averaged across all 436 parcels (figure 1a). However, pwC HCP showed lowest generalization in across-sample predictions with mean accuracies ranging between 52% (eNKI) and 55% (1000Brains). In contrast, pwC GSP, pwC eNKI and pwC 1000Brains achieved lower mean CV accuracies than pwC HCP but higher generalization performance for across-sample predictions (figure 1a). Highest classifying parcels per model application were consistently located in the temporal lobe, inferior parietal lobule, posterior cingulate gyrus and inferior frontal gyrus (figure 2a).
Except for pwC HCP, pwC compound achieved higher mean CV and generalization performance than pwCs trained on single samples: Mean accuracies in across-sample predictions ranged between 61% (HCP test sample) and 65% (GSP test sample, figure 1b) with up to 83% accuracy (eNKI test sample, figure 2b). Highest classifying parcels were located in similar regions as for pwCs trained on single samples (figure 2b). Likewise, pwC compound also showed highest generalization performance for the out-of-sample prediction of the AOMIC sample with a mean accuracy of 59% (figure 1c) with highest classifying parcels - up to 69% - being located in the inferior parietal lobule, inferior frontal gyrus and posterior cingulate gyrus (figure 2c).

The present results showed that classifiers trained on single samples generalized well for some, but not all, test samples. Specifically, classifiers trained on one sample performed well on a certain other sample, and vice versa as e.g. HCP & 1000Brains as well as eNKI & GSP. In contrast, pwC trained on the compound sample outperformed single sample trained classifiers in classification accuracy. This was not only the case for the test samples, parts of which were used to train pwC compound, but also for the independent AOMIC sample. Overall, the aggregation of multiple samples for training sex classification models resulted in superior generalization performance which could be attributed to the high sample size but also the heterogeneous data composition of pwC compound.

Classification and Predictive Modeling ¹

Connectivity (eg. functional, effective, structural) ²

Computing

Data analysis

FUNCTIONAL MRI

Machine Learning

Statistical Methods

Other - sex classification