Reliability and robustness in static and dynamic FC predictions

1468 

Abstract Submission 

Christine Ahrends¹, Mark Woolrich², Diego Vidaurre³

¹Aarhus University, Aarhus, Aarhus, ²University of Oxford, Oxford, Oxfordshire, ³Aarhus University, Aarhus, Aarhus C

Christine Ahrends  
Aarhus University
Aarhus, Aarhus

Mark Woolrich  
University of Oxford
Oxford, Oxfordshire

Diego Vidaurre  
Aarhus University
Aarhus, Aarhus C

Predicting individual traits from brain measures is a major goal in modern neuroscience. Many studies aim to use either time-averaged (static) or time-varying (dynamic) estimates of functional connectivity (FC) for these predictions. While the focus of this work is mainly to increase prediction accuracy, it is crucial to consider the reliability and robustness of different approaches (1). This is particularly important if we want to be able to meaningfully interpret model errors, such as to estimate brain age (2). For instance, a model may predict accurately in some cases but make excessive errors in other cases (poor reliability), or predict generally well, but predict poorly when the training set contains outliers (poor robustness). We here compare a range of different methods using either time-averaged or time-varying estimates of FC in terms of reliability and robustness.

We use the Human Connectome Project (HCP) S1200 dataset (3), where we estimate FC on resting-state fMRI and predict individual cognitive traits. We then compare four methods using time-averaged FC and seven methods using time-varying FC for the prediction. For time-averaged FC, we use two variants of an Elastic Net model (4), one in Euclidean and one in Riemannian space (5), a Selected Edges model where relevant edges of the time-averaged FC matrices are first selected and then used as predictors (6), and a model based on Kullback-Leibler (KL) divergence (7). For time-varying FC prediction, we use a Hidden Markov Model (8) and construct seven different kernels from it using different projections of the individual-level parameters (9). We use kernel ridge regression for the kernel-based models and ridge regression for the other models. All models use nested 10-fold cross validation (CV), and folds are constructed accounting for family structure in the dataset. We compare the methods using two criteria: reliability and robustness. To assess reliability, we compute the normalised maximum absolute errors. This indicates whether the single largest error in the predicted values exceeds the range of the original variable by orders of magnitude. To assess robustness, we run each model 100 times, each time randomising CV folds, i.e., randomising which combinations of subjects the model is trained and which subjects the model is tested on. We then consider the standard deviation across prediction accuracies across these 100 iterations, where low standard deviation indicates high robustness.

In the time-averaged FC models, all but the KL divergence-based model have high reliability and robustness. In the time-varying FC models, the linear kernels are generally more reliable and robust than the Gaussian kernels, with the linear naïve normalised kernel and the linear Fisher kernel performing significantly better than other kernels. The models based on KL divergence (both time-averaged and time-varying FC) were overall the most problematic ones, both in terms of reliability and robustness. Comparing the time-averaged and the time-varying FC models, the better models perform similarly in terms of reliability, but the time-varying FC models outperform the time-averaged FC models in terms of robustness. These models also have a higher prediction accuracy than the time-averaged FC models.

We here proposed an approach to assess reliability and robustness, two important criteria for studies aiming to predict individual traits from FC. We showed that the best-performing methods in terms of reliability and robustness are the Elastic Net and the Selected Edges approach for time-averaged FC and the linear naïve normalised and linear Fisher kernel for time-varying FC. Overall, the time-varying FC models outperformed the time-averaged FC models. By considering not only prediction accuracy, but also the models' reliability and robustness, we may be able to move further towards interpreting model errors and using predictive models based on FC in clinical contexts.

Classification and Predictive Modeling ¹

fMRI Connectivity and Network Modeling ²

Methods Development

Task-Independent and Resting-State Analysis

Data analysis

FUNCTIONAL MRI

Machine Learning

Modeling