Spectral introspection identifies group training dynamics in deep neural networks in neuroimaging

1903 

Abstract Submission 

Bradley Baker¹, Vince Calhoun², Sergey Plis³

¹TReNDs, Atlanta, GA, ²GSU/GATech/Emory, Decatur, GA, ³Georgia State University, Atlanta, GA

Bradley Baker, PhD  
TReNDs
Atlanta, GA

Vince Calhoun  
GSU/GATech/Emory
Decatur, GA

Sergey Plis  
Georgia State University
Atlanta, GA

In this work, we present a novel empirical method for analyzing Deep Neural Network (DNN) learning dynamics, which builds off of previous theoretical work studying changes in gradient rank in auto-differentiation. Our method utilizes the singular values of the gradient and of the component matrices which are used to compute to study evolution of dominant modes during model training. Furthermore, because we analyze gradients dynamically within auto-differentiation, we have the unique opportunity to analyze these dynamics as they adhere to individual samples. Thus, whenever samples have some kind of group labelling, we can perform statistical comparisons of gradient trajectories between groups without breaking normal training behavior. This further allows our method to stand out from post-hoc methods which not only occur outside of training, but can only be evaluated for between different classes in disjoint contexts.

To demonstrate the applicability of our method, we identify distinct training dynamics specific to Major Depressive Disorder (MDD), Bipolar Disorder (BPD), Schizophrenia, and Schizoaffective disorders across multiple studies. We show how these dynamics differ between popular model architectures used in neuroimaging, including convolutional neural networks, transformers and more.

Reverse-mode auto-differentiation is a computational technique for computing the partial derivatives of complex functions, and has been widely applied to deep learning optimization in the form of back-propagation. Gradients for a neural network's weights are computed by first performing a "forward pass" through the network, and recording the input activations at each layer. Then, a "backward pass" is performed in which the partial derivatives on the output neurons are computed and fed into higher layers. The gradients at each layer are thus computed as an outer product between these accumulated input activations and partial derivatives with respect to the output.

To analyze the spectrum of the gradient during training, we compute the singular value decomposition (SVD) of the gradient and its constituent matrices. Because we are working within AD, we can computed the SVD of gradients belonging to particular samples, and thus to particular groups. We then perform statistical tests between singular value trajectories during training to evaluate group differences.

We begin with a Multi-Layer perceptron on FreeSurfer volumes. We then analyze functional MRI from the COBRE and FBIRN data sets. To assess dynamics in recurrent models, we perform Spatially Constrained Independent Component Analysis using the NeuroMark template, which provides us with 53 neurologically relevant spatially independent maps and associated time-series. Using the time-series data, we show can reveal group-specific gradient dynamics in 1D and 2D CNNs, LSTMs and the BERT transformer. We then utilize the spatial maps to demonstrate group-specific gradient dynamics in 3D-CNNs.

The MLP autoencoder trained on FSL data shows significant differences between SZ and HC groups in the middle singular values of the output layer, while the corresponding classifier shows more group differences in the input layer. The LSTM, BERT and 1D CNN models all show significant differences between male SZ and HC groups, with the LSTM showing differences mostly in hidden neurons, the 1D CNN showing significant differences at the output layer, and BERT showing differences across the entire model for the autoencoder task with almost no significant differences in the classifier.

In this work, we have demonstrated a novel, dynamic introspection technique for DNNs applied to neuroimaging. Our method utilizes inherent structure of auto-differentiation to provide analysis which does not affect training, and we can use to compare group differences. We applied the method to multiple neuroimaging studies and demonstrate how group differences appear in different places during training and in the architecture.

Psychiatric (eg. Depression, Anxiety, Schizophrenia)

Classification and Predictive Modeling ²

Exploratory Modeling and Artifact Removal

Methods Development ¹

Data analysis

Machine Learning

Modeling

Psychiatric Disorders