Unlocking the Power of Distributed Learning for MeshNet

2313 

Abstract Submission 

Pratyush Reddy Gaggenapalli¹, Mohamed Masoud¹, Farfalla Hu¹, Sergey Plis¹

¹Georgia State University, Atlanta, GA

Pratyush Reddy Gaggenapalli  
Georgia State University
Atlanta, GA

Mohamed Masoud  
Georgia State University
Atlanta, GA

Farfalla Hu  
Georgia State University
Atlanta, GA

Sergey Plis  
Georgia State University
Atlanta, GA

Decentralised learning, unlike the centralised approach, grapples with private data distributed across various autonomous sites rather than a singular server. This paradigm shift offers heightened privacy [9] and scalability by harnessing multiple device capabilities, particularly crucial in sensitive sectors like healthcare and neuroimaging. However, challenges arise due to increased message sizes, exacerbated by bandwidth limitations, causing latency spikes in large model exchanges. This predicament intensifies for deep learning models, where greater accuracy demands larger parameter reconciliation across sites. Conventional remedies like sparsity, distillation, and quantization often sacrifice accuracy for reduced parameter sizes. In our study, we employed MeshNet [1-3] figure 1a, an full brain volumetric segmentation[5] model, in its original form without any special compression but controlling number of channels, establishing a distributed learning system. Encouragingly, our outcomes exhibit strides in achieving balanced node training, ensuring accuracy, scalability, and resource optimization.

Centralised Gradient Aggregation fig 2a is a groundbreaking approach in the distributed learning landscape, overcoming challenges inherent in decentralised data processing across multiple nodes. Focused on data privacy and computational efficiency, this method transforms the paradigm by enabling nodes to process local data batches, generating gradients that encapsulate unique insights. These gradients converge at a centralised hub, creating a comprehensive model performance overview. This collective narrative guides iterative model updates, fostering collaboration irrespective of geographical or dataset disparities. The synchronised refinement cycle allows nodes to learn not just from local data but from the distributed network's collective wisdom, enhancing individual models and promoting shared learning for holistic progress.The incorporation of Coinstac, alongside the integration of WADB for metric logging, constitutes a pivotal advancement in optimising distributed training environments. Coinstac streamlines communication, reducing overhead, and significantly enhancing training efficiency across multiple nodes. This tool democratises the adoption of distributed training by simplifying fault tolerance and widening accessibility. The integration of WADB for metric fig 2b logging provides a comprehensive monitoring system, ensuring precise tracking of training progress and facilitating informed decision-making. This standardised methodology optimises training efficiency, democratising networked nodes utilisation and shaping a more inclusive and robust machine learning landscape across diverse domains.

Our experiments, conducted in the Coinstac decentralised simulator[10,11] figure 1a, ensure equitable task distribution among nodes, fostering balanced contributions to model optimization. Logging batch-specific metrics for each node provides valuable insights, enhancing overall training efficiency and node-specific dynamics. Figure 2 Table 1 compares Cross-Entropy Loss for standard and decentralised training of MeshNet and CNN models on the Cifar10 dataset, revealing their convergence and accuracy across epochs, showcasing the effectiveness of our decentralised approach in optimising model performance.

Our progress in decentralised MeshNet learning signifies significant advancements in balanced node training. Collaborating with Coinstac[10] amplifies our approach, enriching distributed learning strategies. This ongoing project carries immense potential to revolutionise collaborative learning paradigms, guaranteeing both efficiency and scalability in the future.

Classification and Predictive Modeling ²

Anatomical MRI ¹

Machine Learning

MRI

Other - distributed