Applying Neural Style Transfer in Generative Adversarial Nets for Multi-site MRI Harmonization

1688 

Abstract Submission 

Yongseon Yoo¹, Seonggyu Kim², Hanyeol Yang¹, Jihwan Min¹, Jong-Min Lee^1,2

¹Department of Artificial Intelligence, Hanyang University, Seoul, Korea, Republic of, ²Department of Electronic Engineering, Hanyang University, Seoul, Korea, Republic of

Yongseon Yoo  
Department of Artificial Intelligence, Hanyang University
Seoul, Korea, Republic of

Seonggyu Kim  
Department of Electronic Engineering, Hanyang University
Seoul, Korea, Republic of

Hanyeol Yang  
Department of Artificial Intelligence, Hanyang University
Seoul, Korea, Republic of

Jihwan Min  
Department of Artificial Intelligence, Hanyang University
Seoul, Korea, Republic of

Jong-Min Lee  
Department of Artificial Intelligence, Hanyang University|Department of Electronic Engineering, Hanyang University
Seoul, Korea, Republic of|Seoul, Korea, Republic of

MRI brain scans are essential for diagnosing neurological diseases, and deep learning models trained on data from multiple sites face performance issues due to differences in MR images, known as "scanner effect." To mitigate this, "harmonization" techniques are used, with recent advances in generative models showing better performance in minimizing these variations compared to traditional methods[9, 8]. We plan to apply the Style Transfer technique using GANs for the brain MRI harmonization task across different sites. In GAN models, the AdaIN (Adaptive Instance Normalization)[4] method is used for applying style[6], which changes the first order statistics to impart style. We have enhanced our generative model (GAN) to better learn the style of target MRI images by incorporating the loss function technique from Neural Style Transfer (NST)[2] that considers second order statistics[5] using the Gram Matrix. This approach allows our model to consider both first and second order statistics in the process of learning and applying style. Additionally, to preserve the anatomical structure of the original MRI images, we have incorporated a loss function that reduces the difference in feature maps between the original and generated images, in addition to the existing cycle consistency loss. Our method applies Neural Style Transfer to GANs for performing cross-site brain MRI harmonization, offering an enhanced style application approach compared to existing methods.

We propose a method to enhance GANs by adding loss functions from Neural Style Transfer[2]. We employ the StarGANv2[1, 7] model for this purpose. To calculate NST's style and content loss, we integrate a pre-trained VGG19[3] into the existing GAN model structure. The style loss is computed by passing the reference image and the generated target image through VGG19, creating Gram Matrices from the feature maps obtained after the first convolution in each layer, and then minimizing the difference between these two matrices. The content loss is determined by passing the source image and generated target image through VGG19, then reducing the difference between the feature maps produced after the second convolution. Figure shows our network. The architecture is composed as follows: a generator that creates images, a mapping network that converts latent code z into a style code, a style encoder that converts an image into a style code, discriminator and Pre-trained VGG19.

The training involves several loss functions: an adversarial loss (L_adv) responsible for the training of a generator and discriminator; a style reconstruction loss (L_sty) responsible for the training of a style encoder; a style diversification loss (L_ds) ensuring a style encoder produces various style codes; and a cycle consistency loss (L_cyc) that regenerates an original image from a generated image for comparison. Additionally, there is a NST style loss (L_(NST-S)) comparing the reference image's gram matrix with the generated target image's gram matrix, and a NST content loss (L_(NST-C)) comparing the reference image's feature map with the generated target image's feature map.

Our cross-site brain MRI harmonization method was assessed using quantitative metrics, namely Peak Signal-to-Noise Ratio (PSNR) and Structural Similarity Index Map (SSIM) scores.

In conclusion, our novel approach to cross-site brain MRI harmonization, integrating Neural Style Transfer with Generative Adversarial Networks, demonstrated improvements in two scores. Our methodology, incorporating both first and second-order statistics, offers a comprehensive solution for style transfer and preserving anatomical structure.

Exploratory Modeling and Artifact Removal ¹

Other Methods ²

Data analysis

Data Organization

Machine Learning

Modeling

MRI

Other - synthesis