Volumetric Fetal Brain Reconstruction from Sensor-free Freehand Ultrasound Videos
Poster No:
1850
Submission Type:
Abstract Submission
Authors:
Pak Hei Yeung1, Linde Hesse1, Moska Aliasi2, Monique Haak2, INTERGROWTH-21st Consortium3, Weidi Xie1,4, Ana Namburete5
Institutions:
1University of Oxford, OXFORD, Oxfordshire, 2Leiden University Medical Center, Leiden, Leiden, 3Oxford Maternal & Perinatal Health Institute (OMPHI), Green Templeton College, University of Oxford, Oxford, United Kingdom, 4Shanghai Jiao Tong University, Shanghai, China, 5University of Oxford, Oxford, Oxfordshire
First Author:
Co-Author(s):
INTERGROWTH-21st Consortium
Oxford Maternal & Perinatal Health Institute (OMPHI), Green Templeton College, University of Oxford
Oxford, United Kingdom
Oxford Maternal & Perinatal Health Institute (OMPHI), Green Templeton College, University of Oxford
Oxford, United Kingdom
Introduction:
Three-dimensional (3D) ultrasound (US) imaging has contributed to our understanding of fetal neurodevelopmental processes by providing rich 3D anatomical and contextual information [3]. However, its use is restricted in clinical settings, due to the high purchasing costs and limited clinical practicality. Freehand 2D US imaging, in contrast, is routinely used in standard obstetric exams, but inherently lacks a 3D representation of the anatomies, which limits its potential for more advanced assessment. 3D representations are challenging to recover from 2D scans even with external tracking devices due to internal fetal movement. Capitalizing on the flexibility offered by freehand 2D US acquisition, we propose ImplicitVol to reconstruct 3D volumes from non-sensor-tracked 2D US sweeps.
Methods:
Consider a set of 2D US images capturing different cross-sectional views of a fetal brain. The goal of ImplicitVol (Fig. 1a) is to reconstruct the corresponding 3D brain, such that any 2D cross-sectional view can be generated by inputting the corresponding 3D coordinates of the plane.
Plane Localization: We use PlaneInVol [6] to estimate the 3D locations of each of the 2D US images. Not requiring any external tracking, it is trained with 2D slices, sampled from a set of affinely co-registered 3D brain volumes, and their locations in the 3D alignment space [3].
Training: Intuitively, the 3D volume is stored in a deep neural network (DNN). We first derive the 3D coordinates for each pixel of the 2D US images from the predicted 3D locations. By inputting them to the DNN, the corresponding intensity values are predicted. The weights of the DNN can be learned through backpropagation, where the loss is computed by the structural similarity (SSIM) [4] between the predicted intensities and the 2D US images. Since the 3D plane locations predicted by PlaneInVol are imperfect, we further update the 3D locations simultaneously through joint optimization [5]. The DNN weights are also iteratively reinitialized during training to avoid overfitting to a set of sub-optimal 3D locations.
Inference: The trained DNN represents a continuous 3D fetal brain captured by the set of 2D images. The 3D volume or any 2D cross-sectional oblique view can be obtained as the output, by feeding the corresponding grid coordinates for the desired slice to the trained DNN.
Plane Localization: We use PlaneInVol [6] to estimate the 3D locations of each of the 2D US images. Not requiring any external tracking, it is trained with 2D slices, sampled from a set of affinely co-registered 3D brain volumes, and their locations in the 3D alignment space [3].
Training: Intuitively, the 3D volume is stored in a deep neural network (DNN). We first derive the 3D coordinates for each pixel of the 2D US images from the predicted 3D locations. By inputting them to the DNN, the corresponding intensity values are predicted. The weights of the DNN can be learned through backpropagation, where the loss is computed by the structural similarity (SSIM) [4] between the predicted intensities and the 2D US images. Since the 3D plane locations predicted by PlaneInVol are imperfect, we further update the 3D locations simultaneously through joint optimization [5]. The DNN weights are also iteratively reinitialized during training to avoid overfitting to a set of sub-optimal 3D locations.
Inference: The trained DNN represents a continuous 3D fetal brain captured by the set of 2D images. The 3D volume or any 2D cross-sectional oblique view can be obtained as the output, by feeding the corresponding grid coordinates for the desired slice to the trained DNN.
Results:
ImplicitVol, with ablation studies, was compared against an interpolation-based approach (Baseline 1) and Baseline1+SVRTK [2] (Baseline 2). Both 3D volumes acquired from different sites (Dataset A and B) and native 2D freehand video sequences were tested.
Reconstruction from Volume-Sampled Images: 2D cross-sectional images were sampled from native 3D volumes. Figure 2a-d demonstrated, with two different evaluation metrics, that the 3D volumes reconstructed from ImplicitVol showed a closer match with the corresponding ground-truth by over 50% (SSIM) and 40% (VIF).
Segmentation on Reconstructed Volumes: Using a 3D segmentation DNN [1] to segment four subcortical structures from the brain volumes, ImplicitVol shows a better segmentation accuracy qualitatively (Fig. 1c) and quantitatively by all three evaluation metrics (Fig. 2e-j), suggesting its better performance on semantic level.
Reconstruction from Native 2D US Images: Qualitatively, brain volumes reconstructed by ImplicitVol showed better visual quality in motion-corrupted regions and under-sampled regions when compared to Baseline 1 and 2 (Fig. 1b).
Reconstruction from Volume-Sampled Images: 2D cross-sectional images were sampled from native 3D volumes. Figure 2a-d demonstrated, with two different evaluation metrics, that the 3D volumes reconstructed from ImplicitVol showed a closer match with the corresponding ground-truth by over 50% (SSIM) and 40% (VIF).
Segmentation on Reconstructed Volumes: Using a 3D segmentation DNN [1] to segment four subcortical structures from the brain volumes, ImplicitVol shows a better segmentation accuracy qualitatively (Fig. 1c) and quantitatively by all three evaluation metrics (Fig. 2e-j), suggesting its better performance on semantic level.
Reconstruction from Native 2D US Images: Qualitatively, brain volumes reconstructed by ImplicitVol showed better visual quality in motion-corrupted regions and under-sampled regions when compared to Baseline 1 and 2 (Fig. 1b).
Conclusions:
Without requiring extra equipment or substantial changes to the routine scanning procedures, Implicitvol can enhance the diagnostic power of 2D obstetric ultrasound, by extracting volumetric information from standard 2D scanning sequences. This may facilitate the transformation of US from a screening to a powerful diagnostic tool, ultimately offering personalised and advanced monitoring to the most vulnerable members of society, while capitalizing on the affordability and ubiquity of 2D US imaging at the bedside.
Lifespan Development:
Early life, Adolescence, Aging 2
Modeling and Analysis Methods:
Image Registration and Computational Anatomy
Methods Development 1
Novel Imaging Acquisition Methods:
Imaging Methods Other
Keywords:
Computing
Machine Learning
ULTRASOUND
1|2Indicates the priority used for review
Provide references using author date format
2. Kuklisova-Murgasova, M. (2012). “Reconstruction of fetal brain MRI with intensity matching and complete outlier removal.” Medical image analysis, 16(8), 1550-1564.
3. Namburete, A. IL. (2023), "Normative spatiotemporal fetal brain maturation with satisfactory development at 2 years." Nature, 1-9.
4. Wang, Z. (2004). “Image quality assessment: from error visibility to structural similarity.” IEEE transactions on image processing, 13(4), 600-612.
5. Wang, Z. (2021). “NeRF--: Neural radiance fields without known camera parameters.” arXiv preprint arXiv:2102.07064.
6. Yeung, PH. (2021), "Learning to map 2D ultrasound images into 3D space with minimal human annotation." Medical Image Analysis, 70, 101998.