Ultrasound Volume Reconstruction from Freehand Scans without Tracking

A. 3D Ultrasound Imaging

A 3D ultrasound volume reconstruction system visualizes a region-of-interest (ROI) in 3D by combining a set of 2D ultrasound frames [18]. Based on how the 2D frames are acquired, existing 3D ultrasound reconstruction methods can be divided into three categories [19]: 2D array scanning [20], mechanical scanning [21], and freehand scanning [22]. 2D array scanning systems use 3D ultrasound probes with 2D imaging arrays, which can directly create pyramidal volume scans [21]. However, such systems are more expensive than 1D array transducers and also bulkier in size. The 3D mechanical scanning uses a stepper motor inside a compact casing, which can move the internal imaging array following a tilting, rotating, or linear movement around the ROI. Such a mechanical system acquires regularly spaced 2D ultrasound frames for image volume reconstruction [23]. However, systems falling in the above two categories require specialized US imaging devices to enable the reconstruction, which significantly limit their applications.

Classical freehand US imaging methods use an external tracking device [6], either an optical or electromagnetic tracker, attached to the ultrasound probe attached to record the position and orientation of the US transducer in 3D space. The tracking system increases the complexity of the navigation devices and limits the clinician’s operational space as they need to avoid blocking the tracking signals during the operations.

B. Sensorless Volume Reconstruction

Sensorless freehand scans take a step further by removing such tracking devices. Such methods were supported by the speckle decorrelation algorithms [12], which estimates the elevational distance between neighboring US images based on the US speckle patterns correlation. Using Gaussian model function, Gee et al. [24] proposed a speckle correlation function to approximate the orthogonal distance between two B-model scans and achieved much improved the results. Rivaz et al. [15] designed a speckle detector to classify the irregularly shaped/located regions, and found that the detected fully developed speckles can largely improve the elevational distance measurement. Afsham et al. [25] applied a statistical model based on Rician-Inverse Gaussian stochastic process of the ultrasound speckle formation to estimate the out-of-plane motion. A more recent work by Tetrel et al. [26] proposed to reduce reconstruction error by filtering out unreliable estimations and reported a drift error of 5 mm on sweeps with average length of 35 mm in phantom studies. The above works majorly conducted experiments on exvivo tissues (such as beef, turkey and phantom) with freehand scans and demonstrated the possibility of image-based reconstruction without tracking.

Recent advances in deep learning (DL) methods have shown superior performance in automatic feature extraction. Prevost et al. [13] first proposed to use a convolutional neural network (CNN) to directly estimate the inter-frame motion between two 2D US frames for sensorless volume reconstruction. However, the rich contextual information along the entire ultrasound video was not used. Wein et al. [14] co-registered two DL-reconstructed volumes from transversal and sagittal views, respectively, for a better reconstruction result. Yet, it requires multiple scans in different directions for the same patient, which are typically hard to implement in clinical settings. Our previous work [16] applies 3D CNN on a US video subsequence for better utilizing the temporal context information, which showed promising performance. Recently, Luo et al. [27] proposed a network with a convolutional long short-term memory (LSTM) module and applied a differentiable reconstruction loss to extract the sequential information for US volume reconstruction. This work further demonstrates that using context information from neighboring frames is beneficial in ultrasound volume reconstruction.

C. Contrastive Learning

For supervised learning of deep classification models, commonly used cross-entropy (CE) loss has been reported to have several major shortcomings, including its lack of robustness to noisy labels and the possibility of poor margins [28], leading to reduced generalization performance. To tackle this problem, the contrastive learning strategy was proposed to enhance the discriminative contrast representation across different categories [29]–[31]. The core idea of contrastive learning is to pull positive pairs closer while pushing negative pairs apart in the latent space [32]. The contrastive losses were inspired by noise contrastive estimation [33] or N-pair losses [34], [35]. Intuitively, the contrast loss forces the deep feature generator to extract similar features for images in the same category and distinct feature representations for images from different categories. The losses are often calculated between paired training cases.

A. Problem Definition and Proposed Solution

All the US scanning videos in this work were collected from patients undergoing prostate cancer biopsy using one of the two ultrasound probes, a Philips C9-5 and a Philips mc7-2. During a prostate biopsy procedure, the doctors used an electromagnetic (EM) tracking device to record the spatial location and orientation of the ultrasound probe, which provides the transformation between frames within the scan. This spatial transformation serves as the ground-truth for our network training. Thus, each US frame video has an associated EM-tracking 3D homogeneous transformation matrix $\text{M}=\left[Rt;01\right]$, where M is a 3×3 rotation matrix and t is a 3D translation vector. The EM tracking data is used only as the ground truth for evaluation, but not invovled in the reconstruction process of our algorithm.

The task of sensorless 3D US volume reconstruction is to estimate the relative spatial position between two or more consecutive US frames purely based on the imaging contents: a deep convolutional neural network which takes a US video segment composed of N frames as input and outputs its trajectory estimation in the form of 6 degrees-of-freedom (6DOF) transformation parameters. Without loss of generality, here we use two neighboring frames as an example for illustration. Let I_i and I_i+1 denote two consecutive US frames with corresponding transformation matrices M_i and M_i+1, respectively. The relative transformation matrix $M_i^{\prime }$ can be computed as $M_i^{\prime }=M_{i+1}{M}_i^{-1}$. By decomposing $M_i^{\prime }$ into six transformation parameters ${\theta }_i^{\prime }={\left\{t_x,t_y,t_z,{\alpha }_x,{\alpha }_y,{\alpha }_z\right\}}_i$, which contains the translations in millimeters and rotations in degrees. During training, a video segment of N consecutive frames with height H and width W are passed to the network as input, either separately or stacked together. Let $\left\{{\theta }_i^{\prime }|i=1,\dots ,N-1\right\}$ denote the relative transform parameters between the neighboring frames within this video segment. Instead of directly using all these 6 × (N − 1) transformation parameters as ground-truth labels, we compute the mean parameters $\theta$ as the corresponding label of a video segment:

There are two reasons for using the mean vector as the training label: (1) Since the magnitude of motion between two frames is small, using the mean can effectively smooth the noise in probe motion; (2) Using a labeling representation with a fixed length (6 transformation parameters) does not require us to modify the output layer every time when we change the number of input frames N. Since we compute the motion vector relative to the previous neighboring frame, such interframe motion has small magnitudes for all the six degrees-of-freedom. The direction of each motion is signed, making it feasible to average the motion for label computation. We compose the network predictions from all the video segments into a full volume reconstruction for one entire video during the inference stage.

B. Deep Contextual Learning

This subsection introduces the Deep contextual-contrastive Network (DC²-Net) structure proposed for the 3D ultrasound volume reconstruction task. The structure of DC²-Net, shown in Fig. 2, is designed to make full use of the context information within a US video segment. The ultrasound video sequence contains rich temporal information, which can provide more reference for the robust US probe trajectory estimation. Thus, we propose three designs to fully utilize the US video’s context information: video segment input, self-attention mechanism, and a case-wise correlation loss.

C. Margin Ranking Loss

In our work, estimating the spatial location and orientation of each US frame for volume reconstruction is essentially a regression task. While contrastive learning is a well-defined strategy in image classification tasks, it is much more complex to adapt the idea into regression problems. The label values in classification problems are discreet but become continuous in regression-related tasks. We here enhance the trajectory-estimation feature learning by adopting a novel margin ranking loss [42], which correlates two random samples’ feature vectors distance with their transformation parameters discrepancy. For a given triplet, we regard one sample as the anchor. Among the other two samples, the one with a 6-DOF label closer to the anchor is considered as a near sample, while the other one as a far sample. Intuitively, the feature vectors of the anchor should be closer to a near sample than a far sample in the latent space by a margin M. The margin ranking loss is defined as:

where f_a, f_n, and f_f indicate the feature vectors of the anchor, near and far samples, respectively. The margin M defines the minimum tolerance that separates the anchor-far pair from the anchor-near pair. The network continues to optimize as long as the paired difference is below the margin. To enforce that anchor-far pairs have larger distances than the anchor-near pairs, we compute the adaptive coefficient α using sample labels:

where ${\mathit{\theta }}_a,{\mathit{\theta }}_n,{\mathit{\theta }}_f$ represent the groundtruth 6DOF transformation parameters of the anchor, near and far samples, respectively. The function sign() returns the positive/negative sign of the input value. Of note, during the training, we compute the margin ranking loss within each batch by ranking every pair of samples through matrix manipulation. In summary, the full contextual-contrastive loss is formulated as:

where λ₁, λ₂, and λ₃ are the positive weighting parameters. Empirically, we set λ₁, λ₂, and λ₃ to 5.0, 1.0, and 1.0, respectively, by considering their magnitudes.

A. Materials

Our study was conducted retrospectively using two independent sets of human subject data acquired through an IRB-approved clinical study, following the ethical standards of the institutional and national research committee.

Dataset A contains 618 transrectal US video sequences, with each frame corresponding to a positioning matrix captured by an EM-tracking device. An end-firing Philips C9-5 transrectal US transducer captures axial images by steadily sweeping through the prostate from base to apex. We split dataset A into 488, 66, and 64 cases for training, validation, and testing. Every video in this dataset is from a different patient. There is no overlapping patient between the training, validation and test sets. In dataset A, we have prostate segmentation annotations for each video frame, making it possible to reconstruct 3D prostate segmentation volumes for anatomical evaluation.

Dataset B contains 100 transabdominal/transperineal US video sequences acquired by a Philips mC7-2 US probe. Due to the relatively small size of dataset B, we split it into 80, 10, and 10 for training, validation, and testing in a rotating fashion to conduct 10-fold cross-validation. The US video sequences described above were captured from different subjects at varying lengths and resolutions. Some patients may have multiple US scans. During the cross validation, we split the training, validation and test set according to the patients ID, such that all the videos from the same patient will stay in the same set.

A 6D EM tracking sensor is attached to the US probe using a probe specific fitting to ensure that the spatial relationship, i.e. calibration, between the sensor and the ultrasound imaging plane is fixed. The manufacturer of the tracking sensor is Philips Healthcare, which also provides the calibration between the sensor and the ultrasound probe. The positioning information given by this EM-tracking device serves as the ground truth label in our training phase. Essentially, an EM-tracker may have several millimeter errors for the entire workspace. However, the motion range of the EM tracking sensor for 2D US sweeps is only a small part of the full workspace. As long as the linearity of EM tracking for that small range is good, the EM-based volume reconstruction performs well. All our reported errors are relative to the EM-tracking data and the reconstruction based on that.

B. Evaluation Metrics and Implementation Details

In this work, we adopted the following six evaluation metrics for measuring the volume reconstruction qualities: distance error, frame error, final drift, drift rate, and prostate Dice:

• Distance Error is the average distance between all the corresponding corner-points of the input patches (shown by the white bounding box in Fig.1) throughout a video scan, which reveals the overall difference in speed and orientation across the entire video.

• Frame Error computes the difference between the ground-truth and a predicted frame location. This metric helps evaluate the relative error between a pair of neighboring frames regardless of the cumulative error along the sequence.

• Final Drift [13] denotes the Euclidean distance between the US video’s last frame, posed by the ground-truth transformation and the estimated transformation.

• Drift Rate [27] measures the ratio between the final drift and the ground truth sequence length.

• Prostate Dice computes the Dice coefficient between the prostate’s ground-truth segmentation V_seg and the predicted segmentation ${\widehat{V}}_{seg}$. Given the 2D prostate segmentation for each frame in dataset A, V_seg can be reconstructed using the ground-truth positioning information and ${\widehat{V}}_{seg}$ from the network’s predicted trajectory.

• Prostate Error measures the absolute volume difference between the ground-truth segmentation V_seg and the predicted segmentation ${\widehat{V}}_{seg}$ in cubic centimeters (cc).

We use a paired t-test with a significance level α = 0.05 to compare different methods for the statistical tests carried out in this section.

Our network is trained for 100 epochs with batch size of 48 using Adam optimizer [44] with an initial learning rate of 1×10⁻⁴, which decays by 0.9 after five epochs. We iterate through all the possible samples in our dataset in each training epoch. Since the prostate US image only takes a relatively small part of each frame, each frame is cropped without exceeding the imaging field and then resized to 224 × 224 to fit the design of ResNeXts [36]. The cropping is automatically defined to find the maximum ROI square within the fan-shaped receptive field. During training, we mixed the ultrasound videos with varying depths and applied image intensity normalization, aiming to make the network invariant to the different imaging settings. We implemented the DC²-Net using the publicly available PyTorch library [45]. The entire training phase of the DC²-Net takes about twenty hours, taking five frames as input. It takes about 2.58s to produce all the transformation matrix of a US video with 100 frames during testing.

C. Results Comparison

Tables I and II summarize the overall comparison of the proposed DC²-Net against other existing methods on datasets A and B, respectively. We include the prostate Dice only in Table I because the 2D prostate segmentation is only available in dataset A. The “Linear Motion” assumes that a clinician moves the ultrasound probe at a constant speed along certain direction. It thus applies a constant velocity to reconstruct the ultrasound volume [13]. Specifically, we compute a mean motion vector (6D) in the training set and then directly apply this fixed motion vector to the test set for volume reconstruction [13]. The approach of “Decorrelation” is based on the speckle decorrelation algorithm presented in [43]. For a network structure comparison, we used five baseline designs as shown in Fig. 3: “2D CNN” refers to the method presented by Prevost et al. [13]. “Shared Siamese” refers to the Siamese network with shared parameters as in Fig.3(b). “Multi-branch” is a 2D network where the two branches share the same structure but with different gradient back-propagation, as in Fig.3 (c). “3D CNN (N = 2)” is the vanilla ResNeXt [36] architecture taking only two frames as input. “ConvLSTM [27] is a recent work that combines the convolutional layers with the long-short-term-memory module for US volume reconstruction. We also include our previous work in DCL-Net [16], which did not apply the margin ranking loss for contrastive learning.

Through paired t-test, our proposed DC²-Net is found to have significant improvements over all the baseline methods (p-value<0.05). The performance of the 2D-CNN reproduced in our experiments has consistent performance compared to the accuracy reported in the paper [13]. The Shared Siamese and Multi-branch structures have comparable performance on both datasets. However, due to the nature of 2D convolutions, they both have limited capacity to learn the inter-frame motion, especially the out-of-plane movement. The ConvLSTM uses a recurrent strategy and consistently achieves better performance than the previously described methods.

The statistical results of DCL-Net reported in this paper have a considerable performance leap compared to our previous work [16] because of the updated training strategy. By applying the newly added margin ranking loss and using seven frames as an input video segment, our DC²-Net achieved a significant performance boost on all six evaluation metrics. It is worth noting that the prostate Dice of 0.89 achieved by DC²-Net indicates a good reconstruction performance, especially for the shape of the prostate. With an average prostate volume size of 26.7 cc in our test set, the DC²-Net achieved an average prostate reconstruction error of 3.21 cc. During the prostate biopsy, the prostate volume is a clinical parameter that doctors use as a reference to make diagnosis. It does not have to be very precise for interventional guidance, as long as it provides useful 3D imaging information, which is otherwise unavailable using 2D ultrasound probes. Although the average final drift of 10.20 mm achieved by DC²-Net is not a negligible error, this is the best performance on real clinical data instead of phantom studies. It is challenging to reconstruct 3D US volume using these freehand US scans, and we have been making significant progress in this important area.

A. Model Analysis and Hyper-parameter Sensitivity

In this section, we systematically perform the ablation study to evaluate the effectiveness of each component (attention module, correlation loss, and margin ranking loss) in the DC²-Net. We conduct the ablation study on dataset A, which has more training cases to guarantee a solid comparison in Table III. The first row refers to the vanilla ResNeXt-50 structure, which takes five consecutive frames as input. With both attention module and correlation loss implemented (third row, equivalent to DCL-Net [16]), a significant performance is achieved, demonstrating these two components’ effectiveness. With the additional margin ranking loss, the complete DC²-Net beats the baselines with a considerable margin of improvement.

We perform the hyper-parameter tuning on the contrastive margin M in Fig. 4a, which determines the minimum distance to separate an anchor-near pair and an anchor-far pair. With N = 5 and N = 7 (which we found to be practical settings based on experience), the drift rate exhibits a downward then upward trend when we increase the margin M. By incorporating the margin ranking loss, DC²-Net can perform better than the DCL-Net (black horizontal line) over a wide range of margin choices. Additionally, using seven frames as input achieves almost consistent lower drift rates than using only five frames when exploring the margins. When we search for an optimal choice of frame numbers in Fig. 4b (with M = 0.25), we also observe a downward trend followed by an upward trend, which sufficiently matches the intuition: With relatively fewer frames in a video segment, the input sample provides limited contextual information which hinders the network from extracting sufficient features. On the other hand, since we use an average motion vector over a video segment as the training signal (Sec.III-A), sudden changes in the probe’s speed and orientation will be eliminated by simply averaging the gaps, which makes it difficult for the networks to capture the trajectory variances.

B. Motion Estimation Analysis

Given that the deep learning-based US volume reconstruction is a challenging task, one may question whether the network is simply memorizing the scanning protocols and fitting to a global average. To show the learning capacity of the proposed DC2-Net, Fig.5 visualizes the correlation between the ground-truth and network’s prediction on each of the six degrees-of-freedom. For every test video in Dataset A, we computed the mean motion vector for the ground-truth and network’s prediction. We plot the mean motion of each video as a single dot with the ground-truth magnitude as x-axis coordinate, and prediction as y-axis coordinate. More points distributed along the diagonal line would suggest a more accurate prediction. Fig. 5 shows that the DC²-Net’s prediction (green dots) on the test set exhibits a stronger correlation with the ground-truth label, as more dots distribute along the diagonal line than those of the DCL-Net. That indicates superior performance of the proposed DC²-Net against the baseline method.

In Fig. 6, we draw motion changes along six degrees-of-freedom from the first frame to the last frame in the entire video of a case. By fixing the plotting scales for translations and rotations respectively, we can see that motions along tY, αX and αZ are more dominant than the others, because of the constraint by the rectum during ultrasound sweep. Our DC2-Net’s prediction not only has a strong correlation with the actual ground-truth motion, but also matches the magnitudes of each degrees-of-freedom. The prediction result of DC²-Net has several minor mismatches with the extreme highs and lows in the ground-truth. The extreme values could be the outcome of unsteady scans, which may not be predicted by a well-trained network. On the other hand, since we used the mean motion vector of a sub-sequence (varying from 2 to 10 frames) as the ground-truth for a training sample, the trained network might have learned such a smoothing effect. This could have contributed to some mismatches with the extreme values. However, since these extreme values are relatively sparse in the datasets, they have only minor impact on the overall performance.

Table IV shows the results on datasets A and B, where our DC²-Net achieved strong correlation of 0.62 and 0.46, respectively. We observed an interesting pattern by looking at the correlation of each degrees-of-freedom. The prediction of translations (tX, tY and tZ) has a higher correlation than the rotations (αX, αY and αZ). This is because the rotations are typically “out-of-plane” motion, which are more challenging to regress from the image contents than the in-plane translations. In summary, the matching pattern in Fig. 6 and the high Pearson coefficients in Table IV both indicate that the DC²-Net is capable of capturing the sudden changes in the US scans.

C. Quality of Volume Reconstruction

In addition to the quantitative evaluation metrics such as final drift and distance error, we also visualize the reconstructed volumes for qualitative evaluation as shown in Figs. 7 and 8. For the C9-5 cases in dataset A, since the prostate segmentation is available, we superimpose the reconstructed 3D prostate segmentation to the US volume for anatomical shape evaluation. The 3D prostate segmentation is reconstructed by posing 2D prostate segmentation in each frame to the 3D space, followed by spatial interpolation and filling holes. The DC²-Net’s reconstructed volume all looks similar to the ground-truth reconstruction, and the prostate segmentation largely preserves its shape and size as shown in Table I and Fig. 7. In contrast, the previous DCL-Net produces overly smoothed trajectory estimation (second rows in Figs. 7 and 8). By incorporating the margin ranking loss and adjusting the input frame numbers N properly, the predicted motion trajectories from DC²-Net have a much higher correlation with the ground-truth positions. That is to say, the predicted motion is more sensitive to the variations in speed and orientations introduced during the scanning. For example, there exist some sudden orientation changes in C95 cases 4 and 5 during the scan, which can be observed in the volumes’ rugged bottom lines. While the DCL-Net’s reconstruction is over smooth and eliminates the details, our DC²-Net captured those sudden changes and produce matching positioning information.

In our work, we measure the final drift for the full ultrasound volume, using the corner points of the input patches to compute the errors. The entire ultrasound volume is much larger than the prostate gland, leading to the seemingly large accumulated errors reported in the paper. In other words, these drift errors do not explicitly correspond to the prostate tracking errors. To compensate the effect, we also used the anatomical evaluation metrics, such as the Dice coefficient of prostate segmentation, to measure the quality of volume reconstruction. While it is difficult to certify that the current performance is sufficient for clinical application, the presented work can provide useful 3D imaging information of the prostate regardless of the drift. More importantly, the DC2-Net allows a regular 2D ultrasound probe to acquire a 3D scan without any hardware change.