Nothing Special   »   [go: up one dir, main page]
Skip to main content
Springer Nature Link
Log in

Improving realism in patient-specific abdominal ultrasound simulation using CycleGANs

  • Original Article
  • Published:
International Journal of Computer Assisted Radiology and Surgery Aims and scope Submit manuscript

Abstract

Purpose

In this paper, we propose to apply generative adversarial neural networks trained with a cycle consistency loss, or CycleGANs, to improve realism in ultrasound (US) simulation from computed tomography (CT) scans.

Methods

A ray-casting US simulation approach is used to generate intermediate synthetic images from abdominal CT scans. Then, an unpaired set of these synthetic and real US images is used to train CycleGANs with two alternative architectures for the generator, a U-Net and a ResNet. These networks are finally used to translate ray-casting based simulations into more realistic synthetic US images.

Results

Our approach was evaluated both qualitatively and quantitatively. A user study performed by 21 experts in US imaging shows that both networks significantly improve realism with respect to the original ray-casting algorithm (\(p \ll 0.0001\)), with the ResNet model performing better than the U-Net (\(p \ll 0.0001\)).

Conclusion

Applying CycleGANs allows to obtain better synthetic US images of the abdomen. These results can contribute to reduce the gap between artificially generated and real US scans, which might positively impact in applications such as semi-supervised training of machine learning algorithms and low-cost training of medical doctors and radiologists in US image interpretation.

This is a preview of subscription content, log in via an institution to check access.

Access this article

Log in via an institution

Subscribe and save

Springer+ Basic
$34.99 /Month
  • Get 10 units per month
  • Download Article/Chapter or eBook
  • 1 Unit = 1 Article or 1 Chapter
  • Cancel anytime
Subscribe now

Buy Now

Price excludes VAT (USA)
Tax calculation will be finalised during checkout.

Instant access to the full article PDF.

Institutional subscriptions

Fig. 1
Fig. 2
Fig. 3
Fig. 4
Fig. 5
Fig. 6
Fig. 7
Fig. 8

Similar content being viewed by others

Explore related subjects

Discover the latest articles, news and stories from top researchers in related subjects.

Notes

  1. http://www.vision.ee.ethz.ch/datasets_extra/usliverseq.zip.

  2. http://deepultrasound.ai/.

  3. https://github.com/junyanz/pytorch-CycleGAN-and-pix2pix.

References

  1. American College of Emergency Physicians (2001) Use of ultrasound imaging by emergency physicians. Ann Emerg Med 38(4):469

  2. Behboodi B, Rivaz H (2019) Ultrasound segmentation using u-net: learning from simulated data and testing on real data. arXiv:1904.11031

  3. Cohen JP, Luck M, Honari S (2018) Distribution matching losses can hallucinate features in medical image translation. arXiv:1805.08841

  4. D’Amato JP, Lo Vercio L, Rubí P, Fernández Vera E, Barbuzza R, del Fresno M, Larrabide I (2015) Efficient scatter model for simulation of ultrasound images from computed tomography data. In: 11th International symposium on medical information processing and analysis, vol 9681. International Society for Optics and Photonics, p 968105

  5. De Leeuw JR (2015) jspsych: a JavaScript library for creating behavioral experiments in a web browser. Behav Res Methods 47(1):1–12

    Article  Google Scholar 

  6. De Luca V, Tschannen M, Székely G, Tanner C (2013) A learning-based approach for fast and robust vessel tracking in long ultrasound sequences. In: International conference on medical image computing and computer-assisted intervention. Springer, pp 518–525

  7. Dinh VA, Fu JY, Lu S, Chiem A, Fox JC, Blaivas M (2016) Integration of ultrasound in medical education at United States medical schools: a national survey of directors’ experiences. J Ultrasound Med 35(2):413–419

    Article  Google Scholar 

  8. Engelhardt S, De Simone R, Full PM, Karck M, Wolf I (2018) Improving surgical training phantoms by hyperrealism: Deep unpaired image-to-image translation from real surgeries. arXiv:1806.03627

  9. Goodfellow I, Pouget-Abadie J, Mirza M, Xu B, Warde-Farley D, Ozair S, Courville A, Bengio Y (2014) Generative adversarial nets. In: Ghahramani Z, Welling M, Cortes C, Lawrence ND, Weinberger KQ (eds) Proceedings of the 27th international conference on neural information processing systems, vol 2. MIT Press, Cambridge, pp 2672–2680

  10. He K, Zhang X, Ren S, Sun J (2016) Deep residual learning for image recognition. In: Proceedings of the IEEE conference on computer vision and pattern recognition, pp 770–778

  11. Heer I, Middendorf K, Müller-Egloff S, Dugas M, Strauss A (2004) Ultrasound training: the virtual patient. Ultrasound Obstet Gynecol 24(4):440–444

    Article  CAS  Google Scholar 

  12. Hiasa Y, Otake Y, Takao M, Matsuoka T, Takashima K, Carass A, Prince JL, Sugano N, Sato Y (2018) Cross-modality image synthesis from unpaired data using CycleGAN. In: International workshop on simulation and synthesis in medical imaging. Springer, pp 31–41

  13. Ircad data set. https://www.ircad.fr/research/3dircadb/. Accessed 2018-12-26

  14. Isola P, Zhu JY, Zhou T, Efros AA (2017) Image-to-image translation with conditional adversarial networks. In: CVPR

  15. Kutter O, Shams R, Navab N (2009) Visualization and GPU-accelerated simulation of medical ultrasound from CT images. Comput Methods Progr Biomed 94(3):250–266

    Article  Google Scholar 

  16. Litjens G, Kooi T, Bejnordi BE, Setio AAA, Ciompi F, Ghafoorian M, Van Der Laak JA, Van Ginneken B, Sánchez CI (2017) A survey on deep learning in medical image analysis. Med Image Anal 42:60–88

    Article  Google Scholar 

  17. Noble JA (2016) Reflections on ultrasound image analysis. Med Image Anal 33:33–37. https://doi.org/10.1016/j.media.2016.06.015

    Article  Google Scholar 

  18. Odena A, Dumoulin V, Olah C (2016) Deconvolution and checkerboard artifacts. Distill 1(10):e3

    Article  Google Scholar 

  19. Østergaard ML, Ewertsen C, Konge L, Albrecht-Beste E, Nielsen MB (2016) Simulation-based abdominal ultrasound training—a systematic review. Ultraschall in der Medizin Eur J Ultrasound 37(03):253–261

    Article  Google Scholar 

  20. Petrusca L, Cattin P, De Luca V, Preiswerk F, Celicanin Z, Auboiroux V, Viallon M, Arnold P, Santini F, Terraz S, Scheffler K, Becker CD, Salomir R (2013) Hybrid ultrasound/magnetic resonance simultaneous acquisition and image fusion for motion monitoring in the upper abdomen. Investig Radiol 48(5):333–340

    Article  Google Scholar 

  21. Pham AH, Stage B, Hemmsen MC, Lundgren B, Pedersen MM, Jensen JA (2011) Simulation of shadowing effects in ultrasound imaging from computed tomography images. In: 2011 IEEE international ultrasonics symposium, pp 1411–1414. https://doi.org/10.1109/ULTSYM.2011.0349

  22. Ronneberger O, Fischer P, Brox T (2015) U-net: convolutional networks for biomedical image segmentation. In: International conference on medical image computing and computer-assisted intervention. Springer, pp 234–241

  23. Rubi P, Vera EF, Larrabide I, Calvo M, D’Amato J, Larrabide I (2017) Comparison of real-time ultrasound simulation models using abdominal CT images. In: 12th international symposium on medical information processing and analysis, vol 10160. International Society for Optics and Photonics, p 1016009

  24. Shams R, Hartley R, Navab N (2008) Real-time simulation of medical ultrasound from ct images. In: International conference on medical image computing and computer-assisted intervention. Springer, pp 734–741

  25. Taigman Y, Polyak A, Wolf L (2017) Unsupervised cross-domain image generation. In: International conference on learning representations, ICLR 2017

  26. Terkamp C, Kirchner G, Wedemeyer J, Dettmer A, Kielstein J, Reindell H, Bleck J, Manns M, Gebel M (2003) Simulation of abdomen sonography. evaluation of a new ultrasound simulator. Ultraschall in der Medizin 24(04):239–244

    Article  CAS  Google Scholar 

  27. Walcher F, Weinlich M, Conrad G, Schweigkofler U, Breitkreutz R, Kirschning T, Marzi I (2006) Prehospital ultrasound imaging improves management of abdominal trauma. Br J Surg Inc Eur J Surg Swiss Surg 93(2):238–242

    CAS  Google Scholar 

  28. Wang C, Macnaught G, Papanastasiou G, MacGillivray T, Newby D (2018) Unsupervised learning for cross-domain medical image synthesis using deformation invariant cycle consistency networks. In: International workshop on simulation and synthesis in medical imaging. Springer, pp 52–60

  29. Wein W, Kamen A, Clevert DA, Kutter O, Navab N (2007) Simulation and fully automatic multimodal registration of medical ultrasound. pp 136–43. https://doi.org/10.1007/978-3-540-75757-3_17

    Chapter  Google Scholar 

  30. Zhu JY, Park T, Isola P, Efros AA (2017) Unpaired image-to-image translation using cycle-consistent adversarial networks. In: ICCV

Download references

Acknowledgements

This work was funded by ANPCyT PICT 2016-0116 and PID-UTN SIUTNBA0005139. A NVIDIA GPU hardware grant supported this research with the donation of a Quadro P6000 graphic card. JIO is now a Postdoctoral Fellow at MedUniWien, funded by WWTF AugUniWien/FA7464A0249 (Medical University of Vienna); VRG12-009 (University of Vienna). We thank Lucia Llan de Rosos, Constantine Butakoff, Lidia Quinteros, Sergio Sánchez Martínez, Diego Pegoraro, Debbie Zhao, Matthieu De Craene, Gaurav Phadke and all the anonymous volunteers who participated in the user study. We also thank Claudia Marinelli and Rosana Cepeda from Instituto Multidisciplinario sobre Ecosistemas y Desarrollo Sustentable (UNICEN, Tandil, Argentina) for their assistance with the statistical analysis.

Author information

Authors and Affiliations

  1. Pladema, UNICEN, Tandil, Argentina

    Santiago Vitale & Ignacio Larrabide

  2. OPTIMA, Department of Ophthalmology, Medical University of Vienna, Vienna, Austria

    José Ignacio Orlando

  3. UTN-FRBA, Buenos Aires, Argentina

    Emmanuel Iarussi

  4. CONICET, Buenos Aires, Argentina

    Santiago Vitale, Emmanuel Iarussi & Ignacio Larrabide

Authors
  1. Santiago Vitale
    View author publications

    You can also search for this author in PubMed Google Scholar

  2. José Ignacio Orlando
    View author publications

    You can also search for this author in PubMed Google Scholar

  3. Emmanuel Iarussi
    View author publications

    You can also search for this author in PubMed Google Scholar

  4. Ignacio Larrabide
    View author publications

    You can also search for this author in PubMed Google Scholar

Corresponding author

Correspondence to Santiago Vitale.

Ethics declarations

Human and animals participants

This article does not contain any studies with human participants or animals performed by any of the authors.

Conflict of interest

The authors declare that they have no conflict of interest.

Additional information

Publisher's Note

Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.

Rights and permissions

Reprints and permissions

About this article

Check for updates. Verify currency and authenticity via CrossMark

Cite this article

Vitale, S., Orlando, J.I., Iarussi, E. et al. Improving realism in patient-specific abdominal ultrasound simulation using CycleGANs. Int J CARS 15, 183–192 (2020). https://doi.org/10.1007/s11548-019-02046-5

Download citation

  • Received:

  • Accepted:

  • Published:

  • Issue Date:

  • DOI: https://doi.org/10.1007/s11548-019-02046-5

Keywords

Search

Navigation