Nothing Special   »   [go: up one dir, main page]
Skip to main content
Springer Nature Link
Log in

HEARTBEAT4D: An Open-source Toolbox for Turning 4D Cardiac CT into VR/AR

  • Published:
Journal of Digital Imaging Aims and scope Submit manuscript

Abstract

Four-dimensional data sets are increasingly common in MRI and CT. While clinical visualization often focuses on individual temporal phases capturing the tissue(s) of interest, it may be possible to gain additional insight through exploring animated 3D reconstructions of physiological motion made possible by augmented or virtual reality representations of 4D patient imaging. Cardiac CT acquisitions can provide sufficient spatial resolution and temporal data to support advanced visualization, however, there are no open-source tools readily available to facilitate the transformation from raw medical images to dynamic and interactive augmented or virtual reality representations. To address this gap, we developed a workflow using free and open-source tools to process 4D cardiac CT imaging starting from raw DICOM data and ending with dynamic AR representations viewable on a phone, tablet, or computer. In addition to assembling the workflow using existing platforms (3D Slicer and Unity), we also contribute two new features: 1. custom software which can propagate a segmentation created for one cardiac phase to all others and export to surface files in a fully automated fashion, and 2. a user interface and linked code for the animation and interactive review of the surfaces in augmented reality. Validation of the surface-based areas demonstrated excellent correlation with radiologists’ image-based areas (R > 0.99). While our tools were developed specifically for 4D cardiac CT, the open framework will allow it to serve as a blueprint for similar applications applied to 4D imaging of other tissues and using other modalities. We anticipate this and related workflows will be useful both clinically and for educational purposes.

This is a preview of subscription content, log in via an institution to check access.

Access this article

Log in via an institution

Subscribe and save

Springer+ Basic
$34.99 /Month
  • Get 10 units per month
  • Download Article/Chapter or eBook
  • 1 Unit = 1 Article or 1 Chapter
  • Cancel anytime
Subscribe now

Buy Now

Price excludes VAT (USA)
Tax calculation will be finalised during checkout.

Instant access to the full article PDF.

Institutional subscriptions

Fig. 1
Fig. 2
Fig. 3
Fig. 4
Fig. 5

Similar content being viewed by others

Availability of Data and Material

As the focus of this manuscript was on the software methods, the anonymized imaging source data used in creating these virtual models is available only upon request.

Code Availability

3D Slicer is a free, open-source image computing platform available at https://www.slicer.org, The Unity game development platform is available at http://unity.com, Information and software relating to the Merge Cube is available at https://mergeedu.com/cube, Custom Software resources, The Python code for the custom Slicer module “PropagateSegToOtherPhases” as well as the custom C# scripts used in the Unity project are publicly available in the GitHub repository link at https://github.com/mikebind/Heartbeat4D, Cardiac4DTemplateProject.zip: A template Unity Project into which users may place their own processed data to create their own augmented reality app.

References

  1. Townsend K, Pietila T. 3D printing and modeling of congenital heart defects: A technical review. Birth Defects Res. 2018;110(13):1091-1097. https://doi.org/10.1002/bdr2.1342

    Article  CAS  PubMed  Google Scholar 

  2. Qian Z, Wang K, Liu S, et al. Quantitative Prediction of Paravalvular Leak in Transcatheter Aortic Valve Replacement Based on Tissue-Mimicking 3D Printing. JACC Cardiovasc Imaging. 2017;10(7):719-731. https://doi.org/10.1016/j.jcmg.2017.04.005

    Article  PubMed  Google Scholar 

  3. Sakly H, Said M, Radhouane S, Tagina M. Medical decision making for 5D cardiac model: Template matching technique and simulation of the fifth dimension. Comput Methods Programs Biomed. 2020;191:105382. https://doi.org/10.1016/j.cmpb.2020.105382

    Article  PubMed  Google Scholar 

  4. Chen HH, Liu CM, Chang SL, et al. Automated extraction of left atrial volumes from two-dimensional computer tomography images using a deep learning technique [published online ahead of print, 2020 Apr 11]. Int J Cardiol. 2020;S0167–5273(20)30267–9. https://doi.org/10.1016/j.ijcard.2020.03.075

  5. Costello JP, Olivieri LJ, Su L, et al. Incorporating three-dimensional printing into a simulation-based congenital heart disease and critical care training curriculum for resident physicians. Congenit Heart Dis. 2015;10(2):185-190. https://doi.org/10.1111/chd.12238

    Article  PubMed  Google Scholar 

  6. Southworth MK, Silva JR, Silva JNA. Use of Extended Realities in Cardiology. Trends Cardiovas Med. 2019;30:143–148. https://doi.org/10.1016/j.tcm.2019.04.005

  7. Mena KA, Urbain KP, Fahey KM, Bramlet MT. Exploration of time sequential, patient specific 3D heart unlocks clinical understanding. 3d Print Medicine 2018;4:15. https://doi.org/10.1186/s41205-018-0034-7

  8. Fedorov A., Beichel R., Kalpathy-Cramer J., Finet J., Fillion-Robin J-C., Pujol S., Bauer C., Jennings D., Fennessy F., Sonka M., Buatti J., Aylward S.R., Miller J.V., Pieper S., Kikinis R. 3D Slicer as an Image Computing Platform for the Quantitative Imaging Network. Magnetic Resonance Imaging. 2012 Nov;30(9):1323-41. PMID: 22770690.

    Article  PubMed  PubMed Central  Google Scholar 

  9. Liangjia Zhu, Ivan Kolesov, Yi Gao, Ron Kikinis, Allen Tannenbaum. An Effective Interactive Medical Image Segmentation Method Using Fast GrowCut, International Conference on Medical Image Computing and Computer Assisted Intervention (MICCAI), Interactive Medical Image Computing Workshop, 2014

  10. Hadeed, K. et al. Feasibility and accuracy of printed models of complex cardiac defects in small infants from cardiac computed tomography. Pediatr Radiol 51, 1983–1990 (2021).

    Article  PubMed  Google Scholar 

  11. Wang Y, Zhang Y, Xuan W, et al. Fully automatic segmentation of 4D MRI for cardiac functional measurements. Med Phys. 2019;46(1):180-189. https://doi.org/10.1002/mp.13245

    Article  PubMed  Google Scholar 

  12. Bindschadler, M., Modgil, D., Branch, K. R., Riviere, P. J. L. & Alessio, A. M. Performance comparison between static and dynamic cardiac CT on perfusion quantitation and patient classification tasks. Proc Spie 941224 (2015) doi:https://doi.org/10.1117/12.2082098.

  13. Kim, B. et al. A Novel Virtual Reality Medical Image Display System for Group Discussions of Congenital Heart Disease: Development and Usability Testing. Jmir Cardio 4, e20633 (2020).

    Article  PubMed  PubMed Central  Google Scholar 

  14. Frederiksen, J. G. et al. Cognitive load and performance in immersive virtual reality versus conventional virtual reality simulation training of laparoscopic surgery: a randomized trial. Surg Endosc 34, 1244–1252 (2020).

    Article  PubMed  Google Scholar 

  15. Hamstra, S. J., Brydges, R., Hatala, R., Zendejas, B. & Cook, D. A. Reconsidering Fidelity in Simulation-Based Training. Acad Med 89, 387–392 (2014).

    Article  PubMed  Google Scholar 

  16. Awori, J. et al. 3D models improve understanding of congenital heart disease. 3d Print Medicine 7, 26 (2021).

  17. Lau, I., Gupta, A. & Sun, Z. Clinical Value of Virtual Reality versus 3D Printing in Congenital Heart Disease. Biomol 11, 884 (2021).

    CAS  Google Scholar 

  18. Wen, R., Chng, C.-B. & Chui, C.-K. Augmented Reality Guidance with Multimodality Imaging Data and Depth-Perceived Interaction for Robot-Assisted Surgery. Robotics 6, 13 (2017).

    Article  Google Scholar 

  19. Gonzalez AA, Lizana PA, Pino S, Miller BG, Merino C. Augmented reality-based learning for the comprehension of cardiac physiology in undergraduate biomedical students. Adv Physiol Educ. 2020;44(3):314-322. https://doi.org/10.1152/advan.00137.2019

    Article  PubMed  Google Scholar 

  20. Kang SL, Shkumat N, Dragulescu A, et al. Mixed-reality view of cardiac specimens: a new approach to understanding complex intracardiac congenital lesions [published online ahead of print, 2020 Jul 1]. Pediatr Radiol. 2020. https://doi.org/10.1007/s00247-020-04740-y. https://doi.org/10.1007/s00247-020-04740-y

  21. Napa S, Moore M, Bardyn T. Advancing Cardiac Surgery Case Planning and Case Review Conferences Using Virtual Reality in Medical Libraries: Evaluation of the Usability of Two Virtual Reality Apps. JMIR Hum Factors. 2019;6(1):e12008. Published 2019 Jan 16. https://doi.org/10.2196/12008

  22. Hettig J, Engelhardt S, Hansen C, Mistelbauer G. AR in VR: assessing surgical augmented reality visualizations in a steerable virtual reality environment. Int J Comput Assist Radiol Surg. 2018;13(11):1717-1725. https://doi.org/10.1007/s11548-018-1825-4

    Article  PubMed  Google Scholar 

  23. Wainman B, Pukas G, Wolak L, Mohanraj S, Lamb J, Norman GR. The Critical Role of Stereopsis in Virtual and Mixed Reality Learning Environments. Anat Sci Educ. 2020;13(3):401-412. https://doi.org/10.1002/ase.1928

    Article  PubMed  Google Scholar 

  24. van Helvoort D, Stobbe E, Benning R, Otgaar H, van de Ven V. Physical exploration of a virtual reality environment: Effects on spatiotemporal associative recognition of episodic memory. Mem Cognit. 2020;48(5):691-703. https://doi.org/10.3758/s13421-020-01024-6

    Article  PubMed  PubMed Central  Google Scholar 

Download references

Funding

This work was supported in part by an internal Seattle Children’s Heart Center grant awarded to Sujatha Buddhe, M.D.

Author information

Authors and Affiliations

  1. Department of Neurology, Seattle, WA, USA

    M. Bindschadler & S. D. Friedman

  2. Department of Pediatrics, Seattle Children’s Heart Center and the University of Washington, Seattle, WA, USA

    S. Buddhe & T. Jones

  3. Department of Radiology, University of Washington, Seattle, WA, USA

    M. R. Ferguson & R. K. Otto

  4. Department of Radiology, Seattle Childrens, Seattle, WA, USA

    M. Bindschadler, M. R. Ferguson & R. K. Otto

  5. Department of Improvement and Innovation, Seattle, WA, USA

    S. D. Friedman

Authors
  1. M. Bindschadler
    View author publications

    You can also search for this author in PubMed Google Scholar

  2. S. Buddhe
    View author publications

    You can also search for this author in PubMed Google Scholar

  3. M. R. Ferguson
    View author publications

    You can also search for this author in PubMed Google Scholar

  4. T. Jones
    View author publications

    You can also search for this author in PubMed Google Scholar

  5. S. D. Friedman
    View author publications

    You can also search for this author in PubMed Google Scholar

  6. R. K. Otto
    View author publications

    You can also search for this author in PubMed Google Scholar

Contributions

Michael Bindschadler, Ph.D. created the custom software modules for use with 3D Slicer and manuscript draft. All authors participated in manuscript review and editing, as well as reviewing the virtual Unity/Merge models.

Corresponding author

Correspondence to R. K. Otto.

Ethics declarations

Ethics Approval

A retrospective IRB exemption was obtained for this project for the use of the anonymized imaging data and informed consent from the subjects was not required. This data was previously acquired as part of a separate IRB-approved clinical trial. This study was performed in accordance with the institution’s ethical standards.

Consent to Participate

Not required under our institution’s retrospective IRB exemption.

Consent to Publish

Not required under our institution’s retrospective IRB exemption.

Conflicts of Interest

Authors have no conflicts relating to this work.

Additional information

Publisher's Note

Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.

Supplementary Information

Below is the link to the electronic supplementary material.

Supplementary file1 (MP4 2933 KB)

Supplementary file2 (MP4 1062 KB)

Supplementary file3 (MP4 4467 KB)

Supplementary file4 (MP4 2134 KB)

Supplementary file5 (MP4 3384 KB)

Supplementary file6 (MP4 3405 KB)

Supplementary file7 (MP4 6666 KB)

Supplementary file8 (MOV 47114 KB)

Rights and permissions

Reprints and permissions

About this article

Check for updates. Verify currency and authenticity via CrossMark

Cite this article

Bindschadler, M., Buddhe, S., Ferguson, M.R. et al. HEARTBEAT4D: An Open-source Toolbox for Turning 4D Cardiac CT into VR/AR. J Digit Imaging 35, 1759–1767 (2022). https://doi.org/10.1007/s10278-022-00659-y

Download citation

  • Received:

  • Revised:

  • Accepted:

  • Published:

  • Issue Date:

  • DOI: https://doi.org/10.1007/s10278-022-00659-y

Search

Navigation