Nothing Special   »   [go: up one dir, main page]

skip to main content
10.1145/3688868.3689194acmconferencesArticle/Chapter ViewAbstractPublication PagesmmConference Proceedingsconference-collections
research-article
Free access

Non-Invasive to Invasive: Enhancing FFA Synthesis from CFP with a Benchmark Dataset and a Novel Network

Published: 31 October 2024 Publication History

Abstract

Fundus imaging is a pivotal tool in ophthalmology, and different imaging modalities are characterized by their specific advantages. For example, Fundus Fluorescein Angiography (FFA) uniquely provides detailed insights into retinal vascular dynamics and pathology, surpassing Color Fundus Photographs (CFP) in detecting microvascular abnormalities and perfusion status. However, the conventional invasive FFA involves discomfort and risks due to fluorescein dye injection, and it is meaningful but challenging to synthesize FFA images from non-invasive CFP. Previous studies primarily focused on FFA synthesis in a single disease category. In this work, we explore FFA synthesis in multiple diseases by devising a Diffusion-guided generative adversarial network, which introduces an adaptive and dynamic diffusion forward process into the discriminator and adds a category-aware representation enhancer. Moreover, to facilitate this research, we collect the first multi-disease CFP and FFA paired dataset, named the Multi-disease Paired Ocular Synthesis (MPOS) dataset, with four different fundus diseases. Experimental results show that our FFA synthesis network can generate better FFA images compared to state-of-the-art methods. Furthermore, we introduce a paired-modal diagnostic network to validate the effectiveness of synthetic FFA images in the diagnosis of multiple fundus diseases, and the results show that our synthesized FFA images with the real CFP images have higher diagnosis accuracy than that of the compared FFA synthesizing methods. Our research bridges the gap between non-invasive imaging and FFA, thereby offering promising prospects to enhance ophthalmic diagnosis and patient care, with a focus on reducing harm to patients through non-invasive procedures. Our dataset and code will be released to support further research in this field (https://github.com/whq-xxh/FFA-Synthesis).

References

[1]
Alessandro Arrigo, Emanuela Aragona, Maurizio Battaglia Parodi, and Francesco Bandello. 2023. Quantitative approaches in multimodal fundus imaging: state of the art and future perspectives. Progress in Retinal and Eye Research, Vol. 92 (2023), 101111.
[2]
Mikołaj Bi'nkowski, Danica J Sutherland, Michael Arbel, and Arthur Gretton. 2018. Demystifying mmd gans. arXiv preprint arXiv:1801.01401 (2018).
[3]
Florinel-Alin Croitoru, Vlad Hondru, Radu Tudor Ionescu, and Mubarak Shah. 2023. Diffusion models in vision: A survey. IEEE Transactions on Pattern Analysis and Machine Intelligence (2023).
[4]
Onat Dalmaz, Mahmut Yurt, and Tolga cCukur. 2022. ResViT: Residual vision transformers for multimodal medical image synthesis. IEEE Transactions on Medical Imaging, Vol. 41, 10 (2022), 2598--2614.
[5]
Prafulla Dhariwal and Alexander Nichol. 2021. Diffusion models beat gans on image synthesis. Advances in neural information processing systems, Vol. 34 (2021), 8780--8794.
[6]
Zhaojie Fang, Zhanghao Chen, Pengxue Wei, Wangting Li, Shaochong Zhang, Ahmed Elazab, Gangyong Jia, Ruiquan Ge, and Changmiao Wang. 2023. UWAT-GAN: Fundus Fluorescein Angiography Synthesis via Ultra-Wide-Angle Transformation Multi-scale GAN. In International Conference on Medical Image Computing and Computer-Assisted Intervention. Springer, 745--755.
[7]
Huazhu Fu, Jun Cheng, Yanwu Xu, Changqing Zhang, Damon Wing Kee Wong, Jiang Liu, and Xiaochun Cao. 2018. Disc-aware ensemble network for glaucoma screening from fundus image. IEEE transactions on medical imaging, Vol. 37, 11 (2018), 2493--2501.
[8]
Huan Fu, Mingming Gong, Chaohui Wang, Kayhan Batmanghelich, Kun Zhang, and Dacheng Tao. 2019. Geometry-consistent generative adversarial networks for one-sided unsupervised domain mapping. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. 2427--2436.
[9]
Zhiyuan Gao, Xiangji Pan, Ji Shao, Xiaoyu Jiang, Zhaoan Su, Kai Jin, and Juan Ye. 2023. Automatic interpretation and clinical evaluation for fundus fluorescein angiography images of diabetic retinopathy patients by deep learning. British Journal of Ophthalmology, Vol. 107, 12 (2023), 1852--1858.
[10]
Ian Goodfellow, Jean Pouget-Abadie, Mehdi Mirza, Bing Xu, David Warde-Farley, Sherjil Ozair, Aaron Courville, and Yoshua Bengio. 2020. Generative adversarial networks. Commun. ACM, Vol. 63, 11 (2020), 139--144.
[11]
Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. 2016. Deep residual learning for image recognition. In Proceedings of the IEEE conference on computer vision and pattern recognition. 770--778.
[12]
Martin Heusel, Hubert Ramsauer, Thomas Unterthiner, Bernhard Nessler, and Sepp Hochreiter. 2017. Gans trained by a two time-scale update rule converge to a local nash equilibrium. Advances in neural information processing systems, Vol. 30 (2017).
[13]
Jonathan Ho, Ajay Jain, and Pieter Abbeel. 2020. Denoising diffusion probabilistic models. Advances in neural information processing systems, Vol. 33 (2020), 6840--6851.
[14]
Ming Hu, Siyuan Yan, Peng Xia, Feilong Tang, Wenxue Li, Peibo Duan, Lin Zhang, and Zongyuan Ge. 2024. Diffusion Model Driven Test-Time Image Adaptation for Robust Skin Lesion Classification. arXiv preprint arXiv:2405.11289 (2024).
[15]
Kun Huang, Mingchao Li, Jiale Yu, Jinxin Miao, Zizhong Hu, Songtao Yuan, and Qiang Chen. 2023. Lesion-aware generative adversarial networks for color fundus image to fundus fluorescein angiography translation. Computer Methods and Programs in Biomedicine, Vol. 229 (2023), 107306.
[16]
Phillip Isola, Jun-Yan Zhu, Tinghui Zhou, and Alexei A Efros. 2017. Image-to-image translation with conditional adversarial networks. In Proceedings of the IEEE conference on computer vision and pattern recognition. 1125--1134.
[17]
Sharif Amit Kamran, Khondker Fariha Hossain, Alireza Tavakkoli, Stewart Lee Zuckerbrod, and Salah A Baker. 2021. Vtgan: Semi-supervised retinal image synthesis and disease prediction using vision transformers. In Proceedings of the IEEE/CVF international conference on computer vision. 3235--3245.
[18]
Lingke Kong, Chenyu Lian, Detian Huang, Yanle Hu, Qichao Zhou, et al. 2021. Breaking the dilemma of medical image-to-image translation. Advances in Neural Information Processing Systems, Vol. 34 (2021), 1964--1978.
[19]
Kris A Kwiterovich, Maureen G Maguire, Robert P Murphy, Andrew P Schachat, Neil M Bressler, Susan B Bressler, and Stuart L Fine. 1991. Frequency of adverse systemic reactions after fluorescein angiography: results of a prospective study. Ophthalmology, Vol. 98, 7 (1991), 1139--1142.
[20]
Tao Li, Wang Bo, Chunyu Hu, Hong Kang, Hanruo Liu, Kai Wang, and Huazhu Fu. 2021. Applications of deep learning in fundus images: A review. Medical Image Analysis, Vol. 69 (2021), 101971.
[21]
Tianhong Li, Dina Katabi, and Kaiming He. 2023. Self-conditioned Image Generation via Generating Representations. arXiv preprint arXiv:2312.03701 (2023).
[22]
Duoru Lin, Jianhao Xiong, Congxin Liu, Lanqin Zhao, Zhongwen Li, Shanshan Yu, Xiaohang Wu, Zongyuan Ge, Xinyue Hu, Bin Wang, et al. 2021. Application of Comprehensive Artificial intelligence Retinal Expert (CARE) system: a national real-world evidence study. The Lancet Digital Health, Vol. 3, 8 (2021), e486--e495.
[23]
Lihao Liu, Xiaowei Hu, Lei Zhu, Chi-Wing Fu, Jing Qin, and Pheng-Ann Heng. 2020. ψ-net: Stacking densely convolutional lstms for sub-cortical brain structure segmentation. IEEE transactions on medical imaging, Vol. 39, 9 (2020), 2806--2817.
[24]
Dong Nie, Roger Trullo, Jun Lian, Li Wang, Caroline Petitjean, Su Ruan, Qian Wang, and Dinggang Shen. 2018. Medical image synthesis with deep convolutional adversarial networks. IEEE Transactions on Biomedical Engineering, Vol. 65, 12 (2018), 2720--2730.
[25]
Taesung Park, Alexei A Efros, Richard Zhang, and Jun-Yan Zhu. 2020. Contrastive learning for unpaired image-to-image translation. In Computer Vision--ECCV 2020: 16th European Conference, Glasgow, UK, August 23--28, 2020, Proceedings, Part IX 16. Springer, 319--345.
[26]
Olaf Ronneberger, Philipp Fischer, and Thomas Brox. 2015. U-net: Convolutional networks for biomedical image segmentation. In Medical Image Computing and Computer-Assisted Intervention--MICCAI 2015: 18th International Conference, Munich, Germany, October 5--9, 2015, Proceedings, Part III 18. Springer, 234--241.
[27]
Muhammad Muneeb Saad, Ruairi O'Reilly, and Mubashir Husain Rehmani. 2024. A survey on training challenges in generative adversarial networks for biomedical image analysis. Artificial Intelligence Review, Vol. 57, 2 (2024), 19.
[28]
Divya Saxena and Jiannong Cao. 2021. Generative adversarial networks (GANs) challenges, solutions, and future directions. ACM Computing Surveys (CSUR), Vol. 54, 3 (2021), 1--42.
[29]
Yang Song and Stefano Ermon. 2020. Improved techniques for training score-based generative models. Advances in neural information processing systems, Vol. 33 (2020), 12438--12448.
[30]
Richard F Spaide, James M Klancnik, and Michael J Cooney. 2015. Retinal vascular layers imaged by fluorescein angiography and optical coherence tomography angiography. JAMA ophthalmology, Vol. 133, 1 (2015), 45--50.
[31]
Qing-Qing Tang, Xiang-Gang Yang, Hong-Qiu Wang, Da-Wen Wu, and Mei-Xia Zhang. 2024. Applications of deep learning for detecting ophthalmic diseases with ultrawide-field fundus images. International Journal of Ophthalmology, Vol. 17, 1 (2024), 188.
[32]
Miao Tian, Hongqiu Wang, Yingxue Sun, Shaozhi Wu, Qingqing Tang, and Meixia Zhang. 2023. Fine-grained attention & knowledge-based collaborative network for diabetic retinopathy grading. Heliyon, Vol. 9, 7 (2023).
[33]
Hongqiu Wang, Jian Chen, Shichen Zhang, Yuan He, Jinfeng Xu, Mengwan Wu, Jinlan He, Wenjun Liao, and Xiangde Luo. 2024. Dual-reference source-free active domain adaptation for nasopharyngeal carcinoma tumor segmentation across multiple hospitals. IEEE Transactions on Medical Imaging (2024).
[34]
Hongqiu Wang, Yueming Jin, and Lei Zhu. 2023. Dynamic Interactive Relation Capturing via Scene Graph Learning for Robotic Surgical Report Generation. In 2023 IEEE International Conference on Robotics and Automation (ICRA). IEEE, 2702--2709.
[35]
Hongqiu Wang, Xiangde Luo, Wu Chen, Qingqing Tang, Mei Xin, Qiong Wang, and Lei Zhu. 2024. Advancing UWF-SLO Vessel Segmentation with Source-Free Active Domain Adaptation and a Novel Multi-Center Dataset. arXiv preprint arXiv:2406.13645 (2024).
[36]
Hongqiu Wang, Yingxue Sun, Yunjian Cao, Ganlu Ouyang, Xin Wang, Shaozhi Wu, and Miao Tian. 2022. Classification for diabetic retinopathy by using staged convolutional neural network. In 2022 Asia Conference on Algorithms, Computing and Machine Learning (CACML). IEEE, 228--233.
[37]
Hongqiu Wang, Wei Wang, Haipeng Zhou, Huihui Xu, Shaozhi Wu, and Lei Zhu. 2024. Language-Driven Interactive Shadow Detection. In ACM Multimedia 2024.
[38]
Hongqiu Wang, Guang Yang, Shichen Zhang, Jing Qin, Yike Guo, Bo Xu, Yueming Jin, and Lei Zhu. 2024 d. Video-instrument synergistic network for referring video instrument segmentation in robotic surgery. IEEE Transactions on Medical Imaging (2024).
[39]
Zhendong Wang, Huangjie Zheng, Pengcheng He, Weizhu Chen, and Mingyuan Zhou. 2022. Diffusion-gan: Training gans with diffusion. arXiv preprint arXiv:2206.02262 (2022).
[40]
Hongtao Wu, Yijun Yang, Haoyu Chen, Jingjing Ren, and Lei Zhu. 2023. Mask-Guided Progressive Network for Joint Raindrop and Rain Streak Removal in Videos. In Proceedings of the 31st ACM International Conference on Multimedia. 7216--7225.
[41]
Hongtao Wu, Yijun Yang, Huihui Xu, Weiming Wang, Jinni Zhou, and Lei Zhu. 2024. RainMamba: Enhanced Locality Learning with State Space Models for Video Deraining. arXiv preprint arXiv:2407.21773 (2024).
[42]
Zhaohu Xing, Liang Wan, Huazhu Fu, Guang Yang, and Lei Zhu. 2023. Diff-UNet: A Diffusion Embedded Network for Volumetric Segmentation. arXiv preprint arXiv:2303.10326 (2023).
[43]
Zhaohu Xing, Tian Ye, Yijun Yang, Guang Liu, and Lei Zhu. 2024. Segmamba: Long-range sequential modeling mamba for 3d medical image segmentation. arXiv preprint arXiv:2401.13560 (2024).
[44]
Zhaohu Xing, Lequan Yu, Liang Wan, Tong Han, and Lei Zhu. 2022. NestedFormer: Nested modality-aware transformer for brain tumor segmentation. In International Conference on Medical Image Computing and Computer-Assisted Intervention. Springer, 140--150.
[45]
Zhaohu Xing, Lei Zhu, Lequan Yu, Zhiheng Xing, and Liang Wan. 2024. Hybrid Masked Image Modeling for 3D Medical Image Segmentation. IEEE Journal of Biomedical and Health Informatics (2024).
[46]
Yijun Yang, Huazhu Fu, Angelica I Aviles-Rivero, Carola-Bibiane Schönlieb, and Lei Zhu. 2023. Diffmic: Dual-guidance diffusion network for medical image classification. In International Conference on Medical Image Computing and Computer-Assisted Intervention. Springer, 95--105.
[47]
Yijun Yang, Hongtao Wu, Angelica I Aviles-Rivero, Yulun Zhang, Jing Qin, and Lei Zhu. 2024. Genuine Knowledge from Practice: Diffusion Test-Time Adaptation for Video Adverse Weather Removal. arXiv preprint arXiv:2403.07684 (2024).
[48]
Kaishen Yuan, Zitong Yu, Xin Liu, Weicheng Xie, Huanjing Yue, and Jingyu Yang. 2024. AUFormer: Vision Transformers are Parameter-Efficient Facial Action Unit Detectors. arXiv preprint arXiv:2403.04697 (2024).
[49]
Richard Zhang, Phillip Isola, Alexei A Efros, Eli Shechtman, and Oliver Wang. 2018. The unreasonable effectiveness of deep features as a perceptual metric. In Proceedings of the IEEE conference on computer vision and pattern recognition. 586--595.
[50]
Haipeng Zhou, Hongqiu Wang, Tian Ye, Zhaohu Xing, Jun Ma, Ping Li, Qiong Wang, and Lei Zhu. 2024. Timeline and Boundary Guided Diffusion Network for Video Shadow Detection. In ACM Multimedia 2024.
[51]
Yukun Zhou, Mark A Chia, Siegfried K Wagner, Murat S Ayhan, Dominic J Williamson, Robbert R Struyven, Timing Liu, Moucheng Xu, Mateo G Lozano, Peter Woodward-Court, et al. 2023. A foundation model for generalizable disease detection from retinal images. Nature, Vol. 622, 7981 (2023), 156--163.
[52]
Jun-Yan Zhu, Taesung Park, Phillip Isola, and Alexei A Efros. 2017. Unpaired image-to-image translation using cycle-consistent adversarial networks. In Proceedings of the IEEE international conference on computer vision. 2223--2232.

Index Terms

  1. Non-Invasive to Invasive: Enhancing FFA Synthesis from CFP with a Benchmark Dataset and a Novel Network

    Recommendations

    Comments

    Please enable JavaScript to view thecomments powered by Disqus.

    Information & Contributors

    Information

    Published In

    cover image ACM Conferences
    MCHM'24: Proceedings of the 1st International Workshop on Multimedia Computing for Health and Medicine
    October 2024
    85 pages
    ISBN:9798400711954
    DOI:10.1145/3688868
    Permission to make digital or hard copies of all or part of this work for personal or classroom use is granted without fee provided that copies are not made or distributed for profit or commercial advantage and that copies bear this notice and the full citation on the first page. Copyrights for components of this work owned by others than the author(s) must be honored. Abstracting with credit is permitted. To copy otherwise, or republish, to post on servers or to redistribute to lists, requires prior specific permission and/or a fee. Request permissions from [email protected].

    Sponsors

    Publisher

    Association for Computing Machinery

    New York, NY, United States

    Publication History

    Published: 31 October 2024

    Permissions

    Request permissions for this article.

    Check for updates

    Author Tags

    1. color fundus photographs
    2. diffusion models
    3. fundus diseases
    4. fundus fluorescein angiography
    5. medical image synthesis

    Qualifiers

    • Research-article

    Funding Sources

    • the Guangzhou-HKUST(GZ) Joint Funding Program
    • Xinjiang Key Laboratory of Artificial Intelligence Assisted Imaging Diagnosis Fund, and the Nansha Key Area Science and Technology Project

    Conference

    MM '24
    Sponsor:
    MM '24: The 32nd ACM International Conference on Multimedia
    October 28 - November 1, 2024
    Melbourne VIC, Australia

    Contributors

    Other Metrics

    Bibliometrics & Citations

    Bibliometrics

    Article Metrics

    • 0
      Total Citations
    • 33
      Total Downloads
    • Downloads (Last 12 months)33
    • Downloads (Last 6 weeks)33
    Reflects downloads up to 19 Nov 2024

    Other Metrics

    Citations

    View Options

    View options

    PDF

    View or Download as a PDF file.

    PDF

    eReader

    View online with eReader.

    eReader

    Login options

    Media

    Figures

    Other

    Tables

    Share

    Share

    Share this Publication link

    Share on social media