research-article

Open access

SRT: Shape Reconstruction Transformer for 3D Reconstruction of Point Cloud from 2D MRI

Authors:

Shuqiang WangAuthors Info & Claims

ICMLC '22: Proceedings of the 2022 14th International Conference on Machine Learning and Computing

Pages 507 - 511

https://doi.org/10.1145/3529836.3529902

Published: 21 June 2022 Publication History

All formats PDF

Abstract

There has been some work on 3D reconstruction of organ shapes based on medical images in minimally invasive surgeries. They aim to help lift visualization limitations for procedures with poor visual environments. However, extant models are often based on deep convolutional neural networks and complex, hard-to-train generative adversarial networks; their problems about stability and real-time plague the further development of the technique. In this paper, we propose the Shape Reconstruction Transformer (SRT) based on the self-attentive mechanism and up-down-up generative structure to design fast and accurate 3D brain reconstruction models through fully connected layer networks only. Point clouds are used as the 3D representation of the model. Considering the specificity of the surgical scene, a single 2D image is limited as the input to the model. Qualitative demonstrations and quantitative experiments based on multiple metrics show the generative capability of the proposed model and demonstrate the advantages of the proposed model over other state-of-the-art models.

References

[1]

P. Achlioptas, O. Diamanti, I. Mitliagkas, and L. Guibas. 2018. Learning Representations and Generative Models for 3D Point Clouds. In Proc. PMLR 2018, Vol. 80. 40–49.

[2]

H.Q. Fan, H. Su, and L.J. Guibas. 2017. A Point Set Generation Network for 3D Object Reconstruction From a Single Image. In Proc. IEEE CVPR 2017.

[3]

Bowen Hu, Baiying Lei, Yanyan Shen, Yong Liu, and Shuqiang Wang. 2021. A Point Cloud Generative Model via Tree-Structured Graph Convolutions for 3D Brain Shape Reconstruction. In Chinese Conference on Pattern Recognition and Computer Vision (PRCV). Springer, 263–274.

Digital Library

[4]

Shengye Hu, Baiying Lei, Shuqiang Wang, Yong Wang, Zhiguang Feng, and Yanyan Shen. 2021. Bidirectional Mapping Generative Adversarial Networks for Brain MR to PET Synthesis. IEEE Transactions on Medical Imaging(2021).

[5]

Shengye Hu, Yanyan Shen, Shuqiang Wang, and Baiying Lei. 2020. Brain MR to PET Synthesis via Bidirectional Generative Adversarial Network. In International Conference on Medical Image Computing and Computer-Assisted Intervention. Springer, Cham, 698–707.

Digital Library

[6]

Shengye Hu, Jianpeng Yuan, and Shuqiang Wang. 2019. Cross-modality synthesis from MRI to PET using adversarial U-net with different normalization. In 2019 International Conference on Medical Imaging Physics and Engineering (ICMIPE). IEEE, 1–5.

[7]

Baiying Lei, Enmin Liang, Mengya Yang, Peng Yang, Feng Zhou, Ee-Leng Tan, Yi Lei, Chuan-Ming Liu, Tianfu Wang, Xiaohua Xiao, 2022. Predicting clinical scores for Alzheimer’s disease based on joint and deep learning. Expert Systems with Applications 187 (2022), 115966.

[8]

Baiying Lei, Mengya Yang, Peng Yang, Feng Zhou, Wen Hou, Wenbin Zou, Xia Li, Tianfu Wang, Xiaohua Xiao, and Shuqiang Wang. 2020. Deep and joint learning of longitudinal data for Alzheimer’s disease prediction. Pattern Recognition 102(2020), 107247.

Digital Library

[9]

Lu-Feng Mo and Shu-Qiang Wang. 2009. A variational approach to nonlinear two-point boundary value problems. Nonlinear Analysis: Theory, Methods & Applications 71, 12(2009), e834–e838.

[10]

Junren Pan, Baiying Lei, Yanyan Shen, Yong Liu, Zhiguang Feng, and Shuqiang Wang. 2021. Characterization Multimodal Connectivity of Brain Network by Hypergraph GAN for Alzheimer’s Disease Analysis. In Chinese Conference on Pattern Recognition and Computer Vision (PRCV). Springer, 467–478.

[11]

Junren Pan, Baiying Lei, Shuqiang Wang, Bingchuan Wang, Yong Liu, and Yanyan Shen. 2021. DecGAN: Decoupling Generative Adversarial Network detecting abnormal neural circuits for Alzheimer’s disease. arXiv preprint arXiv:2110.05712(2021).

[12]

Yanyan Shen, Xiaoxia Huang, Kyung Sup Kwak, Bo Yang, and Shuqiang Wang. 2016. Subcarrier-pairing-based resource optimization for OFDM wireless powered relay transmissions with time switching scheme. IEEE Transactions on Signal Processing 65, 5 (2016), 1130–1145.

Digital Library

[13]

D.W. Shu, S.W. Park, and J. Kwon. 2019. 3D Point Cloud Generative Adversarial Network Based on Tree Structured Graph Convolutions. In Proc. ICCV 2019.

[14]

Jonathan G Thomas, Wajd N Al-Holou, Dhiego Chaves de Almeida Bastos, Amol J Ghia, Jing Li, Andrew J Bishop, Behrang Amini, Laurence D Rhines, and Claudio E Tatsui. 2017. A novel use of the intraoperative MRI for metastatic spine tumors: laser interstitial thermal therapy for percutaneous treatment of epidural metastatic spine disease. Neurosurgery Clinics 28, 4 (2017), 513–524.

[15]

D. Valsesia, G. Fracastoro, and E. Magli. 2019. Learning Localized Generative Models for 3D Point Clouds via Graph Convolution. In Proc. ICLR 2019. https://openreview.net/forum?id=SJeXSo09FQ

[16]

Libby van Tonder, Sasha Burn, Anand Iyer, Jo Blair, Mohammed Didi, Michael Carter, Timothy Martland, Conor Mallucci, and Athanasius Chawira. 2018. Open resection of hypothalamic hamartomas for intractable epilepsy revisited, using intraoperative MRI. Child’s Nervous System 34, 9 (2018), 1663–1673.

[17]

Hongfei Wang, Yanyan Shen, Shuqiang Wang, Tengfei Xiao, Liming Deng, Xiangyu Wang, and Xinyan Zhao. 2019. Ensemble of 3D densely connected convolutional network for diagnosis of mild cognitive impairment and Alzheimer’s disease. Neurocomputing 333(2019), 145–156.

Digital Library

[18]

Shuqiang Wang, Yanyan Shen, Changhong Shi, Peng Yin, Zuhui Wang, Prudence Wing-Hang Cheung, Jason Pui Yin Cheung, Keith Dip-Kei Luk, and Yong Hu. 2018. Skeletal maturity recognition using a fully automated system with convolutional neural networks. IEEE Access 6(2018), 29979–29993.

[19]

Shuqiang Wang, Yanyan Shen, Dewei Zeng, and Yong Hu. 2018. Bone age assessment using convolutional neural networks. In 2018 International Conference on Artificial Intelligence and Big Data (ICAIBD). IEEE, 175–178.

[20]

Shuqiang Wang, Hongfei Wang, Yanyan Shen, and Xiangyu Wang. 2018. Automatic recognition of mild cognitive impairment and alzheimers disease using ensemble based 3d densely connected convolutional networks. In 2018 17th IEEE International conference on machine learning and applications (ICMLA). IEEE, 517–523.

[21]

Shuqiang Wang, Xiangyu Wang, Yong Hu, Yanyan Shen, Zhile Yang, Min Gan, and Baiying Lei. 2020. Diabetic retinopathy diagnosis using multichannel generative adversarial network with semisupervision. IEEE Transactions on Automation Science and Engineering (2020).

[22]

Shuqiang Wang, Xiangyu Wang, Yanyan Shen, Bing He, Xinyan Zhao, Prudence Wing-Hang Cheung, Jason Pui Yin Cheung, Keith Dip-Kei Luk, and Yong Hu. 2020. An ensemble-based densely-connected deep learning system for assessment of skeletal maturity. IEEE Transactions on Systems, Man, and Cybernetics: Systems (2020).

[23]

Shu-Qiang Wang. 2009. A variational approach to nonlinear two-point boundary value problems. Computers & Mathematics with Applications 58, 11-12 (2009), 2452–2455.

Digital Library

[24]

Shu-Qiang Wang and Ji-Huan He. 2007. Variational iteration method for solving integro-differential equations. Physics letters A 367, 3 (2007), 188–191.

[25]

Shu-Qiang Wang and Ji-Huan He. 2008. Variational iteration method for a nonlinear reaction-diffusion process. International Journal of Chemical Reactor Engineering 6, 1(2008).

[26]

Kun Wu, Yanyan Shen, and Shuqiang Wang. 2018. 3D convolutional neural network for regional precipitation nowcasting. Journal of Image and Signal Processing 7, 4 (2018), 200–212.

[27]

Chenchen Xiao, Legeng Chen, and Shuqiang Wang. 2021. Cross-modality PET synthesis method based on residual and adversarial networks. Computer Engineering and Applications(2021). http://kns.cnki.net/kcms/detail/11.2127.TP.20210223.1336.014.html

[28]

Senrong You, Shengye Hu, Yanyan Shen, and Shuqiang Wang. 2021. Review and Prospects for Generative Adversarial Networks on Medical Image Computation. Computer Science and Application 11, 7 (2021). https://doi.org/10.12677/CSA.2021.117200

[29]

Shuangzhi Yu, Shuqiang Wang, Xiaohua Xiao, Jiuwen Cao, Guanghui Yue, Dongdong Liu, Tianfu Wang, Yanwu Xu, and Baiying Lei. 2020. Multi-scale enhanced graph convolutional network for early mild cognitive impairment detection. In International Conference on Medical Image Computing and Computer-Assisted Intervention. Springer, Cham, 228–237.

Digital Library

[30]

Wen Yu, Baiying Lei, Yong Liu, Zhiguang Feng, Yong Hu, Yanyan Shen, Shuqiang Wang, and Michael K Ng. 2021. Morphological feature visualization of Alzheimer’s disease via Multidirectional Perception GAN. IEEE Transactions on Neural Networks and Learning Systems (2021).

[31]

Wen Yu, Baiying Lei, Michael K Ng, Albert C Cheung, Yanyan Shen, and Shuqiang Wang. 2021. Tensorizing GAN with high-order pooling for Alzheimer’s disease assessment. IEEE Transactions on Neural Networks and Learning Systems (2021).

[32]

Xiao-Yun Zhou, Zhao-Yang Wang, Peichao Li, Jian-Qing Zheng, and Guang-Zhong Yang. 2019. One-stage shape instantiation from a single 2D image to 3D point cloud. In International Conference on Medical Image Computing and Computer-Assisted Intervention. Springer, 30–38.

Digital Library

[33]

Qiankun Zuo, Baiying Lei, Yanyan Shen, Yong Liu, Zhiguang Feng, and Shuqiang Wang. 2021. Multimodal Representations Learning and Adversarial Hypergraph Fusion for Early Alzheimer’s Disease Prediction. In Chinese Conference on Pattern Recognition and Computer Vision (PRCV). Springer, 479–490.

Cited By

Chen ZZhang HHu DTan CLiu ZYang Y(2024)Point-Cloud Transformer for 3-D Electrical Impedance TomographyIEEE Transactions on Instrumentation and Measurement10.1109/TIM.2024.341316173(1-9)Online publication date: 2024
https://doi.org/10.1109/TIM.2024.3413161
Xia KWang J(2023)Recent advances of Transformers in medical image analysis: A comprehensive reviewMedComm – Future Medicine10.1002/mef2.382:1Online publication date: 24-Mar-2023
https://doi.org/10.1002/mef2.38
左乾(2022)Adversarial Graph Autoencoder for Brain Network Analysis in Alzheimer’s DiseaseJournal of Image and Signal Processing10.12677/JISP.2022.11401911:04(191-201)Online publication date: 2022
https://doi.org/10.12677/JISP.2022.114019

Recommendations

Tomographic surface reconstruction from point cloud

Inspired by computed tomography (CT), this paper presents a novel surface reconstruction algorithm, tomographic surface reconstruction, to reconstruct a surface mesh from a point cloud equipped with oriented normals. In the process of scanning a real ...
A Point Cloud Generative Model via Tree-Structured Graph Convolutions for 3D Brain Shape Reconstruction
Pattern Recognition and Computer Vision
Abstract
Fusing medical images and the corresponding 3D shape representation can provide complementary information and microstructure details to improve the operational performance and accuracy in brain surgery. However, compared to the substantial image ...
Deep learning based MRI reconstruction with transformer
Highlights
- New approach leveraging Transformer’s powerful processing ability to recover structure in deep MRI reconstruction.
- Proposed models achieve best reconstruction performances in reconstruction tasks of different contrasts and anatomies.
Abstract
Magnetic resonance imaging (MRI) has become one of the most powerful imaging techniques in medical diagnosis, yet the prolonged scanning time becomes a bottleneck for application. Reconstruction methods based on compress sensing (CS) have made ...

Comments

Please enable JavaScript to view thecomments powered by Disqus.

Information & Contributors

Information

Published In

cover image ACM Other conferences

ICMLC '22: Proceedings of the 2022 14th International Conference on Machine Learning and Computing

February 2022

570 pages

ISBN:9781450395700

DOI:10.1145/3529836

Copyright © 2022 Owner/Author.

This work is licensed under a Creative Commons Attribution International 4.0 License.

Publisher

Association for Computing Machinery

New York, NY, United States

Publication History

Published: 21 June 2022

Check for updates

Author Tags

Qualifiers

Research-article
Research
Refereed limited

Funding Sources

National Natural Science Foundations of China under Grants

Conference

ICMLC 2022

ICMLC 2022: 2022 14th International Conference on Machine Learning and Computing

February 18 - 21, 2022

Guangzhou, China

Contributors

Other Metrics

View Article Metrics

Bibliometrics & Citations

Bibliometrics

Article Metrics

3
Total Citations
View Citations
566
Total Downloads

Downloads (Last 12 months)234
Downloads (Last 6 weeks)40

Reflects downloads up to 21 Nov 2024

Other Metrics

View Author Metrics

Citations

Cited By

Chen ZZhang HHu DTan CLiu ZYang Y(2024)Point-Cloud Transformer for 3-D Electrical Impedance TomographyIEEE Transactions on Instrumentation and Measurement10.1109/TIM.2024.341316173(1-9)Online publication date: 2024
https://doi.org/10.1109/TIM.2024.3413161
Xia KWang J(2023)Recent advances of Transformers in medical image analysis: A comprehensive reviewMedComm – Future Medicine10.1002/mef2.382:1Online publication date: 24-Mar-2023
https://doi.org/10.1002/mef2.38
左乾(2022)Adversarial Graph Autoencoder for Brain Network Analysis in Alzheimer’s DiseaseJournal of Image and Signal Processing10.12677/JISP.2022.11401911:04(191-201)Online publication date: 2022
https://doi.org/10.12677/JISP.2022.114019

View Options

View options

PDF

View or Download as a PDF file.

eReader

View online with eReader.

HTML Format

View this article in HTML Format.

Login options

Check if you have access through your login credentials or your institution to get full access on this article.

Full Access

Get this Publication

Media

Figures

Other

Tables

View Table of Contents