Cited By
View all- Klawonn ALanser MWeber J(2024)Machine learning and domain decomposition methods - a surveyComputational Science and Engineering10.1007/s44207-024-00003-y1:1Online publication date: 23-Sep-2024
Breaking Boundaries: Distributed Domain Decomposition with Scalable Physics-Informed Neural PDE Solvers
Authors: 
Arthur Feeney, Zitong Li, Ramin Bostanabad, 
Aparna ChandramowlishwaranAuthors Info & Claims
Article No.: 80, Pages 1 - 15
Abstract
Mosaic Flow is a novel domain decomposition method designed to scale physics-informed neural PDE solvers to large domains. Its unique approach leverages pre-trained networks on small domains to solve partial differential equations on large domains purely through inference, resulting in high reusability. This paper presents an end-to-end parallelization of Mosaic Flow, combining data parallel training and domain parallelism for inference on large-scale problems. By optimizing the network architecture and data parallel training, we significantly reduce the training time for learning the Laplacian operator to minutes on 32 GPUs. Moreover, our distributed domain decomposition algorithm enables scalable inferences for solving the Laplace equation on domains 4096× larger than the training domain, demonstrating strong scaling while maintaining accuracy on 32 GPUs. The reusability of Mosaic Flow, combined with the improved performance achieved through the distributed-memory algorithms, makes it a promising tool for modeling complex physical phenomena and accelerating scientific discovery.
Supplemental Material
MP4 File - SC23 paper presentation recording for "Breaking Boundaries: Distributed Domain Decomposition with Scalable Physics-Informed Neural PDE Solvers"
SC23 paper presentation recording for "Breaking Boundaries: Distributed Domain Decomposition with Scalable Physics-Informed Neural PDE Solvers", by Arthur Feeney, Zitong Li, Ramin Bostanabad and Aparna Chandramowlishwaran
- Download
- 211.44 MB
References
[1]
Satish Balay, Shrirang Abhyankar, Mark Adams, Jed Brown, Peter Brune, Kris Buschelman, Lisandro Dalcin, Alp Dener, Victor Eijkhout, W Gropp, et al. 2019. PETSc users manual. (2019).
[2]
Nathan Bell, Luke N. Olson, and Jacob Schroder. 2022. PyAMG: Algebraic Multi-grid Solvers in Python. Journal of Open Source Software 7, 72 (2022), 4142.
[3]
Steven L Brunton, Bernd R Noack, and Petros Koumoutsakos. 2020. Machine learning for fluid mechanics. Annual review of fluid mechanics 52 (2020), 477--508.
[4]
Ernie Chan, Marcel Heimlich, Avi Purkayastha, and Robert Van De Geijn. 2007. Collective communication: theory, practice, and experience. Concurrency and Computation: Practice and Experience 19, 13 (2007), 1749--1783.
[5]
Lisandro Dalcin and Yao-Lung L Fang. 2021. mpi4py: Status update after 12 years of development. Computing in Science & Engineering 23, 4 (2021), 47--54.
[6]
Victorita Dolean, Martin J Gander, Walid Kheriji, Felix Kwok, and Roland Masson. 2016. Nonlinear preconditioning: How to use a nonlinear Schwarz method to precondition Newton's method. SIAM Journal on Scientific Computing 38, 6 (2016), A3357--A3380.
[7]
Victorita Dolean, Alexander Heinlein, Siddhartha Mishra, and Ben Moseley. 2022. Finite basis physics-informed neural networks as a Schwarz domain decomposition method. arXiv preprint arXiv:2211.05560 (2022).
[8]
Victorita Dolean, Alexander Heinlein, Siddhartha Mishra, and Ben Moseley. 2023. Multilevel domain decomposition-based architectures for physics-informed neural networks. arXiv preprint arXiv:2306.05486 (2023).
[9]
Victorita Dolean, Pierre Jolivet, and Frédéric Nataf. 2015. An introduction to domain decomposition methods: algorithms, theory, and parallel implementation. SIAM.
[10]
Olivier Dubois, Martin Gander, Sébastien Loisel, Amik St-Cyr, and Daniel Szyld. 2009. The Optimized Schwarz Method with a Coarse Grid Correction. SIAM Journal on Scientific Computing 34 (11 2009).
[11]
Olivier Dubois, Martin J Gander, Sébastien Loisel, Amik St-Cyr, and Daniel B Szyld. 2012. The optimized Schwarz method with a coarse grid correction. SIAM Journal on Scientific Computing 34, 1 (2012), A421--A458.
[12]
Vikas Dwivedi, Nishant Parashar, and Balaji Srinivasan. 2021. Distributed learning machines for solving forward and inverse problems in partial differential equations. Neurocomputing 420 (2021), 299--316.
[13]
Lawrence C. Evans. 2010. Partial differential equations. American Mathematical Society, Providence, R.I.
[14]
Steven Farrell, Murali Emani, Jacob Balma, Lukas Drescher, Aleksandr Drozd, Andreas Fink, Geoffrey Fox, David Kanter, Thorsten Kurth, Peter Mattson, Dawei Mu, Amit Ruhela, Kento Sato, Koichi Shirahata, Tsuguchika Tabaru, Aristeidis Tsaris, Jan Balewski, Ben Cumming, Takumi Danjo, Jens Domke, Takaaki Fukai, Naoto Fukumoto, Tatsuya Fukushi, Balazs Gerofi, Takumi Honda, Toshiyuki Imamura, Akihiko Kasagi, Kentaro Kawakami, Shuhei Kudo, Akiyoshi Kuroda, Maxime Martinasso, Satoshi Matsuoka, Henrique Mendonça, Kazuki Minami, Prabhat Ram, Takashi Sawada, Mallikarjun Shankar, Tom St. John, Akihiro Tabuchi, Venkatram Vishwanath, Mohamed Wahib, Masafumi Yamazaki, and Junqi Yin. 2021. MLPerf HPC: A Holistic Benchmark Suite for Scientific Machine Learning on HPC Systems. In IEEE/ACM Workshop on Machine Learning in High Performance Computing Environments (MLHPC). 1--45.
[15]
Martin J Gander. 2006. Optimized schwarz methods. SIAM J. Numer. Anal. 44, 2 (2006), 699--731.
[16]
Martin Jakob Gander. 2008. Schwarz methods over the course of time. Electronic transactions on numerical analysis 31 (2008), 228--255.
[17]
Ehsan Haghighat and Ruben Juanes. 2021. SciANN: A Keras/TensorFlow wrapper for scientific computations and physics-informed deep learning using artificial neural networks. Computer Methods in Applied Mechanics and Engineering 373 (2021), 113552.
[18]
Sheikh Md Shakeel Hassan, Arthur Feeney, Akash Dhruv, Jihoon Kim, Youngjoon Suh, Jaiyoung Ryu, Yoonjin Won, and Aparna Chandramowlishwaran. 2023. BubbleML: A Multi-Physics Dataset and Benchmarks for Machine Learning. arXiv preprint arXiv:2307.14623 (2023).
[19]
Frédéric Hecht. 2012. New development in FreeFem++. Journal of numerical mathematics 20, 3--4 (2012), 251--266.
[20]
Dan Hendrycks and Kevin Gimpel. 2016. Gaussian Error Linear Units (GELUs). arXiv preprint arXiv:1606.08415 (2016).
[21]
Oliver Hennigh, Susheela Narasimhan, Mohammad Amin Nabian, Akshay Subramaniam, Kaustubh Tangsali, Zhiwei Fang, Max Rietmann, Wonmin Byeon, and Sanjay Choudhry. 2021. NVIDIA SimNet™: An AI-accelerated multi-physics simulation framework. In Computational Science-ICCS 2021: 21st International Conference, Krakow, Poland, June 16--18, 2021, Proceedings, Part V. Springer, 447--461.
[22]
Torsten Hoefler and Roberto Belli. 2015. Scientific Benchmarking of Parallel Computing Systems: Twelve Ways to Tell the Masses When Reporting Performance Results. In Proceedings of the International Conference for High Performance Computing, Networking, Storage and Analysis (Austin, Texas) (SC '15). Association for Computing Machinery, New York, NY, USA, Article 73, 12 pages.
[23]
Ismayil Ismayilov, Javid Baydamirli, Doğan Sağbili, Mohamed Wahib, and Didem Unat. 2023. Multi-GPU Communication Schemes for Iterative Solvers: When CPUs Are Not in Charge. In Proceedings of the 37th International Conference on Supercomputing (Orlando, FL, USA) (ICS '23). Association for Computing Machinery, New York, NY, USA, 192--202.
[24]
Ameya D Jagtap and George E Karniadakis. 2021. Extended Physics-informed Neural Networks (XPINNs): A Generalized Space-Time Domain Decomposition based Deep Learning Framework for Nonlinear Partial Differential Equations. In AAAI Spring Symposium: MLPS. 2002--2041.
[25]
Ameya D Jagtap, Ehsan Kharazmi, and George Em Karniadakis. 2020. Conservative physics-informed neural networks on discrete domains for conservation laws: Applications to forward and inverse problems. Computer Methods in Applied Mechanics and Engineering 365 (2020), 113028.
[26]
George Em Karniadakis, Ioannis G Kevrekidis, Lu Lu, Paris Perdikaris, Sifan Wang, and Liu Yang. 2021. Physics-informed machine learning. Nature Reviews Physics 3, 6 (2021), 422--440.
[27]
Ehsan Kharazmi, Zhongqiang Zhang, and George Em Karniadakis. 2021. hp-VPINNs: Variational physics-informed neural networks with domain decomposition. Computer Methods in Applied Mechanics and Engineering 374 (2021), 113547.
[28]
Nikola B. Kovachki, Zongyi Li, Burigede Liu, Kamyar Azizzadenesheli, Kaushik Bhattacharya, Andrew M. Stuart, and Anima Anandkumar. 2021. Neural Operator: Learning Maps Between Function Spaces. CoRR abs/2108.08481 (2021).
[29]
Aditi Krishnapriyan, Amir Gholami, Shandian Zhe, Robert Kirby, and Michael W Mahoney. 2021. Characterizing possible failure modes in physics-informed neural networks. Advances in Neural Information Processing Systems 34 (2021), 26548--26560.
[30]
Stig Larrson and Vidar Thomée. 2003. Partial Differential Equations with Numerical Methods. Springer Berlin, Heidelberg.
[31]
Jean-Yves Le Boudec. 2010. Performance Evaluation of Computer and Communication Systems. EPFL Press, Lausanne, Switzerland.
[32]
Ke Li, Kejun Tang, Tianfan Wu, and Qifeng Liao. 2019. D3M: A deep domain decomposition method for partial differential equations. IEEE Access 8 (2019), 5283--5294.
[33]
Shen Li, Yanli Zhao, Rohan Varma, Omkar Salpekar, Pieter Noordhuis, Teng Li, Adam Paszke, Jeff Smith, Brian Vaughan, Pritam Damania, and Soumith Chintala. 2020. PyTorch distributed: experiences on accelerating data parallel training. In Proceedings of the VLDB Endowment. 3005--3018.
[34]
Wuyang Li, Xueshuang Xiang, and Yingxiang Xu. 2020. Deep domain decomposition method: Elliptic problems. In Mathematical and Scientific Machine Learning. PMLR, 269--286.
[35]
Zongyi Li, Nikola Kovachki, Kamyar Azizzadenesheli, Burigede Liu, Kaushik Bhattacharya, Andrew Stuart, and Anima Anandkumar. 2020. Fourier Neural Operator for Parametric Partial Differential Equations. ICLR.
[36]
Pierre-Louis Lions et al. 1988. On the Schwarz alternating method. I. In First international symposium on domain decomposition methods for partial differential equations, Vol. 1. Paris, France, 42.
[37]
Ilya Loshchilov and Frank Hutter. 2019. Decoupled Weight Decay Regularization. In International Conference on Learning Representations. https://openreview.net/forum?id=Bkg6RiCqY7
[38]
Lu Lu, Pengzhan Jin, Guofei Pang, Zhongqiang Zhang, and George Em Karniadakis. 2021. Learning nonlinear operators via DeepONet based on the universal approximation theorem of operators. Nature machine intelligence 3, 3 (2021), 218--229.
[39]
Lu Lu, Xuhui Meng, Zhiping Mao, and George Em Karniadakis. 2021. DeepXDE: A deep learning library for solving differential equations. SIAM review 63, 1 (2021), 208--228.
[40]
Stefano Markidis. 2021. The old and the new: Can physics-informed deep-learning replace traditional linear solvers? Frontiers in big Data (2021), 92.
[41]
G. M. Morton. 1966. A Computer Oriented Geodetic DataBase and a New Technique in File Sequencing. Tech.rep.,IBM, (1966). https://dominoweb.draco.res.ibm.com/0dabf9473b9c86d48525779800566a39.html
[42]
Ben Moseley, Andrew Markham, and Tarje Nissen-Meyer. 2021. Finite Basis Physics-Informed Neural Networks (FBPINNs): a scalable domain decomposition approach for solving differential equations. arXiv preprint arXiv:2107.07871 (2021).
[43]
Octavi Obiols-Sales, Abhinav Vishnu, Nicholas Malaya, and Aparna Chandramowliswharan. 2020. CFDNet: A deep learning-based accelerator for fluid simulations. In Proceedings of the 34th ACM international conference on supercomputing. 1--12.
[44]
Octavi Obiols-Sales, Abhinav Vishnu, Nicholas P Malaya, and Aparna Chandramowlishwaran. 2021. SURFNet: Super-resolution of turbulent flows with transfer learning using small datasets. In 2021 30th International Conference on Parallel Architectures and Compilation Techniques (PACT). IEEE, 331--344.
[45]
Johannes Pekkilä, Miikka S Väisälä, Maarit J Käpylä, Matthias Rheinhardt, and Oskar Lappi. 2022. Scalable communication for high-order stencil computations using CUDA-aware MPI. Parallel Comput. 111 (2022), 102904.
[46]
Maziar Raissi, Paris Perdikaris, and George E Karniadakis. 2019. Physics-informed neural networks: A deep learning framework for solving forward and inverse problems involving nonlinear partial differential equations. Journal of Computational physics 378 (2019), 686--707.
[47]
Hermann Amandus Schwarz. 1869. Ueber einige Abbildungsaufgaben. (1869).
[48]
Yeonjong Shin, Jermone Darbon, and George Karniadakis. 2020. On the convergence of physics informed neural networks for linear second-order elliptic and parabolic type PDEs. In Communications in Computational Physics. 2042--2074.
[49]
Khemraj Shukla, Ameya D Jagtap, and George Em Karniadakis. 2021. Parallel physics-informed neural networks via domain decomposition. J. Comput. Phys. 447 (2021), 110683.
[50]
I.M. Sobol. 1998. On quasi-Monte Carlo integrations. Mathematics and Computers in Simulation 47, 2 (1998), 103--112.
[51]
Makoto Takamoto, Timothy Praditia, Raphael Leiteritz, Daniel MacKinlay, Francesco Alesiani, Dirk Pflüger, and Mathias Niepert. 2022. PDEBench: An extensive benchmark for scientific machine learning. Advances in Neural Information Processing Systems 35 (2022), 1596--1611.
[52]
Hengjie Wang and Aparna Chandramowlishwaran. 2020. Pencil: A pipelined algorithm for distributed stencils. In SC20: International Conference for High Performance Computing, Networking, Storage and Analysis. IEEE, 1--16.
[53]
Hengjie Wang, Robert Planas, Aparna Chandramowlishwaran, and Ramin Bostanabad. 2022. Mosaic flows: A transferable deep learning framework for solving PDEs on unseen domains. Computer Methods in Applied Mechanics and Engineering 389 (2022), 114424.
[54]
Sifan Wang, Yujun Teng, and Paris Perdikaris. 2021. Understanding and mitigating gradient flow pathologies in physics-informed neural networks. SIAM Journal on Scientific Computing 43, 5 (2021), A3055--A3081.
[55]
Sifan Wang, Hanwen Wang, and Paris Perdikaris. 2021. Learning the solution operator of parametric partial differential equations with physics-informed Deep-ONets. Science Advances 7, 40 (2021).
[56]
Sifan Wang, Xinling Yu, and Paris Perdikaris. 2022. When and why PINNs fail to train: A neural tangent kernel perspective. J. Comput. Phys. 449 (2022), 110768.
[57]
Yang You, Sashank Reddi, Jonathan Hseu, Sanjiv Kumar, Srinadh Bhojanapalli, Xiaodan Song, James Demmel, Kurt Keutzer, and Cho-Jui Hsieh. 2020. Large Batch Optimization for Deep Learning: Training BERT in 76 minutes. In ICLR. 2042--2074.
Index Terms
- Breaking Boundaries: Distributed Domain Decomposition with Scalable Physics-Informed Neural PDE Solvers
Recommendations
Domain decomposition preconditioning for parallel PDE software
Engineering computational technologyDomain decomposition methods have been applied to the solution of engineering problems for many years. Over the past two decades however the growth in the use of parallel computing platforms has ensured that interest in these methods, which offer the ...
Parallel physics-informed neural networks via domain decomposition
Highlights- Construction and implementation of new domain-decomposition based parallel algorithm is proposed for cPINNs and XPINNs methods.
AbstractWe develop a distributed framework for the physics-informed neural networks (PINNs) based on two recent extensions, namely conservative PINNs (cPINNs) and extended PINNs (XPINNs), which employ domain decomposition in space and in time-...
Using multiple levels of parallelism to enhance the performance of domain decomposition solvers
Large-scale scientific simulations are nowadays fully integrated in many scientific and industrial applications. Many of these simulations rely on modelisations based on PDEs that lead to the solution of huge linear or nonlinear systems of equations ...
Comments
Please enable JavaScript to view thecomments powered by Disqus.Information & Contributors
Information
Published In
November 2023
1428 pages
ISBN:9798400701092
DOI:10.1145/3581784
- Chair:
- Dorian Arnold,
- Program Chair:
- Rosa M Badia,
- Program Co-chair:
- Kathryn Mohror
Copyright © 2023 Owner/Author(s).
This work is licensed under a Creative Commons Attribution International 4.0 License.
Sponsors
Publisher
Association for Computing Machinery
New York, NY, United States
Publication History
Published: 11 November 2023
Check for updates
Author Tags
Qualifiers
- Research-article
Funding Sources
Conference
SC '23
Sponsor:
SC '23: International Conference for High Performance Computing, Networking, Storage and Analysis
November 12 - 17, 2023
CO, Denver, USA
Acceptance Rates
Overall Acceptance Rate 1,516 of 6,373 submissions, 24%
Upcoming Conference
Contributors
Other Metrics
Bibliometrics & Citations
Bibliometrics
Article Metrics
- View Citations1Total Citations
- 512Total Downloads
- Downloads (Last 12 months)512
- Downloads (Last 6 weeks)42
Reflects downloads up to 21 Nov 2024
Other Metrics
Citations
Cited By
View all- Klawonn ALanser MWeber J(2024)Machine learning and domain decomposition methods - a surveyComputational Science and Engineering10.1007/s44207-024-00003-y1:1Online publication date: 23-Sep-2024
View Options
Login options
Check if you have access through your login credentials or your institution to get full access on this article.
Sign in