research-article

18.9-Pflops nonlinear earthquake simulation on Sunway TaihuLight: enabling depiction of 18-Hz and 8-meter scenarios

Authors:

Wenqiang Zhang,

Tingjian Zhang,

Xiaofei ChenAuthors Info & Claims

SC '17: Proceedings of the International Conference for High Performance Computing, Networking, Storage and Analysis

Article No.: 2, Pages 1 - 12

https://doi.org/10.1145/3126908.3126910

Published: 12 November 2017 Publication History

Abstract

This paper reports our large-scale nonlinear earthquake simulation software on Sunway TaihuLight. Our innovations include: (1) a customized parallelization scheme that employs the 10 million cores efficiently at both the process and the thread levels; (2) an elaborate memory scheme that integrates on-chip halo exchange through register communcation, optimized blocking configuration guided by an analytic model, and coalesced DMA access with array fusion; (3) on-the-fly compression that doubles the maximum problem size and further improves the performance by 24%. With these innovations to remove the memory constraints of Sunway TaihuLight, our software achieves over 15% of the system's peak, better than the 11.8% efficiency achieved by a similar software running on Titan, whose byte to flop ratio is 5 times better than TaihuLight. The extreme cases demonstrate a sustained performance of over 18.9 Pflops, enabling the simulation of Tangshan earthquake as an 18-Hz scenario with an 8-meter resolution.

References

[1]

Don L Anderson. 1989. Theory of the Earth. Blackwell scientific publications.

[2]

Hesheng Bao, Jacobo Bielak, Omar Ghattas, Loukas F Kallivokas, David R O'hallaron, Jonathan R Shewchuk, and Jifeng Xu. 1996. Earthquake ground motion modeling on parallel computers. In Proceedings of the 1996 ACM/IEEE conference on Supercomputing. IEEE Computer Society, 13.

Digital Library

[3]

Alexander Breuer, Alexander Heinecke, and Yifeng Cui. 2017. EDGE: Extreme Scale Fused Seismic Simulations with the Discontinuous Galerkin Method. In International Supercomputing Conference. Springer, 41--60.

[4]

Laura Carrington, Dimitri Komatitsch, Michael Laurenzano, Mustafa M Tikir, David Michéa, Nicolas Le Goff, Allan Snavely, and Jeroen Tromp. 2008. High-frequency simulations of global seismic wave propagation using SPECFEM3D_GLOBE on 62K processors. In Proceedings of the 2008 ACM/IEEE conference on Supercomputing. IEEE Press, 60.

Digital Library

[5]

Charlie Catlett. 2002. The philosophy of TeraGrid: building an open, extensible, distributed TeraScale facility. In Cluster Computing and the Grid, 2002. 2nd IEEE/ACM International Symposium on. IEEE, 8--8.

Digital Library

[6]

Yifeng Cui, Reagan Moore, Kim Olsen, Amit Chourasia, Philip Maechling, Bernard Minster, Steven Day, Yuanfang Hu, Jing Zhu, Amitava Majumdar, and others. 2007. Enabling very-large scale earthquake simulations on parallel machines. In International Conference on Computational Science. Springer, 46--53.

Digital Library

[7]

Yifeng Cui, Kim B Olsen, Tomas H Jordan, Kwangyoon Lee, Jun Zhou, Patrick Small, Daniel Roten, Geoffrey Ely, Dhabaleswar K Panda, Amit Chourasia, and others. 2010. Scalable earthquake simulation on petascale supercomputers. In High Performance Computing, Networking, Storage and Analysis (SC), 2010 International Conference for. IEEE, 1--20.

Digital Library

[8]

Yifeng Cui, Efecan Poyraz, Kim B Olsen, Jun Zhou, Kyle Withers, Scott Callaghan, Jeff Larkin, C Guest, D Choi, Amit Chourasia, and others. 2013. Physics-based seismic hazard analysis on petascale heterogeneous supercomputers. In Proceedings of the International Conference on High Performance Computing, Networking, Storage and Analysis. ACM, 70.

Digital Library

[9]

Haohuan Fu, Junfeng Liao, Jinzhe Yang, Lanning Wang, Zhenya Song, Xiaomeng Huang, Chao Yang, Wei Xue, Fangfang Liu, Fangli Qiao, and others. 2016. The Sunway TaihuLight supercomputer: system and applications. Science China Information Sciences 59, 7 (2016), 072001.

[10]

Hui Guo, Wali Jiang, and XIE Xinsheng. 2011. New evidence for the distribution of surface rupture zone of the 1976 Ms 7.8 Tangshan earthquake. Seismology and geology 33, 3, Article 506 (2011), 18 pages.

[11]

Alexander Heinecke, Alexander Breuer, Sebastian Rettenberger, Michael Bader, Alice-Agnes Gabriel, Christian Pelties, Arndt Bode, William Barth, Xiang-Ke Liao, Karthikeyan Vaidyanathan, and others. 2014. Petascale high order dynamic rupture earthquake simulations on heterogeneous supercomputers. In Proceedings of the International Conference for High Performance Computing, Networking, Storage and Analysis. IEEE Press, 3--14.

Digital Library

[12]

Kuo-Hung Hsiao and YAN Hong-Sen. 2009. THE REVIEW OF RECONSTRUCTION DESIGNS OF ZHANG HENG'S SEISMOSCOPE. Journal of Japan Association for Earthquake Engineering 9, 4 (2009), 4_1--4_10.

[13]

Tsuyoshi Ichimura, Kohei Fujita, Pher Errol Balde Quinay, Lalith Maddegedara, Muneo Hori, Seizo Tanaka, Yoshihisa Shizawa, Hiroshi Kobayashi, and Kazuo Minami. 2015. Implicit nonlinear wave simulation with 1.08 T DOF and 0.270 T unstructured finite elements to enhance comprehensive earthquake simulation. In Proceedings of the International Conference for High Performance Computing, Networking, Storage and Analysis. ACM, 4.

Digital Library

[14]

Tsuyoshi Ichimura, Kohei Fujita, Seizo Tanaka, Muneo Hori, Maddegedara Lalith, Yoshihisa Shizawa, and Hiroshi Kobayashi. 2014. Physics-based urban earthquake simulation enhanced by 10.7 BlnDOFX 30 K time-step unstructured FE non-linear seismic wave simulation. In Proceedings of the International Conference for High Performance Computing, Networking, Storage and Analysis. IEEE Press, 15--26.

Digital Library

[15]

Dimitri Komatitsch, Seiji Tsuboi, Chen Ji, and Jeroen Tromp. 2003. A 14.6 billion degrees of freedom, 5 teraflops, 2.5 terabyte earthquake simulation on the Earth Simulator. In Supercomputing, 2003 ACM/IEEE Conference. IEEE, 4--4.

Digital Library

[16]

Q. Liu, J. Wang, J. Chen, S. Li, and B. Guo. 2007. Seismogenic Tectonic Environment of 1976 Great Tangshan Earthquake: Results from Dense Seismic Array Observations. Earth Science Frontiers 14 (2007), 205--212.

[17]

John Milne. 1886. Earthquakes and other earth movements. Vol. 56. D. Appleton.

[18]

Anthony Nguyen, Nadathur Satish, Jatin Chhugani, Changkyu Kim, and Pradeep Dubey. 2010. 3.5-D blocking optimization for stencil computations on modern CPUs and GPUs. In High Performance Computing, Networking, Storage and Analysis (SC), 2010 International Conference for. IEEE, 1--13.

Digital Library

[19]

Zehua Qiu, Jin Ma, and Liu Guoxi. 2005. Discovery of the great fault of the Tangshan earthquake. Seismology and geology 27, 4 (2005), 669--677.

[20]

Max Rietmann, Peter Messmer, Tarje Nissen-Meyer, Daniel Peter, Piero Basini, Dimitri Komatitsch, Olaf Schenk, Jeroen Tromp, Lapo Boschi, and Domenico Giardini. 2012. Forward and adjoint simulations of seismic wave propagation on emerging large-scale GPU architectures. In Proceedings of the International Conference on High Performance Computing, Networking, Storage and Analysis. IEEE Computer Society Press, 38.

Digital Library

[21]

Daniel Roten, Yifeng Cui, Kim B Olsen, Steven M Day, Kyle Withers, William H Savran, Peng Wang, and Dawei Mu. 2016. High-frequency nonlinear earthquake simulations on petascale heterogeneous supercomputers. In Proceedings of the International Conference for High Performance Computing, Networking, Storage and Analysis. IEEE Press, 82.

Digital Library

[22]

Seth Stein and Michael Wysession. 2009. An introduction to seismology, earthquakes, and earth structure. John Wiley & Sons.

[23]

Zhenguo Zhang, Wei Zhang, and Xiaofei Chen. 2014. Three-dimensional curved grid finite-difference modelling for non-planar rupture dynamics. Geophysical Journal International 199, 2 (2014), 860--879.

Cited By

Wu SCheng JMa JWang TYong XJi Y(2025)Million-core scalable 3D anisotropic reverse time migration on the Sugon exascale supercomputerComputers & Geosciences10.1016/j.cageo.2024.105754194(105754)Online publication date: Dec-2025
https://doi.org/10.1016/j.cageo.2024.105754
Lehmann FGatti FBertin MClouteau D(2024) Synthetic ground motions in heterogeneous geologies from various sources: the HEMEW S -3D database Earth System Science Data10.5194/essd-16-3949-202416:9(3949-3972)Online publication date: 3-Sep-2024
https://doi.org/10.5194/essd-16-3949-2024
Zhang WDong YCrempien JArduino PKurtulus ATaciroglu E(2024)A comparison of ground motions predicted through one-dimensional site response analyses and three-dimensional wave propagation simulations at regional scalesEarthquake Spectra10.1177/8755293024123193540:2(1215-1234)Online publication date: 28-Feb-2024
https://doi.org/10.1177/87552930241231935
Show More Cited By

18.9-Pflops nonlinear earthquake simulation on Sunway TaihuLight: enabling depiction of 18-Hz and 8-meter scenarios
1. Applied computing
  1. Physical sciences and engineering

Recommendations

Simulating the Wenchuan earthquake with accurate surface topography on Sunway TaihuLight
SC '18: Proceedings of the International Conference for High Performance Computing, Networking, Storage, and Analysis

This paper reports our efforts on performing a 50-m resolution earthquake simulation of the Wenchuan Earthquake (Ms 8.0, China) on Sunway TaihuLight. To accurately capture the surface topography, we adopt a curvilinear grid finite-difference method with ...
Performance Optimization of the HPCG Benchmark on the Sunway TaihuLight Supercomputer

In this article, we present some key techniques for optimizing HPCG on Sunway TaihuLight and demonstrate how to achieve high performance in memory-bound applications by exploiting specific characteristics of the hardware architecture. In particular, we ...
Simulating the Wenchuan earthquake with accurate surface topography on Sunway TaihuLight
SC '18: Proceedings of the International Conference for High Performance Computing, Networking, Storage, and Analysis

This paper reports our efforts on performing a 50-m resolution earthquake simulation of the Wenchuan Earthquake (Ms 8.0, China) on Sunway TaihuLight. To accurately capture the surface topography, we adopt a curvilinear grid finite-difference method with ...

Comments

Please enable JavaScript to view thecomments powered by Disqus.

Information & Contributors

Information

Published In

cover image ACM Conferences

SC '17: Proceedings of the International Conference for High Performance Computing, Networking, Storage and Analysis

November 2017

801 pages

ISBN:9781450351140

DOI:10.1145/3126908

General Chair:
Bernd Mohr
Jülich Supercomputing Center, Jülich, Germany
,
Program Chair:
Padma Raghavan
Vanderbilt University, Nashville, TN

Copyright © 2017 ACM.

Permission to make digital or hard copies of all or part of this work for personal or classroom use is granted without fee provided that copies are not made or distributed for profit or commercial advantage and that copies bear this notice and the full citation on the first page. Copyrights for components of this work owned by others than the author(s) must be honored. Abstracting with credit is permitted. To copy otherwise, or republish, to post on servers or to redistribute to lists, requires prior specific permission and/or a fee. Request permissions from [email protected].

Sponsors

SIGHPC: ACM Special Interest Group on High Performance Computing, Special Interest Group on High Performance Computing

In-Cooperation

IEEE CS

Publisher

Association for Computing Machinery

New York, NY, United States

Publication History

Published: 12 November 2017

Permissions

Request permissions for this article.

Request Permissions

Check for updates

Author Tags

Qualifiers

Research-article

Funding Sources

National Key Research & Development Plan of China
NSF of China
Shandong Province Taishan Scholar

Conference

SC '17

Sponsor:

SIGHPC

SC '17: The International Conference for High Performance Computing, Networking, Storage and Analysis

November 12 - 17, 2017

Colorado, Denver

Acceptance Rates

SC '17 Paper Acceptance Rate 61 of 327 submissions, 19%;

Overall Acceptance Rate 1,516 of 6,373 submissions, 24%

Upcoming Conference

Contributors

Other Metrics

View Article Metrics

Bibliometrics & Citations

Bibliometrics

Article Metrics

82
Total Citations
View Citations
1,446
Total Downloads

Downloads (Last 12 months)73
Downloads (Last 6 weeks)11

Reflects downloads up to 21 Nov 2024

Other Metrics

View Author Metrics

Citations

Cited By

Wu SCheng JMa JWang TYong XJi Y(2025)Million-core scalable 3D anisotropic reverse time migration on the Sugon exascale supercomputerComputers & Geosciences10.1016/j.cageo.2024.105754194(105754)Online publication date: Dec-2025
https://doi.org/10.1016/j.cageo.2024.105754
Lehmann FGatti FBertin MClouteau D(2024) Synthetic ground motions in heterogeneous geologies from various sources: the HEMEW S -3D database Earth System Science Data10.5194/essd-16-3949-202416:9(3949-3972)Online publication date: 3-Sep-2024
https://doi.org/10.5194/essd-16-3949-2024
Zhang WDong YCrempien JArduino PKurtulus ATaciroglu E(2024)A comparison of ground motions predicted through one-dimensional site response analyses and three-dimensional wave propagation simulations at regional scalesEarthquake Spectra10.1177/8755293024123193540:2(1215-1234)Online publication date: 28-Feb-2024
https://doi.org/10.1177/87552930241231935
Lindstrom PHittinger JDiffenderfer JFox AOsei-Kuffuor DBanks J(2024)ZFP: A compressed array representation for numerical computationsThe International Journal of High Performance Computing Applications10.1177/10943420241284023Online publication date: 23-Oct-2024
https://doi.org/10.1177/10943420241284023
Wang ZLiu TTao RRen ZLi YGuo YPan J(2024)Parallel optimization of plasma single particle simulation program based on Sunway Bluelight II supercomputerProceedings of the 2024 8th International Conference on High Performance Compilation, Computing and Communications10.1145/3675018.3675774(27-34)Online publication date: 7-Jun-2024
https://dl.acm.org/doi/10.1145/3675018.3675774
Liu ZChu XLv XMeng HLiu HZhu GFu HYang G(2024)SunwayLB: Enabling Extreme-Scale Lattice Boltzmann Method Based Computing Fluid Dynamics Simulations on Advanced Heterogeneous SupercomputersIEEE Transactions on Parallel and Distributed Systems10.1109/TPDS.2023.334370635:2(324-337)Online publication date: Feb-2024
https://doi.org/10.1109/TPDS.2023.3343706
Yuan MLiu QGan LYang G(2024)ESFLOW: Mapping Large-Scale Earthquake Simulation to Spatial Computing Systems2024 IEEE International Symposium on Circuits and Systems (ISCAS)10.1109/ISCAS58744.2024.10558278(1-5)Online publication date: 19-May-2024
https://doi.org/10.1109/ISCAS58744.2024.10558278
Liu ZHuang ZHe WYu HHuang L(2024)Three-dimensional broadband simulation of near-fault seismic ground motion considering complete source-path-site effectComputers and Structures10.1016/j.compstruc.2024.107319297:COnline publication date: 1-Jul-2024
https://dl.acm.org/doi/10.1016/j.compstruc.2024.107319
Wang YLiu B(2024)Hierarchical node method for solving large-scale sparse linear equations in parallel多重节点法并行求解大规模稀疏线性方程组Acta Mechanica Sinica10.1007/s10409-023-23350-x40:4Online publication date: 25-Jan-2024
https://doi.org/10.1007/s10409-023-23350-x
Ren ZLiu TLiu ZGuo YPan JZhao DWu XYang M(2024)SW-TRRM: Parallel Optimization Research of the Random Ray Method Based on Sunway Bluelight II SupercomputerAlgorithms and Architectures for Parallel Processing10.1007/978-981-97-0808-6_22(373-393)Online publication date: 27-Feb-2024
https://doi.org/10.1007/978-981-97-0808-6_22
Show More Cited By

View Options

Login options

Check if you have access through your login credentials or your institution to get full access on this article.

Full Access

Get this Publication

View options

PDF

View or Download as a PDF file.

eReader

View online with eReader.

Media

Figures

Other

Tables

View Table of Contents