Article

Scalable Earthquake Simulation on Petascale Supercomputers

Authors:

Thomas H. Jordan,

Dhabaleswar K. Panda,

Amit Chourasia,

Philip MaechlingAuthors Info & Claims

SC '10: Proceedings of the 2010 ACM/IEEE International Conference for High Performance Computing, Networking, Storage and Analysis

Pages 1 - 20

https://doi.org/10.1109/SC.2010.45

Published: 13 November 2010 Publication History

Abstract

Petascale simulations are needed to understand the rupture and wave dynamics of the largest earthquakes at shaking frequencies required to engineer safe structures (> 1 Hz). Toward this goal, we have developed a highly scalable, parallel application (AWP-ODC) that has achieved “M8”: a full dynamical simulation of a magnitude-8 earthquake on the southern San Andreas fault up to 2 Hz. M8 was calculated using a uniform mesh of 436 billion 40-m3 cubes to represent the three-dimensional crustal structure of Southern California, in a 800 km by 400 km area, home to over 20 million people. This production run producing 360 sec of wave propagation sustained 220 Tflop/s for 24 hours on NCCS Jaguar using 223,074 cores. As the largest-ever earthquake simulation, M8 opens new territory for earthquake science and engineering—the physics-based modeling of the largest seismic hazards with the goal of reducing their potential for loss of life and property.

References

[1]

B.T. Aagaard and T.H. Heaton, "Near-Source Ground Motions from Simulations of Sustained Intersonic and Supersonic Fault Ruptures," Bull. Seis. Soc. Am., vol. 94, no.6, 2004, pp. 2064-2078.

[2]

J. Berenger, "A Perfectly Match Layer for the Absorption of Electromagnetic Waves," J. Comput. Phys., vol. 114, no. 2, 1994, pp. 185-200.

Digital Library

[3]

J. Berenger, "Three-Dimensional Perfectly Matched Layer for the Absorption of Electromagnetic Waves," J. Comput. Phys., vol. 127, no. 2, 1996, pp. 363-379.

Digital Library

[4]

P. Bernard and D. Baumont, "Shear Mach Wave Characterization for Kinematic Fault Rupture Models with Constant Supershear Rupture Velocity," Geophys. J. Int'l, vol. 162, no. 2, 2005, pp. 431- 447.

[5]

J. Bielak, R. Graves, K.B. Olsen, R. Taborda, L. Ramirez-Guzman, S.M. Day, G. Ely, D. Roten, T.H. Jordan, P. Maechling, J. Urbanic, Y. Cui, and G. Juve, "The ShakeOut Earthquake Scenario: Verification of Three Simulation Sets," Geophys. J. Int'l, vol. 180, no. 1, 2010, pp. 375-404.

[6]

J.O. Blanch, J.O.A. Robertsson, and W.W. Symes, "Modeling of a Constant Q: Methodology and Algorithm for an Efficient and Optimally Inexpensive Viscoelastic Technique," Geophysics, vol. 60, no. 1, 1995, pp. 176-184.

[7]

D.M. Boore and G.M. Atkinson, "Ground-motion prediction equations for the average horizontal component of PGA, PGV, and 5%- damped PSA at spectral periods between 0.01 s and 10.0 s", Earthquake Spectra, vol. 24, 2008, pp. 99-138.

[8]

K.W. Campbell and Y. Bozorgnia, "NGA ground motion model for the geometric mean horizontal component of PGA, PGV, and 5%- damped PSA at spectral periods between 0.01 s and 10.0 s", Earthquake Spectra, vol. 24, 2008, pp. 139-171.

[9]

C. Cerjan, D. Kosloff, R. Kosloff, and M. Reshef, "A Nonreflecting Boundary Condition for Direct Acoustic and Elastic Wave Equations," Geophysics, vol. 50, no. 4, 1985, pp. 705-708.

[10]

P. Chen, L. Zhao, and T.H. Jordan, "Full 3D Tomography for the Crustal Structure of the Los Angeles Region," Bull. Seis. Soc. Am., vol. 97, no. 4, 2007, pp. 1094-1120.

[11]

Z. Chen, J.J. Dongarra, "Algorithm-Based Fault Tolerance for Fail-Stop Failures", IEEE Trans. Parallel and Distributed Systems, vol. 19, no. 12, 2008, pp. 1628-1641.

Digital Library

[12]

Y. Cui, R. Moore, K.B. Olsen, A. Chourasia, P. Maechling, B. Minster, S. Day, Y. Hu, J. Zhu, A. Majumdar, and T.H. Jordan, "Enabling Very-Large Scale Earthquake Simulations on Parallel Machines," Proc. Int'l Conf. Comput. Science, Lecture Notes in Computer Science series, vol. 4487, Springer-Verlag, 2007, pp. 46- 53.

Digital Library

[13]

Y. Cui, K.B. Olsen, A. Chourasia, R. Moore, P. Maechling, and T. H. Jordan, "The TeraShake Computational Platform for Large-Scale Earthquake Simulations," Advances in Geocomputing: Lecture Notes in Earth Sciences, vol. 119, Springer-Verlag, 2009, pp. 229- 278.

[14]

L.A. Dalguer and S.M. Day, "Staggered-Grid Split-Node Method for Spontaneous Rupture Simulation," J. Geophys. Res., vol. 112, B02302, 2007.

[15]

L.A. Dalguer and P.M. Mai, "Implications of Style-of-Faulting and Loading Characteristics on the Dynamic Rupture Process," EOS Trans., Am. Geophys. Union 89(53), Fall Meet. Suppl., 2008.

[16]

T.E. Dawson, T.K. Rockwell, R.J. Weldon II, and C.J. Wills, "Summary of Geologic Data and Developments of A Priori Rupture Models for the Elsinore, San Jacinto, and Garlock Faults," Appendix F to USGS Open File Report 2007-1437F, 2008.

[17]

S.M. Day, "Efficient Simulation of Constant Q Using Coarse-Grained Memory Variables," Bull. Seis. Soc. Am. vol. 88, no. 4, 1998, pp. 1051-1062.

[18]

S.M. Day and C.R. Bradley, "Memory-Efficient Simulation of Anelastic Wave Propagation," Bull. Seis. Soc. Am, vol. 91, no. 3, 2001, pp. 520-531.

[19]

E.M. Dunham and H.S. Bhat, "Attenuation of Radiated Ground Motion and Stresses from Three-Dimensional Supershear Ruptures," J. Geophys. Res., vol. 113, 2008.

[20]

M. Faerman, R. Moore, Y. Cui, Y. Hu, J. Zhu, B. Minister, and P. Maechling, "Managing Large Scale Data for Earthquake Simulations," J. Grid Comp., vol. 5, no. 3, 2007, pp. 295-302.

[21]

E.H. Field, T.E. Dawson, K.R. Felzer, A.D. Frankel, V. Gupta, T.H. Jordan, T. Parsons, M.D. Petersen, R.S. Stein, R.J. Weldon II, and C.J. Wills, "Uniform California Earthquake Rupture Forecast, Version 2 (UCERF 2)," Bull. Seis. Soc. Am. vol. 99, no. 4, 2009, pp. 2053-2107.

[22]

E. Gottschammer and K.B. Olsen, "Accuracy of the Explicit Planar Free-Surface Boundary Condition Implemented in a Fourth-Order Staggered-Grid Velocity-Stress Finite-Difference Scheme," Bull. Seis. Soc. Am., vol. 91, no. 3, 2001, pp. 617-623. 0120000244.

[23]

R.W. Graves, "Simulating Seismic Wave Propagation in 3D Elastic Media Using Staggered-Grid Finite Differences," Bull. Seis. Soc. Am., vol. 86, no. 4, 1996, pp. 1091-1106.

[24]

iRODS: Data Grid, Digital Libraries, Persistent Archives, and Real Time Data Systems, http://www.irods.org/.

[25]

T.H. Jordan and P. Maechling, "The SCEC community Modeling Environment: An Information Infrastructure for System-Level Earthquake Science," Seis. Res. Letter, vol. 74, no. 1, 2003, pp. 324- 32.

[26]

K. Lee, Y. Cui, P. Maechling, K.B. Olsen, and T.H. Jordan "Commnication Optimizations of SCEC CME AWP-Olsen Application for Petascale Computing," Supercomputing conf. (SC'09), 2009, Poster.

[27]

K. Lee, Y. Cui, T. Kaiser, P. Maechling, K.B. Olsen, and T.H. Jordan, "I/O Optimizations of SCEC AWP-Olsen Application for Petascale Earthquake Computing," Supercomputing conf. (SC'09), 2009, Poster.

[28]

M8 Simulation Visualization, http://visservices.sdsc.edu/projects/scec/m8/1.0.

[29]

P. Maechling, E. Deelman, and Y. Cui, "Implementing Software Acceptance Tests as Scientific Workflows," PDPTA CSREA Press, 2009, pp. 317-323.

[30]

C. Marcinkovich and K.B. Olsen, "On the Implementation of Perfectly Matched Layers in a 3D Fourth-Order Velocity-Stress Finite-Difference Scheme," J. Geophys. Res., vol. 108 (B5), 2276, 2003.

[31]

E. Mcquinn, A. Chourasia, B. Minster, and J. Schulze, "Visualizing Time-Dependent Seismic Vector Fields With Glyphs," Feb 2010, http://visservices.sdsc.edu/projects/scec/vectorviz/.

[32]

K.C. Meza-Fajardo and A.S. Papageorgiou, "A Nonconventional, Split-field, Perfectly Matched Layer for Wave Propagation in Isotropic and Anisotropic Elastic Media: Stability Analysis," Bull. Seis. Soc. Am., vol. 98, no. 4, 2008, pp. 1811-1836. 0120070223.

[33]

S.E. Minkoff, "Spatial Parallelism of a 3D Finite Difference Velocity-Stress Elastic Wave Propagation Code," SIAM J. Sci. Comput., vol. 24, no. 1, 2002, pp. 1-19.

Digital Library

[34]

NCCS, "Jaguar", 2010, http://www.nccs.gov/jaguar.

[35]

H. Nguyen, Y. Cui, K.B. Olsen, K. Lee, "Single CPU optimization of SCEC AWP-Olsen," SCEC Annual Meeting, 2009, Poster.

[36]

K.B. Olsen, "Simulation of Three-Dimensional Wave Propagation in the Salt Lake Basin," doctoral dissertation, Univ. of Utah, 1994, p. 157.

[37]

K.B. Olsen, S.M. Day, C.R. Bradley, "Estimation of Q for Long-Period (>2 s) Waves in the Los Angeles Basin," Bull. Seis. Soc. Am. vol. 93, no. 2, 2003, pp. 627-638.

[38]

K.B. Olsen, S.M. Day, J.B. Minster, Y. Cui, A. Chourasia, M. Faerman, R. Moore, P. Maechling, T.H. Jordan, "Strong Shaking in Los Angeles Expected from Southern San Andreas Earthquake," Geophys. Res. Letter, vol. 33, 2006, L07305. 2005GL025472.

[39]

K.B. Olsen, S.M. Day, J.B. Minster, Y. Cui, A. Chourasia, D. Okaya, P. Maechling, and T.H. Jordan, "TeraShake2: Spontaneous Rupture Simulations of Mw 7.7 Earthquakes on the Southern San Andreas Fault," Bull. Seism. Soc. Am., vol. 98, no. 3, 2008, pp. 1162-1185.

[40]

K.B. Olsen, W. J. Stephenson, and A. Geisselmeyer, "3D crustal structure and long-period ground motions from a M9.0 Megathrust Earthquake in the Pacific Northwest region," J. Seism., vol. 12, no. 2, April 2008, pp. 145-159.

[41]

K.B. Olsen, L.A. Dalguer, S.M. Day, Y. Cui, J. Zhu, V.M. Cruz, D. Roten, J. Mayhew, P. Maechling, T.H. Jordan, A. Chourasia, and D. Okaya, "ShakeOut-D: Ground Motion Estimates Using an Ensemble of Large Earthquakes on the Southern San Andreas Fault with Spontaneous Rupture Propagation," Geophys. Res. Letter, vol. 36, Feb. 2009, L04303.

[42]

S. Potluri, P. Lai, K. Tomko, S. Sur, Y. Cui, M. Tatineni, K. Schulz, W. Barth, A. Majumdar, and D.K. Panda, "Quantifying Performance Benefits of Overlap using MPI-2 in a Seismic Modeling Application", Proceedings of the 24th ACM Int'l Conference on Supercomputing, 2010, pp. 17-25.

Digital Library

[43]

D.P. Schwartz, and J. Coppersmith (1984), "Fault behaviour and characteristic earthquakes: examples from Wasatch and San Andreas faults", J. Geophys. Res., vol. 89, 1984, pp. 5681-5698.

[44]

R. Weldon, K. Scharer, T. Fumal, and G. Biasi, "Wrightwood and the Earthquake Cycle: What a Long Recurrence Record Tells Us about How Faults Work," Geol. Soc. Am., vol. 14, no. 9, 2004, pp. 4-10.

[45]

J. Zhou, Y. Cui, S. Davis, C.C. Guest, "Workflow-Based High Performance Data Transfer and Ingestion to Petascale Simulations on TeraGrid", IEEE Comput. Sciences and Optimization (CSO'10), vol. 1, 2010, pp. 343-347.

Digital Library

Cited By

de la Puente JRodriguez JMonterrubio-Velasco MRojas OFolch A(2020)Urgent Supercomputing of EarthquakesProceedings of the Platform for Advanced Scientific Computing Conference10.1145/3394277.3401853(1-8)Online publication date: 29-Jun-2020
https://dl.acm.org/doi/10.1145/3394277.3401853
Wei YYou XYang HLuan ZQian D(2020)Towards GPU Acceleration of Phonon Computation with ShengBTEProceedings of the International Conference on High Performance Computing in Asia-Pacific Region10.1145/3368474.3368487(32-42)Online publication date: 15-Jan-2020
https://dl.acm.org/doi/10.1145/3368474.3368487
Ji XYang BZhang TMa XZhu XWang XEl-Sayed NZhai JLiu WXue WMerchant AWeatherspoon H(2019)Automatic, application-aware I/O forwarding resource allocationProceedings of the 17th USENIX Conference on File and Storage Technologies10.5555/3323298.3323323(265-279)Online publication date: 25-Feb-2019
https://dl.acm.org/doi/10.5555/3323298.3323323
Show More Cited By

Scalable Earthquake Simulation on Petascale Supercomputers

Recommendations

18.9-Pflops nonlinear earthquake simulation on Sunway TaihuLight: enabling depiction of 18-Hz and 8-meter scenarios
SC '17: Proceedings of the International Conference for High Performance Computing, Networking, Storage and Analysis

This paper reports our large-scale nonlinear earthquake simulation software on Sunway TaihuLight. Our innovations include: (1) a customized parallelization scheme that employs the 10 million cores efficiently at both the process and the thread levels; (...
Petascale high order dynamic rupture earthquake simulations on heterogeneous supercomputers
SC '14: Proceedings of the International Conference for High Performance Computing, Networking, Storage and Analysis

We present an end-to-end optimization of the innovative Arbitrary high-order DERivative Discontinuous Galerkin (ADER-DG) software SeisSol targeting Intel^® Xeon Phi^™ coprocessor platforms, achieving unprecedented earthquake model complexity through ...
High-frequency nonlinear earthquake simulations on petascale heterogeneous supercomputers
SC '16: Proceedings of the International Conference for High Performance Computing, Networking, Storage and Analysis

The omission of nonlinear effects in large-scale 3D ground motion estimation, which are particularly challenging due to memory and scalability issues, can result in costly misguidance for structural design in earthquake-prone regions. We have ...

Comments

Please enable JavaScript to view thecomments powered by Disqus.

Information & Contributors

Information

Published In

cover image ACM Conferences

SC '10: Proceedings of the 2010 ACM/IEEE International Conference for High Performance Computing, Networking, Storage and Analysis

November 2010

634 pages

ISBN:9781424475599

Conference Chair:
Barry V. Hess

Sponsors

SIGARCH: ACM Special Interest Group on Computer Architecture
IEEE-CS: Computer Society

Publisher

IEEE Computer Society

United States

Publication History

Published: 13 November 2010

Check for updates

Qualifiers

Article

Conference

SC '10

Sponsor:

SIGARCH
IEEE-CS

SC '10: International Conference for High Performance Computing, Networking, Storage and Analysis

November 13 - 19, 2010

Acceptance Rates

SC '10 Paper Acceptance Rate 51 of 253 submissions, 20%;

Overall Acceptance Rate 1,516 of 6,373 submissions, 24%

Upcoming Conference

Contributors

Other Metrics

View Article Metrics

Bibliometrics & Citations

Bibliometrics

Article Metrics

29
Total Citations
View Citations
354
Total Downloads

Downloads (Last 12 months)3
Downloads (Last 6 weeks)1

Reflects downloads up to 22 Nov 2024

Other Metrics

View Author Metrics

Citations

Cited By

de la Puente JRodriguez JMonterrubio-Velasco MRojas OFolch A(2020)Urgent Supercomputing of EarthquakesProceedings of the Platform for Advanced Scientific Computing Conference10.1145/3394277.3401853(1-8)Online publication date: 29-Jun-2020
https://dl.acm.org/doi/10.1145/3394277.3401853
Wei YYou XYang HLuan ZQian D(2020)Towards GPU Acceleration of Phonon Computation with ShengBTEProceedings of the International Conference on High Performance Computing in Asia-Pacific Region10.1145/3368474.3368487(32-42)Online publication date: 15-Jan-2020
https://dl.acm.org/doi/10.1145/3368474.3368487
Ji XYang BZhang TMa XZhu XWang XEl-Sayed NZhai JLiu WXue WMerchant AWeatherspoon H(2019)Automatic, application-aware I/O forwarding resource allocationProceedings of the 17th USENIX Conference on File and Storage Technologies10.5555/3323298.3323323(265-279)Online publication date: 25-Feb-2019
https://dl.acm.org/doi/10.5555/3323298.3323323
Mu DMoran JZhou HCui YHawkins RTatineni MCampbell SFurlani T(2019)In-situ Analysis and Visualization of Earthquake SimulationPractice and Experience in Advanced Research Computing 2019: Rise of the Machines (learning)10.1145/3332186.3332201(1-5)Online publication date: 28-Jul-2019
https://dl.acm.org/doi/10.1145/3332186.3332201
Chen BFu HWei YHe CZhang WLi YWan WZhang WGan LZhang WZhang ZYang GChen X(2018)Simulating the Wenchuan earthquake with accurate surface topography on Sunway TaihuLightProceedings of the International Conference for High Performance Computing, Networking, Storage, and Analysis10.5555/3291656.3291710(1-12)Online publication date: 11-Nov-2018
https://dl.acm.org/doi/10.5555/3291656.3291710
Diep TFürlinger KThoai N(2018)MC-CCheckerProceedings of the 25th European MPI Users' Group Meeting10.1145/3236367.3236369(1-11)Online publication date: 23-Sep-2018
https://dl.acm.org/doi/10.1145/3236367.3236369
Jubertie SDupros FDe Martin F(2018)Vectorization of a spectral finite-element numerical kernelProceedings of the 2018 4th Workshop on Programming Models for SIMD/Vector Processing10.1145/3178433.3178441(1-7)Online publication date: 24-Feb-2018
https://dl.acm.org/doi/10.1145/3178433.3178441
Chen BFu HWei YHe CZhang WLi YWan WZhang WGan LZhang WZhang ZYang GChen X(2018)Simulating the Wenchuan earthquake with accurate surface topography on Sunway TaihuLightProceedings of the International Conference for High Performance Computing, Networking, Storage, and Analysis10.1109/SC.2018.00043(1-12)Online publication date: 11-Nov-2018
https://dl.acm.org/doi/10.1109/SC.2018.00043
Sourouri MRaknes EReissmann NLangguth JHackenberg DSchöne RKjeldsberg PMohr BRaghavan P(2017)Towards fine-grained dynamic tuning of HPC applications on modern multi-core architecturesProceedings of the International Conference for High Performance Computing, Networking, Storage and Analysis10.1145/3126908.3126945(1-12)Online publication date: 12-Nov-2017
https://dl.acm.org/doi/10.1145/3126908.3126945
Fu HHe CChen BYin ZZhang ZZhang WZhang TXue WLiu WYin WYang GChen XMohr BRaghavan P(2017)18.9-Pflops nonlinear earthquake simulation on Sunway TaihuLightProceedings of the International Conference for High Performance Computing, Networking, Storage and Analysis10.1145/3126908.3126910(1-12)Online publication date: 12-Nov-2017
https://dl.acm.org/doi/10.1145/3126908.3126910
Show More Cited By

View Options

Login options

Check if you have access through your login credentials or your institution to get full access on this article.

Full Access

Get this Publication

View options

PDF

View or Download as a PDF file.

eReader

View online with eReader.

Media

Figures

Other

Tables

View Table of Contents