Accelerating CyberShake Calculations on the XE6/XK7 Platform of Blue Waters

CyberShake is a computational platform developed by the Southern California Earthquake Center (SCEC) that explicitly incorporates earthquake rupture time histories and deterministic wave propagation effects into seismic hazard calculations through the use of 3D waveform simulations. Using CyberShake, SCEC has created the first physics-based probabilistic seismic hazard analysis (PSHA) models of the Los Angeles region from suites of simulations comprising ~108 seismograms. The current models are, however, limited to low seismic frequencies (≤ 0.5 Hz). To increase the maximum simulated frequency to above 1 Hz and produce a California state-wide model, we have transformed SCEC Anelastic Wave Propagation code (AWP-ODC) to include strain Green's tensor (SGT) calculations to accelerate CyberShake calculations. This tensor-valued wave field code has both CPU and GPU components in place for flexibility on different architectures. We have demonstrated the performance and scalability of this solver optimized for the heterogeneous Blue Waters system at NCSA. The high performance of the wave propagation computation, coupled with CPU/GPU co-scheduling capabilities of our workflow-managed systems, make a statewide hazard model a goal reachable with existing supercomputers.