research-article

Scalable implicit finite element solver for massively parallel processing with demonstration to 160K cores

Authors:

Mark S. Shephard,

Kenneth E. JansenAuthors Info & Claims

SC '09: Proceedings of the Conference on High Performance Computing Networking, Storage and Analysis

Article No.: 68, Pages 1 - 12

https://doi.org/10.1145/1654059.1654129

Published: 14 November 2009 Publication History

Abstract

Implicit methods for partial differential equations using unstructured meshes allow for an efficient solution strategy for many real-world problems (e.g., simulation-based virtual surgical planning). Scalable solvers employing these methods not only enable solution of extremely-large practical problems but also lead to dramatic compression in time-to-solution. We present a parallelization paradigm and associated procedures that enable our implicit, unstructured flow-solver to achieve strong scalability.

We consider fluid-flow examples in two application areas to show the effectiveness of our procedures that yield near-perfect strong-scaling on various (including near-petascale) systems. The first area includes a double-throat nozzle (DTN) whereas the second considers a patient-specific abdominal aortic aneurysm (AAA) model. We present excellent strong-scaling on three cases ranging from relatively small to large; a DTN model with O(10⁶) elements up to 8,192 cores (9 core-doublings), an AAA model with O(10⁸) elements up to 32,768 cores (6 core-doublings) and O(10⁹) elements up to 163,840 cores.

References

[1]

E. Boman, K. Devine, R. Heaphy, B. Hendrickson, V. Leung, L. Riesen, C. Vaughan, U. Catalyurek, D. Bozdag, W. Mitchell, and J. Teresco. Zoltan 3.0: Parallel Partitioning, Load-balancing, and Data Management Services; User's Guide. Sandia National Laboratories, Albuquerque, NM, 2007. Tech. Report SAND2007-4748W, http://www.cs.sandia.gov/Zoltan/ug_html/ug.html.

[2]

U. Catalyurek, E. Boman, K. Devine, D. Bozdag, R. Heaphy, and L. Riesen. Hypergraph-based dynamic load balancing for adaptive scientific computations. In Proc. of IEEE Intl. Parallel and Distributed Processing Symp. (IPDPS), Long Beach, CA, USA, March, 2007. IEEE.

[3]

T. J. R. Hughes, L. Mazzei, and K. E. Jansen. Large-eddy simulation and the variational multiscale method. Computing and Visualization in Science, 3:47--59, 2000.

Digital Library

[4]

IBM Blue Gene team. Overview of the IBM BG/P project. IBM J. of Res. and Development, 52(1/2):199--220, 2008.

Digital Library

[5]

IBM XL Compiler Development Group. Exploiting the dual FPU in BG/L. In Online IBM Support Documentation, 2006.

[6]

K. E. Jansen, C. H. Whiting, and G. M. Hulbert. A generalized-α method for integrating the filtered Navier-Stokes equations with a stabilized finite element method. Comp. Meth. Appl. Mech. Engng., 190:305--319, 1999.

[7]

G. Karypis and V. Kumar. Parallel multilevel k-way partitioning scheme for irr. graphs. SIAM Review, 41:278--300, 1999.

Digital Library

[8]

A. S. Les, C. P. Chang, M. T. Darney, C. A. Figueroa, J. F. LaDisa, J. M. Park, R. J. Herfkens, R. L. Dalman, and C. A. Taylor. Human abdominal aortic aneurysm hemodynamics during rest and simulated exercise. In Proc. of the ASME 2007 Summer Bioengineering Conference, Keystone, CO, USA, 2007.

[9]

Y. Saad and M. Schultz. GMRES: A generalized minimal residual algorithm for solving nonsymmetric linear systems. SIAM Journal of Scientific and Statistical Computing, 7:856--869, 1986.

Digital Library

[10]

O. Sahni, K. E. Jansen, M. S. Shephard, C. A. Taylor, and M. W. Beall. Adaptive boundary layer meshing for viscous flow simulations. Engng. with Comp., 24(3):267--285, 2008.

Digital Library

[11]

O. Sahni, J. Müller, K. E. Jansen, M. S. Shephard, and C. A. Taylor. Efficient anisotropic adaptive discretization of the cardiovascular system. Comp. Meth. Appl. Mech. Engng., 195(41--43):5634--5655, 2006.

[12]

A. M. Sallam and N. H. Hwang. Human red blood cell hemolysis in a turbulent shear flow: contribution of reynolds shear stresses. Biorheology, 21:783--797, 1984.

[13]

E. Seol and M. Shephard. Efficient distributed mesh data structure for parallel automated adaptive analysis. Engng. with Comp., 22(3):197--213, 2006.

Digital Library

[14]

F. Shakib, T. J. R. Hughes, and Z. Johan. A multi-element group preconditioned GMRES algorithm for nonsymmetric systems arising in finite element analysis. Comp. Meth. Appl. Mech. Engng., 75:415--456, 1989.

[15]

P. D. Stein and H. N. Sabbah. Measured turbulence and its effect on thrombus formation. Circ. Research, 35:608--614, 1974.

[16]

A. E. Tejada-Martínez and K. E. Jansen. On the interaction between dynamic model dissipation and numerical dissipation due to streamline upwind/Petrov-Galerkin stabilization. Comp. Meth. Appl. Mech. Engng., 194(9--11):1225--1248, 2005.

[17]

A. E. Tejada-Martínez and K. E. Jansen. A parameter-free dynamic subgrid-scale model for large-eddy simulation. Comp. Meth. Appl. Mech. Engng., 195(23--24):2919--2938, 2006.

[18]

C. H. Whiting and K. E. Jansen. A stabilized finite element method for the incompressible Navier-Stokes equations using a hierarchical basis. Int. J. Numer. Meth. Fluids, 35:93--116, 2001.

[19]

C. H. Whiting, K. E. Jansen, and S. Dey. Hierarchical basis in stabilized finite element methods for compressible flows. Comp. Meth. Appl. Mech. Engng., 192:5167--5185, 2003.

[20]

M. Zhou. Petascale Adaptive Computational Fluid Dynamics. PhD thesis, Rensselaer Polytechnic Institute, Aug. 2009.

Cited By

Araya GLagares C(2022)Implicit Subgrid-Scale Modeling of a Mach 2.5 Spatially Developing Turbulent Boundary LayerEntropy10.3390/e2404055524:4(555)Online publication date: 15-Apr-2022
https://doi.org/10.3390/e24040555
Chen YGélébart LMarano AMarrow J(2021)FFT phase-field model combined with cohesive composite voxels for fracture of composite materials with interfacesComputational Mechanics10.1007/s00466-021-02041-1Online publication date: 21-Jun-2021
https://doi.org/10.1007/s00466-021-02041-1
Yang FChandra AZhang YTendulkar SNastasia ROberai AShephard MSahni O(2021)A parallel interface tracking approach for evolving geometry problemsEngineering with Computers10.1007/s00366-021-01386-838:5(4289-4305)Online publication date: 23-Aug-2021
https://doi.org/10.1007/s00366-021-01386-8
Show More Cited By

Index Terms

Scalable implicit finite element solver for massively parallel processing with demonstration to 160K cores

Recommendations

Massively parallel finite element programming
EuroMPI'10: Proceedings of the 17th European MPI users' group meeting conference on Recent advances in the message passing interface

Today's large finite element simulations require parallel algorithms to scale on clusters with thousands or tens of thousands of processor cores. We present data structures and algorithms to take advantage of the power of high performance computers in ...
Massively parallel computation of linear recurrence equations with graphics processing units
SAMOS '18: Proceedings of the 18th International Conference on Embedded Computer Systems: Architectures, Modeling, and Simulation

Graphics processing units (GPUs) show very high performance when executing many parallel programs; however their use in solving linear recurrence equations is considered difficult because of the sequential nature of the problem. Previously developed ...
Algorithms and data structures for massively parallel generic adaptive finite element codes

Today's largest supercomputers have 100,000s of processor cores and offer the potential to solve partial differential equations discretized by billions of unknowns. However, the complexity of scaling to such large machines and problem sizes has so far ...

Comments

Please enable JavaScript to view thecomments powered by Disqus.

Information & Contributors

Information

Published In

cover image ACM Conferences

SC '09: Proceedings of the Conference on High Performance Computing Networking, Storage and Analysis

November 2009

778 pages

ISBN:9781605587448

DOI:10.1145/1654059

Conference Chair:
Wilfred Pinfold

Copyright © 2009 ACM.

Permission to make digital or hard copies of all or part of this work for personal or classroom use is granted without fee provided that copies are not made or distributed for profit or commercial advantage and that copies bear this notice and the full citation on the first page. Copyrights for components of this work owned by others than ACM must be honored. Abstracting with credit is permitted. To copy otherwise, or republish, to post on servers or to redistribute to lists, requires prior specific permission and/or a fee. Request permissions from [email protected]

Sponsors

SIGARCH: ACM Special Interest Group on Computer Architecture
IEEE-CS: Computer Society

Publisher

Association for Computing Machinery

New York, NY, United States

Publication History

Published: 14 November 2009

Permissions

Request permissions for this article.

Request Permissions

Check for updates

Qualifiers

Research-article

Funding Sources

Conference

SC '09

Sponsor:

SIGARCH
IEEE-CS

SC '09: International Conference for High Performance Computing, Networking, Storage and Analysis

November 14 - 20, 2009

Oregon, Portland

Acceptance Rates

SC '09 Paper Acceptance Rate 59 of 261 submissions, 23%;

Overall Acceptance Rate 1,516 of 6,373 submissions, 24%

Contributors

Other Metrics

View Article Metrics

Bibliometrics & Citations

Bibliometrics

Article Metrics

45
Total Citations
View Citations
43
Total Downloads

Downloads (Last 12 months)9
Downloads (Last 6 weeks)1

Reflects downloads up to 16 Nov 2024

Other Metrics

View Author Metrics

Citations

Cited By

Araya GLagares C(2022)Implicit Subgrid-Scale Modeling of a Mach 2.5 Spatially Developing Turbulent Boundary LayerEntropy10.3390/e2404055524:4(555)Online publication date: 15-Apr-2022
https://doi.org/10.3390/e24040555
Chen YGélébart LMarano AMarrow J(2021)FFT phase-field model combined with cohesive composite voxels for fracture of composite materials with interfacesComputational Mechanics10.1007/s00466-021-02041-1Online publication date: 21-Jun-2021
https://doi.org/10.1007/s00466-021-02041-1
Yang FChandra AZhang YTendulkar SNastasia ROberai AShephard MSahni O(2021)A parallel interface tracking approach for evolving geometry problemsEngineering with Computers10.1007/s00366-021-01386-838:5(4289-4305)Online publication date: 23-Aug-2021
https://doi.org/10.1007/s00366-021-01386-8
Balin RWright JPatterson JFarnsworth JEvans JLakhani RSpalart PJansen K(2020)Hybrid Turbulence Model Computations of the NASA Juncture Flow Model Using PHASTAAIAA Scitech 2020 Forum10.2514/6.2020-1777Online publication date: 5-Jan-2020
https://doi.org/10.2514/6.2020-1777
Cortes ALins EGuerra GSilva RAlves JElias RRochinha FCoutinho A(2020)EdgeCFD: a parallel residual-based variational multiscale code for multiphysicsInternational Journal of Computational Fluid Dynamics10.1080/10618562.2020.177132534:7-8(529-548)Online publication date: 27-May-2020
https://doi.org/10.1080/10618562.2020.1771325
Smith CAbeysinghe EMarru SJansen KSanielevici S(2018)PHASTA Science Gateway for High Performance Computational Fluid DynamicsProceedings of the Practice and Experience on Advanced Research Computing: Seamless Creativity10.1145/3219104.3229243(1-4)Online publication date: 22-Jul-2018
https://dl.acm.org/doi/10.1145/3219104.3229243
Smith CGranzow BDiamond GIbanez DSahni OJansen KShephard M(2018)In‐memory integration of existing software components for parallel adaptive unstructured mesh workflowsConcurrency and Computation: Practice and Experience10.1002/cpe.451030:18Online publication date: 27-Apr-2018
https://doi.org/10.1002/cpe.4510
McGlynn ETran SSahni O(2017)Large Eddy Simulation of Flow Interactions of Segmented Synthetic Jets on an Airfoil47th AIAA Fluid Dynamics Conference10.2514/6.2017-3310Online publication date: 2-Jun-2017
https://doi.org/10.2514/6.2017-3310
Kocher ACummings RTran SSahni O(2017)Large Eddy Simulation of Surging Airfoils with Moderate to Large Streamwise Oscillations55th AIAA Aerospace Sciences Meeting10.2514/6.2017-1006Online publication date: 5-Jan-2017
https://doi.org/10.2514/6.2017-1006
Tran SMcGlynn ESahni O(2017)Large Eddy Simulation of Flow Interactions of a Finite-span Synthetic Jet on an Airfoil55th AIAA Aerospace Sciences Meeting10.2514/6.2017-1005Online publication date: 5-Jan-2017
https://doi.org/10.2514/6.2017-1005
Show More Cited By

View Options

Login options

Check if you have access through your login credentials or your institution to get full access on this article.

Full Access

Get this Publication

View options

PDF

View or Download as a PDF file.

eReader

View online with eReader.

Media

Figures

Other

Tables

View Table of Contents