A Portable and Efficient Lagrangian Particle Capability for Idealized Atmospheric Phenomena
Article No.: 26, Pages 1 - 11
Abstract
The Cloud Model version 1 is an atmospheric model that allows for idealized studies of atmospheric phenomena. A new Lagrangian microphysics capability has been added, enabling a significantly more accurate representation than the traditional bulk or multi-moment approaches frequently found in mesoscale atmospheric models. We have utilized a directive-based approach to enable a single source code to efficiently support execution on both CPU and GPU-based computing platforms. In addition to the use of accelerator directives, changes to the data structures and the message-passing approach used by the Lagrangian particle-based microphysics module were necessary to enable efficient execution for a large number of particles. We focus on a configuration that will be used to investigate the impact of oceanic sea spray on the atmospheric boundary layer within a hurricane. We observe a factor of 5.1× reduction in time to the solution when comparing the execution time for 256 NVIDIA A100 GPUs versus 256 AMD EPYC™ Milan-based compute nodes using 1 billion particles.
References
[1]
Bareford, M.: Monitoring the Cray XC30 power management hardware counters. Proceedings of CUG2015, Seattle, WA, USA (2015)
[2]
Bieringer, P.E., Piña, A.J., Lorenzetti, D.M., Jonker, H.J.J., Sohn, M.D., Annunzio, A.J., Fry, R.N.: A graphics processing unit (GPU) approach to large eddy simulation (LES) for transport and contaminant dispersion. Atmosphere 12(7) (2021).
[3]
Bryan, G.H., Worsnop, R.P., Lundquist, J.K., Zhang, J.A.: A simple method for simulating wind profiles in the boundary layer of tropical cyclones. Boundary-Layer Meteorology 162(3), 475--502 (March 2017).
[4]
Bryan, G.H., Wyngaard, J.C., Fritsch, J.M.: Resolution requirements for the simulation of deep moist convection. Monthly Weather Review 131(10), 2394--2416 (October 2003). <2394:RRFTSO>2.0.CO;2
[5]
Chandrakar, K.K., Morrison, H., Grabowski, W.W., Bryan, G.H., Shaw, R.A.: Supersaturation variability from scalar mixing: Evaluation of a new subgridscale model using direct numerical simulations of turbulent Rayleigh-Bénard convection. Journal of the Atmospheric Sciences 79(4), 1191--1210 (April 2022).
[6]
Chen, X., Bryan, G.H., Zhang, J.A., Cione, J.J., Marks, F.D.: A framework for simulating the tropical-cyclone boundary layer using large-eddy simulation and its use in evaluating PBL parameterizations. Journal of the Atmospheric Sciences (September 2021).
[7]
Cray Performance measurement and Analysis Tools. https://hpctools.readthedocs.io/en/latest/perftools.html (2022)
[8]
cuTENSOR: A high-performance CUDA library for tensor primitives. https://docs.nvidia.com/cuda/cutensor/latest/index.html (2023)
[9]
Deardorff, J.W.: Stratocumulus-capped mixed layers derived from a three-dimensional model. Boundary-Layer Meteorology 18(4), 495--527 (June 1980).
[10]
Dennis, J.M., Baker, A.H., Dobbins, B., Bell, M.M., Sun, J., Kim, Y., Cha, T.Y.: Enabling efficient execution of a variational data assimilation application. The International Journal of High Performance Computing Applications (2022).
[11]
Dennis, J.M., Sun, J., Voelz, S., Bryan, G., Richter, D.: A portable and efficient lagrangian particle capability for idealized atmospheric phenomena. Dataset on Zenodo, (2024)
[12]
Computer and Information Systems Laboratory, Derecho: HPE Cray EX system (university community computing). (2023)
[13]
Gettelman, A., Morrison, H., Eidhammer, T., Thayer-Calder, K., Sun, J., Forbes, R., McGraw, Z., Zhu, J., Storelvmo, T., Dennis, J.: Importance of ice nucleation and precipitation on climate with the parameterization of unified microphysics across scales version 1 (pumasv1). EGUsphere 2022, 1--28 (2022).
[14]
Grabowski, W.W., Morrison, H., Shima, S.I., Abade, G.C., Dziekan, P., Pawlowska, H.: Modeling of cloud microphysics: Can we do better? Bulletin of the American Meteorological Society 100 100(4), 655--672 (2019).
[15]
van Heerwaarden, C.C., van Stratum, B.J.H., Heus, T., Gibbs, J.A., Fedorovich, E., Mellado, J.P.: MicroHH 1.0: a computational fluid dynamics code for direct numerical simulation and large-eddy simulation of atmospheric boundary layer flows. Geoscientific Model Development 10(8), 3145--3165 (2017).
[16]
Maronga, B., Gryschka, M., Heinze, R., Hoffmann, F., Kanani-Sühring, F., Keck, M., Ketelsen, K., Letzel, M.O., Sühring, M., Raasch, S.: The parallelized large-eddy simulation model (PALM) version 4.0 for atmospheric and oceanic flows: model formulation, recent developments, and future perspectives. Geoscientific Model Development 8(8), 2515--2551 (2015).
[17]
Michalakes, J., Vachharajani, M.: GPU acceleration of numerical weather prediction. Parallel Processing Letters 18(04), 531--548 (2008).
[18]
Nsight Compute Developer tools documentation. https://docs.nvidia.com/nsight-compute/ (2023)
[19]
NCAR-Wyoming Supercomputing Center. https://en.wikipedia.org/wiki/NCAR-Wyoming_Supercomputing_Center (June 2022)
[20]
Ponder, C.: Folded maxloc algorithm. Personal Conversation (2022)
[21]
Ponder, C.: Cutensor based indirect address gather operation. Personal Conversation (2023)
[22]
Rotunno, R., Bryan, G.H.: Numerical simulations of two-layer flow past topography. part II: Lee vortices. Journal of the Atmospheric Sciences 77(3), 965--980 (March 2020).
[23]
Sauer, J.A., Muñoz-Esparza, D.: The FastEddy® resident-GPU accelerated large-eddy simulation framework: Model formulation, dynamical-core validation and performance benchmarks. Journal of Advances in Modeling Earth Systems 12(11), e2020MS002100 (2020).
[24]
Schalkwijk, J., Griffith, E.J., Post, F.H., Jonker, H.J.J.: High-performance simulations of turbulent clouds on a desktop PC: Exploiting the GPU. Bulletin of the American Meteorological Society 93(3), 307 -- 314 (2012).
[25]
Shima, S., Kusano, K., Kawano, A., Sugiyama, T., Kawahara, S.: The super-droplet method for the numerical simulation of clouds and precipitation: A particle-based and probabilistic microphysics model coupled with a non-hydrostatic model. Quarterly Journal of the Royal Meteorological Society 135, 1307--1320 (2009).
[26]
Skamarock, W.C., Klemp, J.B.: The stability of time-split numerical methods for the hydrostatic and the nonhydrostatic elastic equations. Monthly Weather Review 120(9), 2109--2127 (1992)
[27]
Stern, D.P., Bryan, G.H., Lee, C., Doyle, J.D.: Estimating the risk of extreme wind gusts in tropical cyclones using idealized large-eddy simulations and a statistical-dynamical model. Monthly Weather Review 149(12), 4183--4204 (2021).
[28]
Sun, J., Dennis, J.M., Mickelson, S.A., Vanderwende, B., Gettelman, A., Thayer-Calder, K.: Acceleration of the parameterization of unified microphysics across scales (PUMAS) on the graphics processing unit (GPU) with directive-based methods. JAMES 15 (2023).
[29]
Thakur, R., Rabenseifner, R., Group, W.: Optimization of collective communication operations in MPICH. The International Journal of High Performance Computing Applications 19 (September 2016).
[30]
Wicker, L.J., Skamarock, W.C.: Time-splitting methods for elastic models using forward time schemes. Monthly Weather Review 130(8), 2088--2097 (August 2002). <2088:TSMFEM>2.0.CO;2.
Index Terms
- A Portable and Efficient Lagrangian Particle Capability for Idealized Atmospheric Phenomena
Recommendations
Accelerating Radiative Transfer Simulation on NVIDIA GPUs with OpenACC
Parallel and Distributed Computing, Applications and TechnologiesAbstractTo accelerate multiphysics applications, making use of not only GPUs but also FPGAs has been emerging. Multiphysics applications are simulations involving multiple physical models and multiple simultaneous physical phenomena. Operations with ...
Evaluation of a Directive-Based GPU Programming Approach for High-Order Unstructured Mesh Computational Fluid Dynamics
PASC '17: Proceedings of the Platform for Advanced Scientific Computing ConferenceIn this work we evaluate the effectiveness of using OpenACC as a paradigm for the auto-parallelization of a high-order unstructured CFD code on Graphics Processing Units (GPUs). This is in lieu of hand-written CUDA or OpenCL code for the algorithms that ...
On the Programmability and Performance of Heterogeneous Platforms
ICPADS '13: Proceedings of the 2013 International Conference on Parallel and Distributed SystemsGeneral-purpose computing on an ever-broadening array of parallel devices has led to an increasingly complex and multi-dimensional landscape with respect to programmability and performance optimization. The growing diversity of parallel architectures ...
Comments
Please enable JavaScript to view thecomments powered by Disqus.Information & Contributors
Information
Published In
June 2024
296 pages
ISBN:9798400706394
DOI:10.1145/3659914
- Proceedings Chairs:
- Timothy Robinson,
- Michèle Woodtli,
- Program Chairs:
- Axel Huebl,
- Cristina Silvano
Copyright © 2024 Copyright held by the owner/author(s).
Permission to make digital or hard copies of all or part of this work for personal or classroom use is granted without fee provided that copies are not made or distributed for profit or commercial advantage and that copies bear this notice and the full citation on the first page. Copyrights for third-party components of this work must be honored. For all other uses, contact the owner/author(s).
Sponsors
Publisher
Association for Computing Machinery
New York, NY, United States
Publication History
Published: 03 June 2024
Check for updates
Author Tags
Qualifiers
- Research-article
Funding Sources
Conference
PASC '24
Sponsor:
Acceptance Rates
PASC '24 Paper Acceptance Rate 26 of 36 submissions, 72%;
Overall Acceptance Rate 109 of 221 submissions, 49%
Contributors
Other Metrics
Bibliometrics & Citations
Bibliometrics
Article Metrics
- 0Total Citations
- 36Total Downloads
- Downloads (Last 12 months)36
- Downloads (Last 6 weeks)7
Reflects downloads up to 25 Nov 2024
Other Metrics
Citations
View Options
Login options
Check if you have access through your login credentials or your institution to get full access on this article.
Sign in