research-article

Open access

ANT-MOC: Scalable Neutral Particle Transport Using 3D Method of Characteristics on Multi-GPU Systems

Authors:

Xuebin ChiAuthors Info & Claims

SC '23: Proceedings of the International Conference for High Performance Computing, Networking, Storage and Analysis

Article No.: 12, Pages 1 - 13

https://doi.org/10.1145/3581784.3607063

Published: 11 November 2023 Publication History

Editorial Notes

The authors have requested minor, non-substantive changes to the VoR and, in accordance with ACM policies, a Corrected VoR was published on November 14, 2023.

Abstract

The Method Of Characteristic (MOC) to solve the Neutron Transport Equation (NTE) is the core of full-core simulation for reactors. High resolution is enabled by discretizing the NTE through massive tracks to traverse the 3D reactor geometry. However, the 3D full-core simulation is prohibitively expensive because of the high memory consumption and the severe load imbalance. To deal with these challenges, we develop ANT-MOC¹. Specifically, we build a performance model for memory footprint, computation and communication, based on which a track management strategy is proposed to overcome the resolution bottlenecks caused by limited GPU memory. Furthermore, we implement a novel multi-level load mapping strategy to ensure load balancing among nodes, GPUs, and CUs. ANT-MOC enables a 3D full-core reactor simulation with 100 billion tracks on 16,000 GPUs, with 70.69% and 89.38% parallel efficiency for strong scalability and weak scalability, respectively.

Supplemental Material

MP4 File - SC23 video presentation for "ANT-MOC: Scalable Neutral Particle Transport Using 3D Method of Characteristics on Multi-GPU Systems"

SC23 video presentation for the main program paper "ANT-MOC: Scalable Neutral Particle Transport Using 3D Method of Characteristics on Multi-GPU Systems" by Shunde Li, Zongguo Wang, Lingkun Bu, Jue Wang, Zhikuang Xin, Shigang Li, Yangang Wang, Yangde Feng, Peng Shi, Yun Hu and Xuebin Chi

Download
272.37 MB

References

[1]

JR Askew. 1972. A characteristics formulation of the neutron transport equation in complicated geometries. Technical Report. United Kingdom Atomic Energy Authority.

[2]

William Boyd, Samuel Shaner, Lulu Li, Benoit Forget, and Kord Smith. 2014. The OpenMOC method of characteristics neutral particle transport code. Annals of Nuclear Energy 68 (2014), 43--52.

[3]

William Boyd, Andrew Siegel, Shuo He, Benoit Forget, and Kord Smith. 2016. Parallel performance results for the OpenMOC neutron transport code on multicore platforms. The International Journal of High Performance Computing Applications 30, 3 (2016), 360--375.

[4]

K Bradley. 2013. NEAMS: the nuclear energy advanced modeling and simulation program. Technical Report. Argonne National Lab.(ANL), Argonne, IL (United States).

[5]

Christian Chauliac, José-Maria Aragonés, Dominique Bestion, Dan Gabriel Cacuci, Nicolas Crouzet, Frank-Peter Weiss, and Martin A Zimmermann. 2011. NURESIM-A European simulation platform for nuclear reactor safety: Multi-scale and multi-physics calculations, sensitivity and uncertainty analysis. Nuclear Engineering and Design 241, 9 (2011), 3416--3426.

[6]

Jun Chen, Zhouyu Liu, Chen Zhao, Qingming He, Tiejun Zu, Liangzhi Cao, and Hongchun Wu. 2018. A new high-fidelity neutronics code NECP-X. Annals of Nuclear Energy 116 (2018), 417--428.

[7]

JY Cho. 2002. Three-dimensional heterogeneous whole core transport calculation employing planar MOC solutions. Trans. Am. Nucl. Soc. 87 (2002), 234.

[8]

Charbel Farhat. 1988. A simple and efficient automatic FEM domain decomposer. Computers & Structures 28, 5 (1988), 579--602.

[9]

Geoffrey Gunow, Benoit Forget, and Kord Smith. 2019. Full core 3D simulation of the BEAVRS benchmark with OpenMOC. Annals of Nuclear Energy 134 (2019), 299--304.

[10]

Geoffrey Gunow, Samuel Shaner, William Boyd, Benoit Forget, and Kord Smith. 2017. Accuracy and performance of 3D MOC for full-core PWR problems. Korean Nuclear Society - KNS, Korea, Republic of.

[11]

Geoggrey Gunow, Samuel Shanner, Benoit Forget, and Kord Smith. 2016. Reducing 3D MOC storage requirements with axial on-the-fly ray tracing. Physics of Reactors 2016 (PHYSOR 2016) (2016).

[12]

Geoffrey Alexander Gunow. 2018. Full core 3D neutron transport simulation using the method of characteristics with linear sources. Ph. D. Dissertation. Massachusetts Institute of Technology.

[13]

Cho Jin Young, Joo Han Gyu, Kim Ha Yong, and Chang Moon-Hee. 2003. Parallelization of a three-dimensional whole core transport code DeCART. (Jul 2003).

[14]

Yeon Sang Jung, Cheon Bo Shim, Chang Hyun Lim, and Han Gyu Joo. 2013. Practical numerical reactor employing direct whole core neutron transport and subchannel thermal/hydraulic solvers. Annals of Nuclear Energy 62 (2013), 357--374.

[15]

George Karypis, Kirk Schloegel, and Vipin Kumar. 1997. Parmetis: Parallel graph partitioning and sparse matrix ordering library. (1997).

[16]

Blake W Kelley, Benjamin Collins, and Edward W Larsen. 2013. 2D/1D approximations to the 3D neutron transport equation. II: Numerical comparisons. Technical Report. American Nuclear Society, 555 North Kensington Avenue, La Grange Park, IL ....

[17]

Kitware Inc. 1999. ParaView. https://www.paraview.org.

[18]

Brendan Kochunas, Benjamin Collins, Dan Jabaay, Thomas J Downar, and William R Martin. 2013. Overview of development and design of MPACT: Michigan parallel characteristics transport code. Technical Report. American Nuclear Society, 555 North Kensington Avenue, La Grange Park, IL ....

[19]

Brendan Matthew Kochunas. 2013. A Hybrid Parallel Algorithm for the 3-D Method of Characteristics Solution of the Boltzmann Transport Equation on High Performance Compute Clusters. Thesis. http://deepblue.lib.umich.edu/handle/2027.42/100072 Accepted: 2013-09-24T16:03:54Z.

[20]

Brendan Matthew Kochunas. 2013. A Hybrid Parallel Algorithm for the 3-D Method of Characteristics Solution of the Boltzmann Transport Equation on High Performance Compute Clusters. Ph.D. Dissertation.

[21]

Shinya Kosaka and Etsuro Saji. 2000. Transport theory calculation for a heterogeneous multi-assembly problem by characteristics method with direct neutron path linking technique. Journal of nuclear science and technology 37, 12 (2000), 1015--1023.

[22]

Elmer Eugene Lewis and Warren F Miller. 1984. Computational methods of neutron transport. (1984).

[23]

Michael J Quinn. 2003. Parallel programming. TMH CSE 526 (2003), 105.

[24]

Paul K Romano, Nicholas E Horelik, Bryan R Herman, Adam G Nelson, Benoit Forget, and Kord Smith. 2015. OpenMC: A state-of-the-art Monte Carlo code for research and development. Annals of Nuclear Energy 82 (2015), 90--97.

[25]

Didier Schneider, F Dolci, F Gabriel, J-M Palau, M Guillo, and B Pothet. 2016. APOLLO3® CEA/DEN deterministic multi-purpose code for reactor physics analysis. In PHYSOR 2016--Unifying Theory and Experiments in the 21st Century.

[26]

Daniele Sciannandrone, Simone Santandrea, and Richard Sanchez. 2016. Optimized tracking strategies for step MOC calculations in extruded 3D axial geometries. Annals of Nuclear Energy 87 (2016), 49--60.

[27]

Samuel Shaner, Geoffrey Gunow, Benoit Forget, and Kord Smith. 2015. Theoretical Analysis of Track Generation in 3d Method of Characteristics. Prof. Forget via Chris Sherratt (April 2015). https://dspace.mit.edu/handle/1721.1/108661 Accepted: 2017-05-04T14:49:17Z ISBN: 9781510808041 Publisher: American Nuclear Society.

[28]

Peitao Song, Zhijian Zhang, Liang Liang, Qian Zhang, and Qiang Zhao. 2019. Implementation and performance analysis of the massively parallel method of characteristics based on GPU. Annals of Nuclear Energy 131 (2019), 257--272.

[29]

Xiao Tang, Qing Li, Xiaoming Chai, Xiaolan Tu, Wenbin Wu, and Kan Wang. 2017. Efficient procedure for radial MOC and axial SN coupled 3D neutron transport calculation. In Proc. Int. Conf. Mathematics and Computational Methods Applied to Nuclear Science and Engineering (M&C2017). 16--20.

[30]

Top500. 2022. TOP500 Lists. http://www.top500.org/lists/

[31]

John A Turner, Kevin Clarno, Matt Sieger, Roscoe Bartlett, Benjamin Collins, Roger Pawlowski, Rodney Schmidt, and Randall Summers. 2016. The virtual environment for reactor applications (VERA): design and architecture. J. Comput. Phys. 326 (2016), 544--568.

[32]

GJ Wu and R Roy. 2003. A new characteristics algorithm for 3D transport calculations. Annals of Nuclear Energy 30, 1 (2003), 1--16.

[33]

Wenbin Wu, Yingrui Yu, Qi Luo, Dong Yao, Qing Li, and Xiaoming Chai. 2020. Verification of the 3D capability of OpenMOC with the C5G7 3D extension benchmark. Annals of Nuclear Energy 140 (June 2020), 107293.

[34]

Wenbin Wu, Yingrui Yu, Qi Luo, Dong Yao, Qing Li, and Xiaoming Chai. 2020. Verification of the 3D capability of OpenMOC with the C5G7 3D extension benchmark. Annals of Nuclear Energy 140 (June 2020), 107293.

[35]

Akio Yamamoto, Masahiro Tatsumi, Yasunori Kitamura, and Yoshihiro Yamane. 2004. Improvement of the SPH method for pin-by-pin core calculations. Journal of nuclear science and technology 41, 12 (2004), 1155--1165.

[36]

Wen YANG, Changjun HU, Tiancai LIU, An WANG, and Mingyu WU. 2019. Research progress of China virtual reactor (CVR1. 0). Atomic Energy Science and Technology 53, 10 (2019), 1821.

[37]

Ao Zhang, Ming Dai, Maosong Cheng, Jianhui Wu, and Jingen Chen. 2022. Development of a GPU-based three-dimensional neutron transport code. Annals of Nuclear Energy 174 (Sept. 2022), 109156.

[38]

Ao Zhang, Ming Dai, Maosong Cheng, Jianhui Wu, and Jingen Chen. 2022. Development of a GPU-based three-dimensional neutron transport code. Annals of Nuclear Energy 174 (2022), 109156.

[39]

Hongbo Zhang, Hongchun Wu, and Liangzhi Cao. 2011. An acceleration technique for 2D MOC based on Krylov subspace and domain decomposition methods. Annals of Nuclear Energy 38, 12 (Dec. 2011), 2742--2751.

[40]

H. Zhang, Y. Zheng, H. Wu, and L. Cao. 2013. A 2D/1D coupling neutron transport method based on the matrix MOC and NEM methods. Technical Report. American Nuclear Society - ANS; La Grange Park (United States). https://www.osti.gov/biblio/22212728

[41]

Jingchao Zheng, Zhiqiang Wang, Zeyi Xie, Xingjie Peng, Chen Zhao, and Wenbin Wu. 2023. Parallel Communication Optimization Based on Graph Partition for Hexagonal Neutron Transport Simulation Using MOC Method. Energies 16, 6 (Jan. 2023), 2823. Number: 6 Publisher: Multidisciplinary Digital Publishing Institute.

Cited By

You XXuan ZYang HLuan ZLiu YQian D(2024)GVARP: Detecting Performance Variance on Large-Scale Heterogeneous SystemsProceedings of the International Conference for High Performance Computing, Networking, Storage, and Analysis10.1109/SC41406.2024.00063(1-16)Online publication date: 17-Nov-2024
https://dl.acm.org/doi/10.1109/SC41406.2024.00063

Index Terms

ANT-MOC: Scalable Neutral Particle Transport Using 3D Method of Characteristics on Multi-GPU Systems
1. Applied computing
  1. Physical sciences and engineering
    1. Physics
2. Computing methodologies
  1. Parallel computing methodologies
    1. Parallel algorithms
      1. Massively parallel algorithms

Recommendations

GPU accelerated simulations of 3D deterministic particle transport using discrete ordinates method

Graphics Processing Unit (GPU), originally developed for real-time, high-definition 3D graphics in computer games, now provides great faculty in solving scientific applications. The basis of particle transport simulation is the time-dependent, multi-...
Accelerating MapReduce framework on multi-GPU systems

Graphics processors evolve rapidly and promise to support power-efficient, cost, differentiated price-performance, and scalable high performance computing. MapReduce is a well-known distributed programming model to ease the development of applications ...
Advances in Multi-GPU Smoothed Particle Hydrodynamics Simulations

We present a multi-GPU version of GPUSPH, a CUDA implementation of fluid-dynamics models based on the smoothed particle hydrodynamics (SPH) numerical method. The SPH is a well-known Lagrangian model for the simulation of free-surface fluid flows; it ...

Comments

Please enable JavaScript to view thecomments powered by Disqus.

Information & Contributors

Information

Published In

cover image ACM Conferences

SC '23: Proceedings of the International Conference for High Performance Computing, Networking, Storage and Analysis

November 2023

1428 pages

ISBN:9798400701092

DOI:10.1145/3581784

Chair:
Dorian Arnold,
Program Chair:
Rosa M Badia,
Program Co-chair:
Kathryn Mohror

Copyright © 2023 Owner/Author(s).

This work is licensed under a Creative Commons Attribution International 4.0 License.

Sponsors

Publisher

Association for Computing Machinery

New York, NY, United States

Publication History

Published: 11 November 2023

Check for updates

Badges

Author Tags

Qualifiers

Research-article

Funding Sources

the National Key Research and Development Program of China
the Strategic Priority Research Program of Chinese Academy of Sciences
the Fundamental Reasearch Funds for the Central Universities
GuangHe Fund B

Conference

SC '23

Sponsor:

SIGHPC

SC '23: International Conference for High Performance Computing, Networking, Storage and Analysis

November 12 - 17, 2023

CO, Denver, USA

Acceptance Rates

Overall Acceptance Rate 1,516 of 6,373 submissions, 24%

Contributors

Other Metrics

View Article Metrics

Bibliometrics & Citations

Bibliometrics

Article Metrics

1
Total Citations
View Citations
741
Total Downloads

Downloads (Last 12 months)741
Downloads (Last 6 weeks)81

Reflects downloads up to 14 Nov 2024

Other Metrics

View Author Metrics

Citations

Cited By

You XXuan ZYang HLuan ZLiu YQian D(2024)GVARP: Detecting Performance Variance on Large-Scale Heterogeneous SystemsProceedings of the International Conference for High Performance Computing, Networking, Storage, and Analysis10.1109/SC41406.2024.00063(1-16)Online publication date: 17-Nov-2024
https://dl.acm.org/doi/10.1109/SC41406.2024.00063

View Options

View options

PDF

View or Download as a PDF file.

eReader

View online with eReader.

Get Access

Login options

Check if you have access through your login credentials or your institution to get full access on this article.

Full Access

Get this Publication

Media

Figures

Other

Tables

View Table of Contents