Nothing Special   »   [go: up one dir, main page]

skip to main content
10.1109/SC41406.2024.00037acmconferencesArticle/Chapter ViewAbstractPublication PagesscConference Proceedingsconference-collections
research-article
Free access

An Evaluation of the Effect of Network Cost Optimization for Leadership Class Supercomputers

Published: 17 November 2024 Publication History

Abstract

Dragonfly-based networks are an extensively deployed network topology in large-scale high-performance computing due to their cost-effectiveness and efficiency. The US will soon have three Exascale supercomputers for leadership class workloads deployed using dragonfly networks. Compared to indirect networks of similar scale, the dragonfly network has considerably reduced cable lengths, cable counts, and switch counts, resulting in significant network cost savings for a given system size, however, these cost reductions result in reduced global minimal paths and more challenging routing. Additionally, large scale dragonfly networks often require a taper at the global link level, resulting in less bisection bandwidth than is achievable in other traditional non-blocking topologies of equivalent scale. While dragonfly networks have been extensively studied, they have yet to be fully evaluated in an extreme scale (i.e., exascale) system that targets capability workloads. In this paper, we present the results of the first large scale evaluation of a dragonfly network on an exascale system (Frontier) and compare its behavior to a similar scale fat-tree network on a previous generation TOP500 system (Summit). This evaluation aims to determine the effect of network cost optimizations by measuring a tapered topology's impact on capability workloads. Our evaluation is based on a collection of synthetic microbenchmarks, mini-apps, and full scale applications. It compares the scaling efficiencies of each benchmark between the dragonfly-based Frontier and the fat-tree-based Summit systems. Our results show that a dragonfly network is ~30% more cost efficient than a fat-tree topology, which amortizes to ~3% of an exascale system cost. Furthermore, while tapered dragonfly networks impose significant tradeoffs, the impacts are not as broad as initially thought and are mostly seen in applications with global communication patterns, particularly all-to-all (e.g., FFT-based algorithms), but also local communication patterns (e.g., nearest-neighbor algorithms) that are sensitive to network performance variability.

References

[1]
TOP500 Lists, 2024. http://www.top500.org/lists/.
[2]
S. S. Vazhkudai, B. R. de Supinski, A. S. Bland, A. Geist, J. Sexton, J. Kahle, C. J. Zimmer, S. Atchley, S. Oral, D. E. Maxwell, V. G. V. Larrea, A. Bertsch, R. Goldstone, W. Joubert, C. Chambreau, D. Appelhans, R. Blackmore, B. Casses, G. Chochia, G. Davison, M. A. Ezell, T. Gooding, E. Gonsiorowski, L. Grinberg, B. Hanson, B. Hartner, I. Karlin, M. L. Leininger, D. Leverman, C. Marroquin, A. Moody, M. Ohmacht, R. Pankajakshan, F. Pizzano, J. H. Rogers, B. Rosenburg, D. Schmidt, M. Shankar, F. Wang, P. Watson, B. Walkup, L. D. Weems, and J. Yin, "The design, deployment, and evaluation of the CORAL pre-exascale systems," in SC18: International Conference for High Performance Computing, Networking, Storage and Analysis, pp. 661--672, 2018.
[3]
A. Khan, H. Sim, S. S. Vazhkudai, A. R. Butt, and Y. Kim, "An analysis of system balance and architectural trends based on TOP500 supercomputers," in The International Conference on High Performance Computing in Asia-Pacific Region, HPC Asia 2021, (New York, NY, USA), p. 11--22, Association for Computing Machinery, 2021.
[4]
D. Chen, P. Heidelberger, C. Stunkel, Y. Sugawara, C. Minkenberg, B. Prisacari, and G. Rodriguez, "An evaluation of network architectures for next generation supercomputers," in 2016 7th International Workshop on Performance Modeling, Benchmarking and Simulation of High Performance Computer Systems (PMBS), pp. 11--21, 2016.
[5]
J. Li, G. Michelogiannakis, B. Cook, D. Cooray, and Y. Chen, "Analyzing resource utilization in an hpc system: A case study of nersc's perlmutter," in International Conference on High Performance Computing, pp. 297--316, Springer, 2023.
[6]
S. Atchley, C. Zimmer, J. Lange, D. Bernholdt, V. Melesse Vergara, T. Beck, M. Brim, R. Budiardja, S. Chandrasekaran, M. Eisenbach, T. Evans, M. Ezell, N. Frontiere, A. Georgiadou, J. Glenski, P. Grete, S. Hamilton, J. Holmen, A. Huebl, D. Jacobson, W. Joubert, K. Mcmahon, E. Merzari, S. Moore, A. Myers, S. Nichols, S. Oral, T. Papatheodore, D. Perez, D. M. Rogers, E. Schneider, J.-L. Vay, and P. K. Yeung, "Frontier: Exploring Exascale," in Proceedings of the International Conference for High Performance Computing, Networking, Storage and Analysis, SC '23, (New York, NY, USA), Association for Computing Machinery, 2023.
[7]
D. Roweth, G. Faanes, J. Treger, and M. Terpstra, "Hpe slingshot launched into network space,"
[8]
M. Flajslik, E. Borch, and M. A. Parker, "Megafly: A topology for exascale systems," in High Performance Computing: 33rd International Conference, ISC High Performance 2018, Frankfurt, Germany, June 24-28, 2018, Proceedings 33, pp. 289--310, Springer, 2018.
[9]
K. Lakhotia, M. Besta, L. Monroe, K. Isham, P. Iff, T. Hoefler, and F. Petrini, "PolarFly: a cost-effective and flexible low-diameter topology," in SC22: International Conference for High Performance Computing, Networking, Storage and Analysis, pp. 1--15, IEEE, 2022.
[10]
M. Besta and T. Hoefler, "Slim fly: A cost effective low-diameter network topology," in SC'14: proceedings of the international conference for high performance computing, networking, storage and analysis, pp. 348--359, IEEE, 2014.
[11]
N. Liu, A. Haider, D. Jin, and X.-H. Sun, "Modeling and simulation of extreme-scale fat-tree networks for HPC systems and data centers," ACM Transactions on Modeling and Computer Simulation (TOMACS), vol. 27, no. 2, pp. 1--23, 2017.
[12]
M. A. Mollah, P. Faizian, M. S. Rahman, X. Yuan, S. Pakin, and M. Lang, "A comparative study of topology design approaches for HPC interconnects," in 2018 18th IEEE/ACM International Symposium on Cluster, Cloud and Grid Computing (CCGRID), pp. 392--401, 2018.
[13]
K. Sano, "Interconnection network: Design space exploration of network for supercomputers," in Sustained Simulation Performance 2014: Proceedings of the joint Workshop on Sustained Simulation Performance, University of Stuttgart (HLRS) and Tohoku University, 2014, pp. 151--161, Springer, 2015.
[14]
J. Domke, S. Matsuoka, I. R. Ivanov, Y. Tsushima, T. Yuki, A. Nomura, S. Miura, N. McDonald, D. L. Floyd, and N. Dubé, "Hyperx topology: First at-scale implementation and comparison to the fat-tree," in Proceedings of the International Conference for High Performance Computing, Networking, Storage and Analysis, pp. 1--23, 2019.
[15]
I. Drago, M. Mellia, M. M Munafo, A. Sperotto, R. Sadre, and A. Pras, "Inside dropbox: understanding personal cloud storage services," in Proceedings of the 2012 ACM Conference on Internet Measurement Conference, ACM, 2012.
[16]
C. Zimmer, S. Atchley, R. Pankajakshan, B. E. Smith, I. Karlin, M. L. Leininger, A. Bertsch, B. S. Ryujin, J. Burmark, A. Walker-Loud, et al., "An Evaluation of the CORAL Interconnects," in Proceedings of the international conference for high performance computing, networking, storage and analysis, pp. 1--18, 2019.
[17]
M. Al-Fares, A. Loukissas, and A. Vahdat, "A scalable, commodity data center network architecture," ACM SIGCOMM computer communication review, vol. 38, no. 4, pp. 63--74, 2008.
[18]
P.-J. Lu, M.-C. Lai, and J.-S. Chang, "A survey of high-performance interconnection networks in high-performance computer systems," Electronics, vol. 11, no. 9, p. 1369, 2022.
[19]
J. Kim, W. J. Dally, S. Scott, and D. Abts, "Technology-driven, highly-scalable dragonfly topology," ACM SIGARCH Computer Architecture News, vol. 36, no. 3, pp. 77--88, 2008.
[20]
B. Arimilli, R. Arimilli, V. Chung, S. Clark, W. Denzel, B. Drerup, T. Hoefler, J. Joyner, J. Lewis, J. Li, et al., "The PERCS highperformance interconnect," in 2010 18th IEEE Symposium on High Performance Interconnects, pp. 75--82, IEEE, 2010.
[21]
R. Curtsinger and D. Bunde, "Shortest paths in dragonfly systems," in 2019 International Workshop of High-Perfomance Interconnection Networks in the Exascale and Big-Data Era (HiPNEB), pp. 1--8, 2019.
[22]
D. De Sensi, S. Di Girolamo, and T. Hoefler, "Mitigating network noise on dragonfly networks through application-aware routing," in Proceedings of the International Conference for High Performance Computing, Networking, Storage and Analysis, pp. 1--32, 2019.
[23]
A. Shpiner, Z. Haramaty, S. Eliad, V. Zdornov, B. Gafni, and E. Zahavi, "Dragonfly+: Low cost topology for scaling datacenters," in 2017 IEEE 3rd International Workshop on High-Performance Interconnection Networks in the Exascale and Big-Data Era (HiPINEB), pp. 1--8, IEEE, 2017.
[24]
A. Bhatele, J. J. Thiagarajan, T. Groves, R. Anirudh, S. A. Smith, B. Cook, and D. K. Lowenthal, "The case of performance variability on dragonfly-based systems," in 2020 IEEE International Parallel and Distributed Processing Symposium (IPDPS), pp. 896--905, 2020.
[25]
A. Moody, "Contention-free routing for shift-based communication in MPI applications on large-scale InfiniBand clusters," tech. rep., Lawrence Livermore National Lab.(LLNL), Livermore, CA (United States), 2009.
[26]
Scalable Computer Architectures group at Sandia National Laboratories, NM, "Ember communication pattern library," 2018. https://github.com/sstsimulator/ember/.
[27]
"OLCF-6 Benchmarks - Oak Ridge Leadership Computing Facility." https://www.olcf.ornl.gov/benchmarks/, 2024. (Accessed on 04/01/2024).
[28]
"LANL/PsDNS: Psuedo-spectral direct numerical simulation in Python/MPI." https://github.com/lanl/PsDNS/tree/master. (Accessed on 03/02/2024).
[29]
A. P. Thompson, H. M. Aktulga, R. Berger, D. S. Bolintineanu, W. M. Brown, P. S. Crozier, P. J. in 't Veld, A. Kohlmeyer, S. G. Moore, T. D. Nguyen, R. Shan, M. J. Stevens, J. Tranchida, C. Trott, and S. J. Plimpton, "Lammps - a flexible simulation tool for particle-based materials modeling at the atomic, meso, and continuum scales," Computer Physics Communications, vol. 271, p. 108171, 2022.
[30]
LAMMPS Homepage. https://www.lammps.org/.
[31]
H. J. C. Berendsen, J. R. Grigera, and T. P. Straatsma, "The missing term in effective pair potentials," J. Phys. Chem., vol. 91, pp. 6269--6271, 1987.
[32]
P. Grete, J. C. Dolence, J. M. Miller, J. Brown, B. Ryan, A. Gaspar, F. Glines, S. Swaminarayan, J. Lippuner, C. J. Solomon, G. Shipman, C. Junghans, D. Holladay, J. M. Stone, and L. F. Roberts, "Parthenon---a performance portable block-structured adaptive mesh refinement framework," The International Journal of High Performance Computing Applications, vol. 37, no. 5, pp. 465--486, 2023.
[33]
A. Dubey, K. Weide, J. O'Neal, A. Dhruv, S. Couch, J. A. Harris, T. Klosterman, R. Jain, J. Rudi, B. Messer, M. Pajkos, J. Carlson, R. Chu, M. Wahib, S. Chawdhary, P. M. Ricker, D. Lee, K. Antypas, K. M. Riley, C. Daley, M. Ganapathy, F. X. Timmes, D. M. Townsley, M. Vanella, J. Bachan, P. M. Rich, S. Kumar, E. Endeve, W. R. Hix, A. Mezzacappa, and T. Papatheodore, "Flash-X: A multiphysics simulation software instrument," SoftwareX, vol. 19, p. 101168, July 2022.
[34]
S. M. Couch, C. Graziani, and N. Flocke, "An Improved Multipole Approximation for Self-gravity and Its Importance for Core-collapse Supernova Simulations" vol. 778, p. 181, Dec. 2013.
[35]
"Benchmark implementation of cosmoflow in tensorflow keras." https://github.com/sparticlesteve/cosmoflow-benchmark. (Accessed on 08/19/2024).
[36]
M. Wilkins, Y. Guo, R. Thakur, P. Dinda, and N. Hardavellas, "Acclaim: Advancing the practicality of mpi collective communication autotuning using machine learning," in 2022 IEEE International Conference on Cluster Computing (CLUSTER), pp. 161--171, IEEE, 2022.

Index Terms

  1. An Evaluation of the Effect of Network Cost Optimization for Leadership Class Supercomputers
      Index terms have been assigned to the content through auto-classification.

      Recommendations

      Comments

      Please enable JavaScript to view thecomments powered by Disqus.

      Information & Contributors

      Information

      Published In

      cover image ACM Conferences
      SC '24: Proceedings of the International Conference for High Performance Computing, Networking, Storage, and Analysis
      November 2024
      1758 pages
      ISBN:9798350352917

      Sponsors

      Publisher

      IEEE Press

      Publication History

      Published: 17 November 2024

      Check for updates

      Badges

      Author Tags

      1. Dragonfly & Fat-tree network topologies
      2. HPC systems
      3. network cost optimization

      Qualifiers

      • Research-article
      • Research
      • Refereed limited

      Conference

      SC '24
      Sponsor:

      Acceptance Rates

      Overall Acceptance Rate 1,516 of 6,373 submissions, 24%

      Upcoming Conference

      Contributors

      Other Metrics

      Bibliometrics & Citations

      Bibliometrics

      Article Metrics

      • 0
        Total Citations
      • 129
        Total Downloads
      • Downloads (Last 12 months)129
      • Downloads (Last 6 weeks)129
      Reflects downloads up to 10 Dec 2024

      Other Metrics

      Citations

      View Options

      View options

      PDF

      View or Download as a PDF file.

      PDF

      eReader

      View online with eReader.

      eReader

      Login options

      Media

      Figures

      Other

      Tables

      Share

      Share

      Share this Publication link

      Share on social media