Implementation and testing of a KNS topology in an InfiniBand cluster
Authors: 
Gabriel Gomez-Lopez, Jesus Escudero-Sahuquillo, Pedro J. Garcia, Francisco J. QuilesAuthors Info & Claims
Pages 21306 - 21338
Abstract
The InfiniBand (IB) interconnection technology is widely used in the networks of modern supercomputers and data centers. Among other advantages, the IB-based network devices allow for building multiple network topologies, and the IB control software (subnet manager) supports several routing engines suitable for the most common topologies. However, the implementation of some novel topologies in IB-based networks may be difficult if suitable routing algorithms are not supported, or if the IB switch or NIC architectures are not directly applicable for that topology. This work describes the implementation of the network topology known as KNS in a real HPC cluster using an IB network. As far as we know, this is the first implementation of this topology in an IB-based system. In more detail, we have implemented the KNS routing algorithm in the OpenSM software distribution of the subnet manager, and we have adapted the available IB-based switches to the particular structure of this topology. We have evaluated the correctness of our implementation through experiments in the real cluster, using well-known benchmarks. The obtained results, which match the expected performance for the KNS topology, show that this topology can be implemented in IB-based clusters as an alternative to other interconnection patterns.
References
[1]
Chen X and Lin X Big data deep learning: challenges and perspectives IEEE Access 2014 2 514-525
[2]
Top500.org: Top 500 list. https://www.top500.org. Accessed 4 June 2024 (2024)
[3]
Rocher-Gonzalez J, Escudero-Sahuquillo J, García PJ, Quiles FJ (2017) On the Impact of Routing Algorithms in the Effectiveness of Queuing Schemes in High-Performance Interconnection Networks. In: 25th IEEE Annual Symposium on High-Performance Interconnects, HOTI 2017, Santa Clara, CA, USA, August 28-30, 2017, pp. 65–72. IEEE Computer Society, USA.
[4]
Rocher-Gonzalez J, Escudero-Sahuquillo J, García PJ, Quiles FJ, and Mora G towards an efficient combination of adaptive routing and queuing schemes in fat-tree topologies J Parallel Distrib Comput 2021 147 46-63
[5]
Rocher-González J, Gran EG, Reinemo S, Skeie T, Escudero-Sahuquillo J, García PJ, Flor FJQ (2022) Adaptive routing in InfiniBand Hardware. In: 22nd IEEE International Symposium on Cluster, Cloud and Internet Computing, CCGrid 2022, Taormina, Italy, May 16–19, 463–472. IEEE, USA (2022).
[6]
Duato J, Yalamanchili S, and Ni L Interconnection networks 2003 San Francisco Elsevier Science
[7]
Dally WJ and Towles BP Principles and practices of interconnection networks 2004 San Francisco Elsevier
[8]
Leiserson CE Fat-trees: universal networks for hardware-efficient supercomputing IEEE Trans Comput 1985 34 10 892-901
[9]
Singh A, Ong J, Agarwal A, Anderson G, Armistead A, Bannon R, Boving S, Desai G, Felderman B, Germano P, Kanagala A, Provost J, Simmons J, Tanda E, Wanderer J, Hölzle U, Stuart S, Vahdat A (2015) Jupiter rising: a decade of clos topologies and centralized control in Google’s datacenter network. In: Proceedings of the 2015 ACM Conference on Special Interest Group on Data Communication. SIGCOMM ’15, pp. 183–197. Association for Computing Machinery, New York, NY, USA .
[10]
Ajima Y, Inoue T, Hiramoto S, Takagi Y, and Shimizu T The Tofu Interconnect IEEE Micro 2012 32 1 21-31
[11]
Rodriguez G, Minkenberg C, Beivide R, Luijten RP, Labarta J, Valero M (2009) Oblivious routing schemes in extended generalized fat tree networks. In: 2009 IEEE International Conference on Cluster Computing and Workshops, pp 1–8 .
[12]
Desai N, Balaji P, Sadayappan P, Islam M (2008) Are nonblocking networks really needed for high-end-computing workloads? In: 2008 IEEE International Conference on Cluster Computing, pp 152–159 .
[13]
Kim J, Dally WJ, Scott S, Abts D (2008) Technology-driven, highly-scalable dragonfly topology. In: Proceedings of the 35th Annual International Symposium on Computer Architecture. ISCA ’08, pp 77–88. IEEE Computer Society, USA .
[14]
Flajslik M, Borch E, Parker MA (2018) MegaFly: a topology for Exascale systems. In: High performance computing: 33rd international conference, ISC High Performance 2018, Frankfurt, Germany, June 24–28, 2018, Proceedings 33, pp 289–310. Springer, Cham .
[15]
Peñaranda R, Gómez C, Gómez ME, López P, Duato J (2012) A new family of hybrid topologies for large-scale interconnection networks. In: 2012 IEEE 11th International Symposium on Network Computing and Applications, pp 220–227 .
[16]
Yebenes Segura P, Escudero-Sahuquillo J, Gomez C, Garcia PJ, Quiles FJ, Duato J (2013) BBQ: a straightforward queuing scheme to reduce hol-blocking in high-performance hybrid networks. In: Euro-Par 2013 Parallel Processing: 19th International Conference, Aachen, Germany, August 26-30, 2013. Proceedings 19, pp 699–712. Springer, Berlin.
[17]
Peñaranda R, Gómez C, Gómez ME, López P, and Duato J The k-ary n-direct s-indirect family of topologies for large-scale interconnection networks J Supercomput 2016 72 3 1035-1062
[18]
Shanley T InfiniBand network architecture 2003 Boston Addison-Wesley
[19]
Mellanox Technologies: Mellanox OFED for Linux User Manual. Mellanox OFED for Linux User Manual, Rev 2.0-3.0.0 ed., Sunnyvale, CA, USA (2013)
[20]
Zahavi E, Johnson G, Kerbyson DJ, and Lang M Optimized InfiniBand fat-tree routing for shift all-to-all communication patterns Concurr Comput Pract Exp 2010 22 2 217-231
[21]
Sullivan H, Bashkow TR (1977) A large scale, homogenous, fully distributed Parallel machine, I. In: Proceedings of the 4th Annual Symposium on Computer Architecture. ISCA ’77, pp 105–117. Association for Computing Machinery, New York, NY, USA .
[22]
Dally WJ and Seitz CL Deadlock-free message routing in multiprocessor interconnection networks IEEE Trans Comput 1987 C–36 5 547-553
[23]
Hoefler T, Schneider T, Lumsdaine A (2009) Optimized routing for large-scale InfiniBand networks. In: 2009 17th IEEE Symposium on High Performance Interconnects, pp 103–111 .
[24]
Domke J, Hoefler T, Nagel WE (2011) Deadlock-free oblivious routing for arbitrary topologies. In: 2011 IEEE International Parallel & Distributed Processing Symposium, pp 616–627 .
[25]
Luszczek PR, Bailey DH, Dongarra JJ, Kepner J, Lucas RF, Rabenseifner R, Takahashi D (2006) The HPC challenge (HPCC) benchmark suite. In: Proceedings of the 2006 ACM/IEEE conference on supercomputing, 213, 1
[26]
Dongarra J, Luszczek P (2011) In: Padua, D. (ed.) LINPACK benchmark, pp 1033–1036. Springer, Boston.
[27]
Dongarra J, Heroux MA, and Luszczek P A new metric for ranking high-performance computing systems Natl Sci Rev 2016 3 1 30-35
[28]
Murphy RC, Wheeler KB, Barrett BW, and Ang JA Introducing the graph 500 Cray Users Group (CUG) 2010 19 45-74
[29]
Hoefler T, Mehlan T, Lumsdaine A, Rehm W (2007) Netgauge: a network performance measurement framework. In: High Performance Computing and Communications: Third International Conference, HPCC 2007, Houston, USA, September 26–28, 2007. Proceedings 3, pp 659–671. Springer, Berlin.
[30]
Sancho JC, Robles A, Duato J (2001) Effective strategy to compute forwarding tables for InfiniBand networks. In: International Conference on Parallel Processing, 2001, pp 48–57 .
[31]
Maglione-Mathey G, Escudero-Sahuquillo J, Garcia PJ, Quiles FJ, and Duato J Path2SL: leveraging Infiniband resources to reduce head-of-line blocking in fat trees IEEE Micro 2020 40 1 8-14
[32]
Bhandarkar SM and Arabnia HR The Hough transform on a reconfigurable multi-ring network J Parallel Distrib Comput 1995 24 1 107-114
[33]
Das R, Eachempati S, Mishra AK, Narayanan V, Das CR (2009) Design and evaluation of a hierarchical on-chip interconnect for next-generation CMPs. In: 2009 IEEE 15th International Symposium on High Performance Computer Architecture, pp 175–186 .
[34]
Yang Y, Funahashi A, Jouraku A, Nishi H, Amano H, and Sueyoshi T Recursive diagonal torus: an interconnection network for massively parallel computers IEEE Trans Parallel Distrib Syst 2001 12 7 701-715
[35]
Guo C, Lu G, Li D, Wu H, Zhang X, Shi Y, Tian C, Zhang Y, Lu S (2009) BCube: a high performance, server-centric network architecture for modular data centers. In: Proceedings of the ACM SIGCOMM 2009 Conference on Applications, Technologies, Architectures, and Protocols for Computer Communications, Barcelona, Spain, August 16–21, 2009. SIGCOMM ’09, pp 63–74. Association for Computing Machinery, New York, NY, USA .
Index Terms
- Implementation and testing of a KNS topology in an InfiniBand cluster
Index terms have been assigned to the content through auto-classification.
Recommendations
SlimUpdate: Minimal Routing Update for Performance-Based Reconfigurations in Fat-Trees
CLUSTER '15: Proceedings of the 2015 IEEE International Conference on Cluster ComputingAs the size of high-performance computing systems grows, the number of events requiring a network reconfiguration, as well as the complexity of each reconfiguration, is likely to increase. In large systems, the probability of component failure is high. ...
A Weighted Fat-Tree Routing Algorithm for Efficient Load-Balancing in Infini Band Enterprise Clusters
PDP '15: Proceedings of the 2015 23rd Euromicro International Conference on Parallel, Distributed, and Network-Based ProcessingInfini Band (IB) has become a popular network interconnect for high performance computing (HPC) systems. Many of the large IB-based HPC systems use some variant of the fat-tree topology to take advantage of the useful properties fat-trees offer. The fat-...
Infiniband-Based Multi-path Mesh/Torus Interconnection Network for Massively Parallel Systems
FCST '09: Proceedings of the 2009 Fourth International Conference on Frontier of Computer Science and TechnologyInterconnection network plays an important role in massively parallel systems. As a high performance switched network standard, Infiniband is widely used in MPP systems. Compared to fat-tree topology which is commonly used in Infiniband networks, mesh/...
Comments
Please enable JavaScript to view thecomments powered by Disqus.Information & Contributors
Information
Published In
© The Author(s) 2024.
Publisher
Kluwer Academic Publishers
United States
Publication History
Published: 07 June 2024
Accepted: 09 May 2024
Author Tags
Qualifiers
- Research-article
Funding Sources
- BBVA foundation and Becas Leonardo (call 2020)
Contributors
Other Metrics
Bibliometrics & Citations
Bibliometrics
Article Metrics
- 0Total Citations
- 0Total Downloads
- Downloads (Last 12 months)0
- Downloads (Last 6 weeks)0
Reflects downloads up to 17 Feb 2025