research-article

Bundlefly: a low-diameter topology for multicore fiber

Authors:

José DuatoAuthors Info & Claims

ICS '20: Proceedings of the 34th ACM International Conference on Supercomputing

Article No.: 20, Pages 1 - 11

https://doi.org/10.1145/3392717.3392747

Published: 29 June 2020 Publication History

Abstract

High-performance computing (HPC) systems keep increasing in size and bandwidth, thus requiring larger and higher-bandwidth interconnection networks. The race to exascale just exacerbated this trend. The resulting longer average distance and more links between modules makes the use of optical fiber mandatory. However, the system meets the challenge of cable packaging complexity, cable tolerance, and cable maintainability. Splitter cable, like multi-core fiber (MCF), is a new and cost-effective approach that has the potential to replace a bundle of fibers between any pairs of modules with a single cable, thus lowering the packaging complexity and enhancing the maintainability. To the best of our knowledge, we are the first to formally study the problem of building a cost-effective HPC network topology using multicore fiber. In this paper, a new diameter-3 topology is proposed, namely Bundlefly. It achieves a flexible tradeoff between intra-module radixes and inter-module radixes of routers with merely moderate radix to build a diameter-3 exascale interconnection network. It is suitable for the use of multi-core fiber for the requirement of inter-module bandwidth and cable packaging complexity. We analyze the properties of Bundlefly and present effective routing algorithms. We simulate and analyze the performance of Bundlefly against state-of-the-art topologies. The results show that Bundlefly with flexible configurations can achieve better performance than most existing topologies.

References

[1]

David E. Womble, Mallikarjun Shankar, and et al. Early experiences on summit: Data analytics and AI applications. IBM Journal of Research and Development, 63(6):2:1--2:9, 2019.

[2]

Xiang-Ke Liao, Zheng-Bin Pang, and et al. High performance interconnect network for tianhe system. Journal of Computer Science and Technology, 30(2):259--272, 2015.

[3]

Fuad Doany. Power-efficient, high-bandwidth optical interconnects for high performance computing. http://www.hoti.org/hoti20/program/, 2012.

[4]

R. Rajamony, L. B. Arimilli, and K. Gildea. Percs: The ibm power7-ih high-performance computing system. IBM Journal of Research and Development, 55(3):3:1--3:12, 2011.

Digital Library

[5]

Yuichiro Ajima, Tomohiro Inoue, and et al. The tofu interconnect 2. In Proceedings of Symposium on High-Performance Interconnects (HOTI), pages 57--62, 2014.

[6]

COMMSCOPE. Fabric networks:designing your network for the futureąłfrom 10g through 400g and beyond. https://www.commscope.com/Docs/Fabric_Networks_Design_Guide_TP-1101171-EN, 2016.

[7]

Mellanox. Maximize hpc performance with infiniband aocs from the infiniband experts. http://www.mellanox.com/products/interconnect/infiniband-active-optical-cables-splitters.php, 2018.

[8]

OFS. Multicore optical fiber. http://www.ofsoptics.com/multicore.html, 2018.

[9]

FIBERCORE. Multicore. https://www.fibercore.com/product-category/multicore, 2018.

[10]

Furukawa. Multicore optical fiber technology. http://www.furukawa.co.jp/english/museum/floor3/05/a05.htm, 2012.

[11]

AFL. Multicore fiber fanouts. https://www.aflglobal.com/Products/Fusion-Splicing/Specialty-Fiber-Optic-Compon\ents-and-Services/Multicore-Fiber-Fanouts.aspx, 2017.

[12]

Matthew Peach. 8-strand multi-core fiber will boost optical interconnects. http://optics.org/news/6/3/47, 2015.

[13]

Xueqin Xie, Jiajing tu, and et al. Design and optimization of 32-core rod/trench assisted square-lattice structured single-mode multi-core fiber. 25(5):5119--5132, 2017.

[14]

Kota Shikama, Yoshiteru Abe, and et al. Low-loss fiber-bundle-type fan-in/fan-out device for 6-mode 19-core fiber. In Proceedings of Optical Fiber Communication Conference (OFC), page M3G.3, 2017.

[15]

Masato Yoshida, Toshihiko Hirooka, and Masataka Nakazawa. Low-loss and reflection-free fused type fan-out device for 7-core fiber based on a bundled structure. Optics Express, 25(16):18817--18826, 2017.

[16]

Yoshiteru Abe, Kota Shikama, and Shuichiro Asakawa. Multi-core fiber connector technology for low-loss physical-contact connection. https://www.ntt-review.jp/archive/ntttechnical.php?con\tents=ntr201706fa6.html, 2017.

[17]

Maciej Besta and Torsten Hoefler. Slim fly: A cost effective low-diameter network topology. In Proceedings of International Conference for High Performance Computing, Networking, Storage, and Analysis (SC), pages 348--359, 2014.

[18]

S. Birrittella Mark, Debbage Mark, and et al. Intel omni-path architecture: Enabling scalable, high performance fabrics. In Proceedings of Symposium on High-Performance Interconnects, pages 402--414, 2015.

[19]

Mellanox. Mellanox quantum^™. http://www.mellanox.com/page/products_dyn?product_family=2\64&mtag=quantum, 2017.

[20]

Fei Lei, Dezun Dong, and et al. Galaxyfly: A novel family of flexible-radix low-diameter topologies for large-scales interconnection networks. In Proceedings of International Conference on Supercomputing (ICS), pages 24:1--24:12, 2016.

[21]

John Kim, William J. Dally, and et al. Technology-driven, highly-scalable dragonfly topology. In Proceedings of International Symposium on Computer Architecture (ISCA), pages 77--88, 2008.

[22]

Alexander Shpiner, Zachy Haramaty, and et al. Dragonfly+: Low cost topology for scaling datacenters. In Proceedings of International Workshop on High-Performance Interconnection Networks in the Exascale and Big-Data Era (HiPINEB), pages 1--8, 2017.

[23]

Abhinav Bhatele, Nikhil Jain, and et al. Analyzing network health and congestion in dragonfly-based supercomputers. In Proceedings of International Parallel and Distributed Processing Symposium (IPDPS), pages 93--102, 2016.

[24]

Paul R. Hafner. Geometric realisation of the graphs of mckay miller siran. Journal of Combinatorial Theory, 90(2):223--232, 2004.

Digital Library

[25]

Brendan D McKay, Mirka Miller, and Jozef Siran. A note on large graphs of diameter two and given maximum degree. Journal of Combinatorial Theory, 74(1):110 -- 118, 1998.

Digital Library

[26]

Mirka Miller and Jozef Siran. Moore graphs and beyond: A survey of the degree/diameter problem. Electronic Journal of Combinatorics, 14:2005, 2005.

[27]

Ryosuke Mizuno and Yawara Ishida. Constructing large-scale low-latency network from small optimal networks. In Proceedings of International Symposium on Networks-on-Chip (NOCS), pages 1--5, 2016.

[28]

John Kim, William J. Dally, and Dennis Abts. Flattened butterfly: A cost-efficient topology for high-radix networks. In Proceedings of International Symposium on Computer Architecture (ISCA), pages 126--137, 2007.

Digital Library

[29]

J.C. Bermond, C. Delorme, and G. Farhi. Large graphs with given degree and diameter iii. North-Holland Mathematics Studies, 62:23 -- 31, 1982.

[30]

Greg Faanes, Abdulla Bataineh, and et al. Cray cascade: A scalable hpc system based on a dragonfly network. In Proceedings of International Conference for High Performance Computing, Networking, Storage, and Analysis (SC), pages 103:1--103:9, 2012.

Digital Library

[31]

George Karypis and Vipin Kumar. A fast and high quality multilevel scheme for partitioning irregular graphs. Siam Journal on Scientific Computing, 20(1):359--392, 1998.

Digital Library

[32]

Jiang Nan, U. Becker Daniel, and et al. A detailed and flexible cycle-accurate network-on-chip simulator. In Proceedings of International Symposium on Performance Analysis of Systems and Software (ISPASS), pages 86 -- 96, 2013.

[33]

Marina Garcia, Enrique Vallejo, and et al. On-the-fly adaptive routing in high-radix hierarchical networks. In Proceedings of International Conference on Parallel Processing (ICPP), pages 279--288, 2012.

[34]

Emily Hastings, David Rincon-Cruz, and et al. Comparing global link arrangements for dragonfly networks. In Proceedings of International Conference on Cluster Computing (Cluster), pages 361--370, 2015.

[35]

Cristóbal Camarero, Enrique Vallejo, and Ramón Beivide. Topological characterization of hamming and dragonfly networks and its implications on routing. ACM Transactions on Architecture and Code Optimization, 11(4):39:1--39:25, 2014.

Digital Library

[36]

Paul Erdos, Siemion Fajtlowicz, and Alan J. Hoffman. Maximum degree in graphs of diameter 2. 10:87 -- 90, 1980.

[37]

FUJITSU. K computer. http://www.fujitsu.com/global/about/busine\sspolicy/tech/k/, 2011.

Cited By

Blach NBesta MDe Sensi DDomke JHarake HLi SIff PKonieczny MLakhotia KKubicek AFerrari MPetrini FHoefler TVanbever LZhang I(2024)A high-performance design, implementation, deployment, and evaluation of the slim fly networkProceedings of the 21st USENIX Symposium on Networked Systems Design and Implementation10.5555/3691825.3691882(1025-1044)Online publication date: 16-Apr-2024
https://dl.acm.org/doi/10.5555/3691825.3691882
Lakhotia KMonroe LIsham KBesta MBlach NHoefler TPetrini FAgrawal KPetrank E(2024)PolarStar: Expanding the Horizon of Diameter-3 NetworksProceedings of the 36th ACM Symposium on Parallelism in Algorithms and Architectures10.1145/3626183.3659975(345-357)Online publication date: 17-Jun-2024
https://dl.acm.org/doi/10.1145/3626183.3659975
Wang SZhang XWang CWu KLi CDong D(2024)DRLAR: A deep reinforcement learning-based adaptive routing framework for network-on-chipsComputer Networks10.1016/j.comnet.2024.110419(110419)Online publication date: Apr-2024
https://doi.org/10.1016/j.comnet.2024.110419
Show More Cited By

Index Terms

Bundlefly: a low-diameter topology for multicore fiber
1. Computer systems organization
  1. Architectures
    1. Parallel architectures
      1. Interconnection architectures

Recommendations

Practical traffic grooming scheme for single-hub SONET/WDM rings
LCN '00: Proceedings of the 25th Annual IEEE Conference on Local Computer Networks

In SONET/WDM networks, one fiber supports multiple wavelengths and each wavelength supports several low rate tributary streams. Traffic grooming is then defined as properly using the SONET add/drop multiplexer (ADM) to electronically multiplex and ...
10 Gbps CPRI signals transmission impaired by intercore crosstalk in 5G network fronthauls with multicore fibers

The impact of intercore crosstalk (ICXT) of weakly-coupled multicore fibers (MCFs) on the transmission performance of a Common Public Radio Interface (CPRI) signal in 5G networks fronthaul is studied by numerical simulation. The results show that ...
Fault-tolerance schemes for WDM-based multiprocessor networks
MPPOI '95: Proceedings of the Second Workshop on Massively Parallel Processing Using Optical Interconnections

Due to the difference in implementing network topologies in the optical and electronic domain, schemes to tolerate link failures in electronic implementations may not be the most efficient ones for optical implementations of multiprocessor networks. ...

Comments

Please enable JavaScript to view thecomments powered by Disqus.

Information & Contributors

Information

Published In

cover image ACM Conferences

ICS '20: Proceedings of the 34th ACM International Conference on Supercomputing

June 2020

499 pages

ISBN:9781450379830

DOI:10.1145/3392717

General Chairs:
Eduard Ayguadé
Universitat Politècnica de Catalunya and Barcelona Supercomputing Center
,
Wen-mei Hwu
University of Illinois at Urbana-Champaign
,
Program Chairs:
Rosa M. Badia
Barcelona Supercomputing Center and Universitat Politècnica de Catalunya
,
H. Peter Hofstee
IBM Austin

Copyright © 2020 ACM.

Permission to make digital or hard copies of all or part of this work for personal or classroom use is granted without fee provided that copies are not made or distributed for profit or commercial advantage and that copies bear this notice and the full citation on the first page. Copyrights for components of this work owned by others than ACM must be honored. Abstracting with credit is permitted. To copy otherwise, or republish, to post on servers or to redistribute to lists, requires prior specific permission and/or a fee. Request permissions from [email protected]

Sponsors

SIGARCH: ACM Special Interest Group on Computer Architecture

Publisher

Association for Computing Machinery

New York, NY, United States

Publication History

Published: 29 June 2020

Permissions

Request permissions for this article.

Request Permissions

Check for updates

Author Tags

Qualifiers

Research-article

Conference

ICS '20

Sponsor:

SIGARCH

ICS '20: 2020 International Conference on Supercomputing

June 29 - July 2, 2020

Spain, Barcelona

Acceptance Rates

Overall Acceptance Rate 629 of 2,180 submissions, 29%

Contributors

Other Metrics

View Article Metrics

Bibliometrics & Citations

Bibliometrics

Article Metrics

11
Total Citations
View Citations
340
Total Downloads

Downloads (Last 12 months)37
Downloads (Last 6 weeks)2

Reflects downloads up to 17 Feb 2025

Other Metrics

View Author Metrics

Citations

Cited By

Blach NBesta MDe Sensi DDomke JHarake HLi SIff PKonieczny MLakhotia KKubicek AFerrari MPetrini FHoefler TVanbever LZhang I(2024)A high-performance design, implementation, deployment, and evaluation of the slim fly networkProceedings of the 21st USENIX Symposium on Networked Systems Design and Implementation10.5555/3691825.3691882(1025-1044)Online publication date: 16-Apr-2024
https://dl.acm.org/doi/10.5555/3691825.3691882
Lakhotia KMonroe LIsham KBesta MBlach NHoefler TPetrini FAgrawal KPetrank E(2024)PolarStar: Expanding the Horizon of Diameter-3 NetworksProceedings of the 36th ACM Symposium on Parallelism in Algorithms and Architectures10.1145/3626183.3659975(345-357)Online publication date: 17-Jun-2024
https://dl.acm.org/doi/10.1145/3626183.3659975
Wang SZhang XWang CWu KLi CDong D(2024)DRLAR: A deep reinforcement learning-based adaptive routing framework for network-on-chipsComputer Networks10.1016/j.comnet.2024.110419(110419)Online publication date: Apr-2024
https://doi.org/10.1016/j.comnet.2024.110419
Kumari AMishra J(2024)Performance Analysis of Hybrid BPSK-MPPM Modulated Multicore Fiber Interconnect SystemCognitive Computing and Cyber Physical Systems10.1007/978-3-031-48891-7_15(185-192)Online publication date: 5-Jan-2024
https://doi.org/10.1007/978-3-031-48891-7_15
Lakhotia KIsham KMonroe LBesta MHoefler TPetrini FAgrawal KShun J(2023)In-network Allreduce with Multiple Spanning Trees on PolarFlyProceedings of the 35th ACM Symposium on Parallelism in Algorithms and Architectures10.1145/3558481.3591073(165-176)Online publication date: 17-Jun-2023
https://dl.acm.org/doi/10.1145/3558481.3591073
Lan BLei FDong DWu KZhang X(2023)DFAR: Dynamic-threshold Fault-tolerant Adaptive Routing for Fat Tree Networks2023 IEEE 29th International Conference on Parallel and Distributed Systems (ICPADS)10.1109/ICPADS60453.2023.00110(721-728)Online publication date: 17-Dec-2023
https://doi.org/10.1109/ICPADS60453.2023.00110
Lakhotia KBesta MMonroe LIsham KIff PHoefler TPetrini F(2022)PolarFly: A Cost-Effective and Flexible Low-Diameter TopologySC22: International Conference for High Performance Computing, Networking, Storage and Analysis10.1109/SC41404.2022.00017(1-15)Online publication date: Nov-2022
https://doi.org/10.1109/SC41404.2022.00017
Young SAksoy SFiroz JGioiosa RHagge TKempton MEscobedo JRaugas M(2022)SpectralFly: Ramanujan Graphs as Flexible and Efficient Interconnection Networks2022 IEEE International Parallel and Distributed Processing Symposium (IPDPS)10.1109/IPDPS53621.2022.00105(1040-1050)Online publication date: May-2022
https://doi.org/10.1109/IPDPS53621.2022.00105
Wang YDong DLei F(2022)Understanding Node Connection Modes in Multi-Rail Fat-treeJournal of Parallel and Distributed Computing10.1016/j.jpdc.2022.04.019Online publication date: May-2022
https://doi.org/10.1016/j.jpdc.2022.04.019
Li CDong DYang SLiao XSun GLiu Y(2021)CIB-HIERACM Transactions on Architecture and Code Optimization10.1145/346806218:4(1-21)Online publication date: 17-Jul-2021
https://dl.acm.org/doi/10.1145/3468062
Show More Cited By

View Options

Login options

Check if you have access through your login credentials or your institution to get full access on this article.

Full Access

Get this Publication

View options

PDF

View or Download as a PDF file.

eReader

View online with eReader.

Figures

Tables

Media

View Table of Conten