short-paper

Open access

Towards a Framework for One-sided RDMA Multicast

Authors:

Mun Choon ChanAuthors Info & Claims

ANCS '21: Proceedings of the Symposium on Architectures for Networking and Communications Systems

Pages 129 - 132

https://doi.org/10.1145/3493425.3502766

Published: 18 January 2022 Publication History

Abstract

We present the design and prototyping of a framework to support multicast for remote direct memory accesses (RDMA), specifically the one-sided WRITE operation. We use P4 programmable hardware to augment fixed-function RDMA transport hardware found on commodity NICs to enable one-sided RDMA multicast with zero-CPU overhead. Finally, we outline the potential challenges and future directions in realizing the framework for large-scale data center deployments.

Supplementary Material

MP4 File (s2-khooi.mp4)

Presentation video

Download
122.37 MB

References

[1]

Infiniband Trade Association. 2014. Annex A17: RoCEv2. https://cw.infinibandta.org/document/dl/7781 [Accessed: Oct 2021].

[2]

Tarick Bedeir. 2010. Rdma read and write with ib verbs. Technical Report. Technical report, HPC Advisory Council, 2010. URL: https://www.hpcadvisorycouncil.com/pdf/rdma-read-and-write-with-ib-verbs.pdf.

[3]

Rutger Beltman, Silke Knossen, Joseph Hill, and Paola Grosso. 2020. Using P4 and RDMA to collect telemetry data. In 2020 IEEE/ACM Innovating the Network for Data-Intensive Science (INDIS). 1--9.

[4]

Pat Bosshart, Dan Daly, Glen Gibb, Martin Izzard, Nick McKeown, Jennifer Rexford, Cole Schlesinger, Dan Talayco, Amin Vahdat, George Varghese, and David Walker. 2014. P4: Programming Protocol-Independent Packet Processors. SIGCOMM Comput. Commun. Rev. 44, 3 (July 2014), 87--95.

Digital Library

[5]

Jiaxin Cao, Chuanxiong Guo, Guohan Lu, Yongqiang Xiong, Yixin Zheng, Yongguang Zhang, Yibo Zhu, Chen Chen, and Ye Tian. 2013. Datacast: A Scalable and Efficient Reliable Group Data Delivery Service for Data Centers. IEEE Journal on Selected Areas in Communications 31, 12 (2013), 2632--2645.

[6]

Mellanox Community. 2019. How to disable ICRC validation with RoCE v2? https://community.mellanox.com/s/question/0D51T00007A2M1fSAF/how-to-disable-icrc-validation-with-roce-v2 [Accessed: Oct 2021].

[7]

Dmitry Duplyakin, Robert Ricci, Aleksander Maricq, Gary Wong, Jonathon Duerig, Eric Eide, Leigh Stoller, Mike Hibler, David Johnson, Kirk Webb, Aditya Akella, Kuangching Wang, Glenn Ricart, Larry Landweber, Chip Elliott, Michael Zink, Emmanuel Cecchet, Snigdhaswin Kar, and Prabodh Mishra. 2019. The Design and Operation of CloudLab. In 2019 USENIX Annual Technical Conference (USENIX ATC 19). 1--14.

[8]

Linux Foundation. [n. d.]. Data Plane Development Kit (DPDK). http://www.dpdk.org [Accessed: Oct 2021].

[9]

Yixiao Gao, Qiang Li, Lingbo Tang, Yongqing Xi, Pengcheng Zhang, Wenwen Peng, Bo Li, Yaohui Wu, Shaozong Liu, Lei Yan, Fei Feng, Yan Zhuang, Fan Liu, Pan Liu, Xingkui Liu, Zhongjie Wu, Junping Wu, Zheng Cao, Chen Tian, Jinbo Wu, Jiaji Zhu, Haiyong Wang, Dennis Cai, and Jiesheng Wu. 2021. When Cloud Storage Meets RDMA. In 18th USENIX Symposium on Networked Systems Design and Implementation (NSDI 21). 519--533.

[10]

Chuanxiong Guo, Haitao Wu, Zhong Deng, Gaurav Soni, Jianxi Ye, Jitu Padhye, and Marina Lipshteyn. 2016. RDMA over Commodity Ethernet at Scale. In Proceedings of the 2016 ACM SIGCOMM Conference. 202--215.

Digital Library

[11]

Toke Føiland-Jørgensen, Jesper Dangaard Brouer, Daniel Borkmann, John Fastabend, Tom Herbert, David Ahern, and David Miller. 2018. The express data path: Fast programmable packet processing in the operating system kernel. In Proceedings of the 14th international conference on emerging networking experiments and technologies. 54--66.

Digital Library

[12]

Intel. [n.d.]. Intel® Tofino™ Programmable Ethernet Switch ASIC. https://www.intel.com/content/www/xa/en/products/network-io/programmable-ethernet-switch/tofino-series.html [Accessed: Oct 2021]

[13]

Theo Jepsen, Ali Fattaholmanan, Masoud Moshref, Nate Foster, Antonio Carzaniga, and Robert Soulé. 2020. Forwarding and Routing with Packet Subscriptions. In Proceedings of the 16th International Conference on Emerging Networking EXperiments and Technologies. 282--294.

Digital Library

[14]

Anuj Kalia, Michael Kaminsky, and David G. Andersen. 2016. Design Guidelines for High Performance RDMA Systems. In 2016 USENIX Annual Technical Conference (USENIX ATC 16). 437--450.

Digital Library

[15]

Xin Zhe Khooi, Levente Csikor, Jialin Li, Min Suk Kang, and Dinil Mon Divakaran. 2021. Revisiting Heavy-Hitter Detection on Commodity Programmable Switches. In 2021 IEEE 7th International Conference on Network Softwarization (NetSoft). 79--87.

[16]

Patricia Kummrow. 2021. The IPU: A new, strategic resource for Cloud Service Providers. https://itpeernetwork.intel.com/ipu-cloud/ [Accessed: Oct 2021].

[17]

Jialin Li, Ellis Michael, Naveen Kr. Sharma, Adriana Szekeres, and Dan R. K. Ports. 2016. Just Say NO to Paxos Overhead: Replacing Consensus with Network Ordering. In 12th USENIX Symposium on Operating Systems Design and Implementation (OSDI 16). 467--483.

[18]

Youyou Lu, Jiwu Shu, Youmin Chen, and Tao Li. 2017. Octopus: an RDMA-enabled Distributed Persistent Memory File System. In 2017 USENIX Annual Technical Conference (USENIX ATC 17). Santa Clara, CA, 773--785.

Digital Library

[19]

Mike McBride and Olufemi Komolafe. 2020. Multicast in the Data Center Overview. Internet-Draft draft-ietf-mboned-dc-deploy-09. Internet Engineering Task Force. https://datatracker.ietf.org/doc/html/draft-ietf-mboned-dc-deploy-09 Work in Progress.

[20]

NVIDIA. [n.d.]. DOCA SDK. https://developer.nvidia.com/networking/doca [Accessed: Oct 2021].

[21]

NVIDIA. [n.d.]. Mellanox BlueField2 DPU SmartNICs. https://store.mellanox.com/categories/dpu.html [Accessed: Oct 2021].

[22]

Benjamin Rothenberger, Konstantin Taranov, Adrian Perrig, and Torsten Hoefler. 2021. ReDMArk: Bypassing RDMA Security Mechanisms. In 30th USENIX Security Symposium (USENIX Security 21). 4277--4292.

[23]

Arjun Roy, Hongyi Zeng, Jasmeet Bagga, George Porter, and Alex C. Snoeren. 2015. Inside the Social Network's (Datacenter) Network. SIGCOMM Comput. Commun. Rev. 45, 4 (Aug. 2015), 123--137.

Digital Library

[24]

Amedeo Sapio, Marco Canini, Chen-Yu Ho, Jacob Nelson, Panos Kalnis, Changhoon Kim, Arvind Krishnamurthy, Masoud Moshref, Dan Ports, and Peter Richtarik. 2021. Scaling Distributed Machine Learning with In-Network Aggregation. In 18th USENIX Symposium on Networked Systems Design and Implementation (NSDI 21). 785--808.

[25]

Muhammad Shahbaz, Lalith Suresh, Jennifer Rexford, Nick Feamster, Ori Rottenstreich, and Mukesh Hira. 2019. Elmo: Source Routed Multicast for Public Clouds. In Proceedings of the ACM Special Interest Group on Data Communication. 458--471.

Digital Library

[26]

Konstantin Shvachko, Hairong Kuang, Sanjay Radia, and Robert Chansler. 2010. The hadoop distributed file system. In 2010 IEEE 26th Symposium on Mass Storage Systems and Technologies (MSST). 1--10.

Digital Library

[27]

Xiaoye Steven Sun, Yiting Xia, Simbarashe Dzinamarira, Xin Sunny Huang, Dingming Wu, and TS Eugene Ng. 2018. Republic: Data multicast meets hybrid rack-level interconnections in data center. In 2018 IEEE 26th International Conference on Network Protocols (ICNP). IEEE, 77--87.

[28]

Xingda Wei, Rong Chen, and Haibo Chen. 2020. Fast RDMA-based Ordered Key-Value Store using Remote Learned Cache. In 14th USENIX Symposium on Operating Systems Design and Implementation (OSDI 20). 117--135.

[29]

Yiting Xia, T. S. Eugene Ng, and Xiaoye Steven Sun. 2015. Blast: Accelerating high-performance data analytics applications by optical multicast. In 2015 IEEE Conference on Computer Communications (INFOCOM). 1930--1938.

[30]

Xilinx. [n.d.]. Alveo SN1000 SmartNICs. https://www.xilinx.com/applications/data-center/network-acceleration/alveo-sn1000.html [Accessed: Oct 2021].

Cited By

Li WZhang JLiu YZeng GWang ZZeng CZhou PWang QChen K(2024)Cepheus: Accelerating Datacenter Applications with High-Performance RoCE-Capable Multicast2024 IEEE International Symposium on High-Performance Computer Architecture (HPCA)10.1109/HPCA57654.2024.00074(908-921)Online publication date: 2-Mar-2024
https://doi.org/10.1109/HPCA57654.2024.00074
Lee HChoi KLee HSim J(2023)SDM: Sharing-Enabled Disaggregated Memory System with Cache Coherent Compute Express Link2023 32nd International Conference on Parallel Architectures and Compilation Techniques (PACT)10.1109/PACT58117.2023.00016(86-98)Online publication date: 21-Oct-2023
https://doi.org/10.1109/PACT58117.2023.00016
Huang CGao YChen WLi DXiao YZhang RTian CWang XDou WChen GWang YXiao F(2023)MC-RDMA: Improving Replication Performance of RDMA-based Distributed Systems with Reliable Multicast Support2023 IEEE 31st International Conference on Network Protocols (ICNP)10.1109/ICNP59255.2023.10355619(1-11)Online publication date: 10-Oct-2023
https://doi.org/10.1109/ICNP59255.2023.10355619

Index Terms

Towards a Framework for One-sided RDMA Multicast
1. Networks
  1. Network architectures
    1. Programming interfaces
  2. Network types
    1. Data center networks

Recommendations

A Hybrid I/O Virtualization Framework for RDMA-capable Network Interfaces
VEE '15: Proceedings of the 11th ACM SIGPLAN/SIGOPS International Conference on Virtual Execution Environments

DMA-capable interconnects, providing ultra-low latency and high bandwidth, are increasingly being used in the context of distributed storage and data processing systems. However, the deployment of such systems in virtualized data centers is currently ...
A Hybrid I/O Virtualization Framework for RDMA-capable Network Interfaces
VEE '15

DMA-capable interconnects, providing ultra-low latency and high bandwidth, are increasingly being used in the context of distributed storage and data processing systems. However, the deployment of such systems in virtualized data centers is currently ...
Software-based Live Migration for Containerized RDMA
APNet '24: Proceedings of the 8th Asia-Pacific Workshop on Networking

Container live migration is critical to ensure services are not interrupted during host maintenance in data centers. On the other hand, RDMA containerization has attracted both academia and industry for years. However, live migration for containerized ...

Comments

Please enable JavaScript to view thecomments powered by Disqus.

Information & Contributors

Information

Published In

cover image ACM Conferences

ANCS '21: Proceedings of the Symposium on Architectures for Networking and Communications Systems

December 2021

174 pages

ISBN:9781450391689

DOI:10.1145/3493425

Copyright © 2021 Owner/Author.

This work is licensed under a Creative Commons Attribution International 4.0 License.

Sponsors

Publisher

Association for Computing Machinery

New York, NY, United States

Publication History

Published: 18 January 2022

Permissions

Request permissions for this article.

Request Permissions

Check for updates

Author Tags

Qualifiers

Short-paper
Research
Refereed limited

Funding Sources

Ministry of Education ? Singapore

Conference

ANCS '21

Sponsor:

ANCS '21: Symposium on Architectures for Networking and Communications Systems

December 13 - 16, 2021

IN, Layfette, USA

Acceptance Rates

Overall Acceptance Rate 88 of 314 submissions, 28%

Contributors

Other Metrics

View Article Metrics

Bibliometrics & Citations

Bibliometrics

Article Metrics

3
Total Citations
View Citations
1,017
Total Downloads

Downloads (Last 12 months)347
Downloads (Last 6 weeks)34

Reflects downloads up to 10 Nov 2024

Other Metrics

View Author Metrics

Citations

Cited By

Li WZhang JLiu YZeng GWang ZZeng CZhou PWang QChen K(2024)Cepheus: Accelerating Datacenter Applications with High-Performance RoCE-Capable Multicast2024 IEEE International Symposium on High-Performance Computer Architecture (HPCA)10.1109/HPCA57654.2024.00074(908-921)Online publication date: 2-Mar-2024
https://doi.org/10.1109/HPCA57654.2024.00074
Lee HChoi KLee HSim J(2023)SDM: Sharing-Enabled Disaggregated Memory System with Cache Coherent Compute Express Link2023 32nd International Conference on Parallel Architectures and Compilation Techniques (PACT)10.1109/PACT58117.2023.00016(86-98)Online publication date: 21-Oct-2023
https://doi.org/10.1109/PACT58117.2023.00016
Huang CGao YChen WLi DXiao YZhang RTian CWang XDou WChen GWang YXiao F(2023)MC-RDMA: Improving Replication Performance of RDMA-based Distributed Systems with Reliable Multicast Support2023 IEEE 31st International Conference on Network Protocols (ICNP)10.1109/ICNP59255.2023.10355619(1-11)Online publication date: 10-Oct-2023
https://doi.org/10.1109/ICNP59255.2023.10355619

View Options

View options

PDF

View or Download as a PDF file.

eReader

View online with eReader.

Get Access

Login options

Check if you have access through your login credentials or your institution to get full access on this article.

Full Access

Get this Publication

Media

Figures

Other

Tables

View Table of Contents