Nothing Special   »   [go: up one dir, main page]

skip to main content
10.1145/3477132.3483555acmconferencesArticle/Chapter ViewAbstractPublication PagessospConference Proceedingsconference-collections
research-article

Xenic: SmartNIC-Accelerated Distributed Transactions

Published: 26 October 2021 Publication History

Abstract

High-performance distributed transactions require efficient remote operations on database memory and protocol metadata. The high communication cost of this workload calls for hardware acceleration. Recent research has applied RDMA to this end, leveraging the network controller to manipulate host memory without consuming CPU cycles on the target server. However, the basic read/write RDMA primitives demand trade-offs in data structure and protocol design, limiting their benefits. SmartNICs are a flexible alternative for fast distributed transactions, adding programmable compute cores and on-board memory to the network interface. Applying measured performance characteristics, we design Xenic, a SmartNIC-optimized transaction processing system. Xenic applies an asynchronous, aggregated execution model to maximize network and core efficiency. Xenic's co-designed data store achieves low-overhead remote object accesses. Additionally, Xenic uses flexible, point-to-point communication patterns between SmartNICs to minimize transaction commit latency. We compare Xenic against prior RDMA- and RPC-based transaction systems with the TPC-C, Retwis, and Smallbank benchmarks. Our results for the three benchmarks show 2.42x, 2.07x, and 2.21x throughput improvement, 59%, 42%, and 22% latency reduction, while saving 2.3, 8.1, and 10.1 threads per server.

References

[1]
Alpha Data. ADM-PCIE-9V3-High-Performance Network Accelerator, Sept. 2021. https://www.alpha-data.com/dcp/products.php?product=adm-pcie-9v3.
[2]
M. T. Arashloo, A. Lavrov, M. Ghobadi, J. Rexford, D. Walker, and D. Wentzlaff. Enabling programmable transport protocols in highspeed nics. In 17th USENIX Symposium on Networked Systems Design and Implementation (NSDI 20), pages 93--109, Santa Clara, CA, Feb. 2020. USENIX Association.
[3]
Broadcom. The TruFlow Flow processing engine. https://www.broadcom.com/applications/data-center/cloud-scale-networking, 2021.
[4]
Broadcom Inc. Stingray SmartNIC Adapters and IC, Sept. 2021. https://www.broadcom.com/products/ethernet-connectivity/network-adapters/smartnic.
[5]
P. Celis, P. Larson, and J. I. Munro. Robin hood hashing (preliminary report). In 26th Annual Symposium on Foundations of Computer Science, Portland, Oregon, USA, 21-23 October 1985, pages 281--288. IEEE Computer Society, 1985.
[6]
Y. Chen, X. Wei, J. Shi, R. Chen, and H. Chen. Fast and general distributed transactions using rdma and htm. In Proceedings of the Eleventh European Conference on Computer Systems, EuroSys '16, New York, NY, USA, 2016. Association for Computing Machinery.
[7]
D. W. Cornell, D. M. Dias, and P. S. Yu. On multisystem coupling through function request shipping. IEEE Transactions on Software Engineering, SE-12(10):1006--1017, 1986.
[8]
A. Dragojević, D. Narayanan, M. Castro, and O. Hodson. Farm: Fast remote memory. In 11th USENIX Symposium on Networked Systems Design and Implementation (NSDI 2014). USENIX - Advanced Computing Systems Association, April 2014.
[9]
A. Dragojevic, D. Narayanan, E. B. Nightingale, M. Renzelmann, A. Shamis, A. Badam, and M. Castro. No compromises: Distributed transactions with consistency, availability, and performance. In Proceedings of the 25th Symposium on Operating Systems Principles, SOSP '15, page 54--70, New York, NY, USA, 2015. Association for Computing Machinery.
[10]
Exablaze. ExaNIC V5P High Density Network Application Card, Sept. 2021. https://exablaze.com/exanic-v5p.
[11]
D. Firestone, A. Putnam, S. Mundkur, D. Chiou, A. Dabagh, M. Andrewartha, H. Angepat, V. Bhanu, A. Caulfield, E. Chung, H. K. Chandrappa, S. Chaturmohta, M. Humphrey, J. Lavier, N. Lam, F. Liu, K. Ovtcharov, J. Padhye, G. Popuri, S. Raindel, T. Sapre, M. Shaw, G. Silva, M. Sivakumar, N. Srivastava, A. Verma, Q. Zuhair, D. Bansal, D. Burger, K. Vaid, D. A. Maltz, and A. Greenberg. Azure accelerated networking: Smartnics in the public cloud. In 15th USENIX Symposium on Networked Systems Design and Implementation, 2018.
[12]
S. Grant, A. Yelam, M. Bland, and A. C. Snoeren. Smartnic performance isolation with fairnic: Programmable networking for the cloud. In Proceedings of the Annual Conference of the ACM Special Interest Group on Data Communication on the Applications, Technologies, Architectures, and Protocols for Computer Communication, 2020.
[13]
H-Store Project. SmallBank Benchmark - H-Store, Sept. 2021. https://hstore.cs.brown.edu/documentation/deployment/benchmarks/smallbank/.
[14]
S. Ibanez, M. Shahbaz, and N. McKeown. The case for a network fast path to the cpu. In Proceedings of the 18th ACM Workshop on Hot Topics in Networks, HotNets '19, page 52--59, New York, NY, USA, 2019. Association for Computing Machinery.
[15]
A. Kalia, M. Kaminsky, and D. G. Andersen. Fasst: Fast, scalable and simple distributed transactions with two-sided (RDMA) datagram rpcs. In 12th USENIX Symposium on Operating Systems Design and Implementation (OSDI 16), pages 185--201, Savannah, GA, Nov. 2016. USENIX Association.
[16]
A. Kaufmann, S. Peter, N. K. Sharma, T. Anderson, and A. Krishnamurthy. High performance packet processing with flexnic. In Proceedings of the Twenty-First International Conference on Architectural Support for Programming Languages and Operating Systems, 2016.
[17]
Y. Le, H. Chang, S. Mukherjee, L. Wang, A. Akella, M. Swift, and T. Lakshman. Uno: uniflying host and smart nic offload for flexible packet processing. In Proceedings of the 2017 Symposium on Cloud Computing, pages 506--519, 09 2017.
[18]
B. Li, Z. Ruan, W. Xiao, Y. Lu, Y. Xiong, A. Putnam, E. Chen, and L. Zhang. Kv-direct: High-performance in-memory key-value store with programmable nic. In Proceedings of the 26th Symposium on Operating Systems Principles, 2017.
[19]
B. Li, K. Tan, L. L. Luo, Y. Peng, R. Luo, N. Xu, Y. Xiong, P. Cheng, and E. Chen. Clicknp: Highly flexible and high performance network processing with reconfigurable hardware. In Proceedings of the 2016 ACM SIGCOMM Conference, 2016.
[20]
J. Lin, K. Patel, B. E. Stephens, A. Sivaraman, and A. Akella. PANIC: A high-performance programmable NIC for multi-tenant networks. In 14th USENIX Symposium on Operating Systems Design and Implementation (OSDI 20), 2020.
[21]
M. Liu, T. Cui, H. Schuh, A. Krishnamurthy, S. Peter, and K. Gupta. Offloading distributed applications onto smartnics using ipipe. In Proceedings of the ACM Special Interest Group on Data Communication, 2019.
[22]
M. Liu, L. Luo, J. Nelson, L. Ceze, A. Krishnamurthy, and K. Atreya. Incbricks: Toward in-network computation with an in-network cache. In Proceedings of the Twenty-Second International Conference on Architectural Support for Programming Languages and Operating Systems, 2017.
[23]
M. Liu, S. Peter, A. Krishnamurthy, and P. M. Phothilimthana. E3: Energy-efficient microservices on smartnic-accelerated servers. In 2019 USENIX Annual Technical Conference (USENIX ATC 19), pages 363--378, Renton, WA, July 2019. USENIX Association.
[24]
Marvell Technology Group Ltd. LiquidIO III Solutions Brief, Sept. 2021. https://www.marvell.com/content/dam/marvell/en/public-collateral/embedded-processors/marvell-liquidio-III-solutions-brief.pdf.
[25]
Marvell Technology Group Ltd. Multi-Core Processors - LiquidIO Smart NICs | Network adapter, Sept. 2021. https://www.marvell.com/products/infrastructure-processors/multi-core-processors/liquidio-smart-nics.html.
[26]
F. Matus. Pensando: Distributed services architecture. In 2020 IEEE Hot Chips 32 Symposium (HCS), pages 1--17. IEEE Computer Society, 2020.
[27]
Mellanox. Accelerated Switch and Packet Processing. http://www.mellanox.com/page/asap2?mtag=asap2, 2021.
[28]
Mellanox. BlueField SmartNIC Ethernet, Sept. 2021. https://www.mellanox.com/products/BlueField-SmartNIC-Ethernet.
[29]
Mellanox. ConnectX-5 EN Single/Dual-Port Adapter, Sept. 2021. https://www.mellanox.com/products/ethernet-adapters/connectx-5-en.
[30]
Mellanox. Mellanox Innova SmartNIC. http://www.mellanox.com/page/products_dyn?product_family=275&mtag=bluefield_smart_nic, 2021.
[31]
Mellanox. OFED Documentation Rev 7.4.1.0.0.1, Sept. 2021. https://docs.mellanox.com/display/MLNXOFEDv471001.
[32]
W. M. Mellette, R. Das, Y. Guo, R. McGuinness, A. C. Snoeren, and G. Porter. Expanding across time to deliver bandwidth efficiency and low latency. In 17th USENIX Symposium on Networked Systems Design and Implementation (NSDI 20), pages 1--18, Santa Clara, CA, Feb. 2020. USENIX Association.
[33]
Netronome. Agilio LX SmartNICs, Sept. 2021. https://www.netronome.com/products/agilio-cx/.
[34]
Pensando. Pensando DSC-100 Distributed Services Card, Sept. 2021. https://pensando.io/documents/pensando-dsc- 100-distributed-services-card/.
[35]
Pensando floor plan. https://www.servethehome.com/pensando-distributed-services-architecture-smartnic/, 2021.
[36]
P. M. Phothilimthana, M. Liu, A. Kaufmann, S. Peter, R. Bodik, and T. Anderson. Floem: A programming system for nic-accelerated network applications. In 13th USENIX Symposium on Operating Systems Design and Implementation, 2018.
[37]
S. Radhakrishnan, Y. Geng, V. Jeyakumar, A. Kabbani, G. Porter, and A. Vahdat. SENIC: Scalable NIC for end-host rate limiting. In 11th USENIX Symposium on Networked Systems Design and Implementation (NSDI 14), pages 475--488, Seattle, WA, Apr. 2014. USENIX Association.
[38]
Redis. Retwis - Example Twitter clone based on the Redis Key-Value DB, Sept. 2021. http://retwis.redis.io.
[39]
B. Stephens, A. Akella, and M. Swift. Loom: Flexible and efficient NIC packet scheduling. In 16th USENIX Symposium on Networked Systems Design and Implementation (NSDI 19), pages 33--46, Boston, MA, Feb. 2019. USENIX Association.
[40]
B. Stephens, A. Akella, and M. M. Swift. Your programmable nic should be a programmable switch. In Proceedings of the 17th ACM Workshop on Hot Topics in Networks, HotNets '18, page 36--42, New York, NY, USA, 2018. Association for Computing Machinery.
[41]
A. Szekeres, M. Whittaker, J. Li, N. K. Sharma, A. Krishnamurthy, D. R. K. Ports, and I. Zhang. Meerkat: Multicore-scalable replicated transactions following the zero-coordination principle. In Proceedings of the Fifteenth European Conference on Computer Systems, EuroSys '20, New York, NY, USA, 2020. Association for Computing Machinery.
[42]
Transaction Processing Performance Council. TPC Benchmark C Standard Specification, Revision 5.11, Sept. 2021. http://www.tpc.org/tpc_documents_current_versions/pdf/tpc-c_v5.11.0.pdf.
[43]
S. Tu, W. Zheng, E. Kohler, B. Liskov, and S. Madden. Speedy transactions in multicore in-memory databases. In Proceedings of the Twenty-Fourth ACM Symposium on Operating Systems Principles, SOSP '13, page 18--32, New York, NY, USA, 2013. Association for Computing Machinery.
[44]
X. Wei, Z. Dong, R. Chen, and H. Chen. Deconstructing rdma-enabled distributed transactions: Hybrid is better! In 13th USENIX Symposium on Operating Systems Design and Implementation (OSDI 18), pages 233--251, Carlsbad, CA, Oct. 2018. USENIX Association.
[45]
X. Wei, J. Shi, Y. Chen, R. Chen, and H. Chen. Fast in-memory transaction processing using rdma and htm. In Proceedings of the 25th Symposium on Operating Systems Principles, SOSP '15, page 87--104, New York, NY, USA, 2015. Association for Computing Machinery.
[46]
Xilinx. Alveo Adaptable Accelerator Cards for Data Center Workloads, Sept. 2021. https://www.xilinx.com/products/boards-and-kits/alveo.html.
[47]
I. Zhang, N. K. Sharma, A. Szekeres, A. Krishnamurthy, and D. R. K. Ports. Building consistent transactions with inconsistent replication. ACM Trans. Comput. Syst., 35(4), Dec. 2018.

Cited By

View all
  • (2024)DDS: DPU-Optimized Disaggregated StorageProceedings of the VLDB Endowment10.14778/3681954.368200217:11(3304-3317)Online publication date: 30-Aug-2024
  • (2024)A Survey of RDMA Distributed StorageProceedings of the 2024 5th International Conference on Computing, Networks and Internet of Things10.1145/3670105.3670199(534-539)Online publication date: 24-May-2024
  • (2024)A Memory-Disaggregated Radix TreeACM Transactions on Storage10.1145/366428920:3(1-41)Online publication date: 6-Jun-2024
  • Show More Cited By

Recommendations

Comments

Please enable JavaScript to view thecomments powered by Disqus.

Information & Contributors

Information

Published In

cover image ACM Conferences
SOSP '21: Proceedings of the ACM SIGOPS 28th Symposium on Operating Systems Principles
October 2021
899 pages
ISBN:9781450387095
DOI:10.1145/3477132
Permission to make digital or hard copies of all or part of this work for personal or classroom use is granted without fee provided that copies are not made or distributed for profit or commercial advantage and that copies bear this notice and the full citation on the first page. Copyrights for components of this work owned by others than ACM must be honored. Abstracting with credit is permitted. To copy otherwise, or republish, to post on servers or to redistribute to lists, requires prior specific permission and/or a fee. Request permissions from [email protected]

Sponsors

Publisher

Association for Computing Machinery

New York, NY, United States

Publication History

Published: 26 October 2021

Permissions

Request permissions for this article.

Check for updates

Author Tags

  1. Distributed Transactions
  2. RDMA
  3. SmartNICs

Qualifiers

  • Research-article
  • Research
  • Refereed limited

Conference

SOSP '21
Sponsor:

Acceptance Rates

Overall Acceptance Rate 131 of 716 submissions, 18%

Upcoming Conference

SOSP '24

Contributors

Other Metrics

Bibliometrics & Citations

Bibliometrics

Article Metrics

  • Downloads (Last 12 months)305
  • Downloads (Last 6 weeks)19
Reflects downloads up to 21 Sep 2024

Other Metrics

Citations

Cited By

View all
  • (2024)DDS: DPU-Optimized Disaggregated StorageProceedings of the VLDB Endowment10.14778/3681954.368200217:11(3304-3317)Online publication date: 30-Aug-2024
  • (2024)A Survey of RDMA Distributed StorageProceedings of the 2024 5th International Conference on Computing, Networks and Internet of Things10.1145/3670105.3670199(534-539)Online publication date: 24-May-2024
  • (2024)A Memory-Disaggregated Radix TreeACM Transactions on Storage10.1145/366428920:3(1-41)Online publication date: 6-Jun-2024
  • (2024)D2Comp: Efficient Offload of LSM-tree Compaction with Data Processing Units on Disaggregated StorageACM Transactions on Architecture and Code Optimization10.1145/365658421:3(1-22)Online publication date: 9-Apr-2024
  • (2024)The I/O Driven Server: From SmartNICs to Data Movement ControllersACM SIGCOMM Computer Communication Review10.1145/3649171.364917453:3(9-17)Online publication date: 20-Feb-2024
  • (2024)RR-Compound: RDMA-Fused gRPC for Low Latency, High Throughput, and Easy InterfaceIEEE Transactions on Parallel and Distributed Systems10.1109/TPDS.2024.340439435:8(1488-1505)Online publication date: Aug-2024
  • (2024)P4Hauler: An Accelerator-Aware In-Network Load Balancer for Applications Performance BoostingIEEE Transactions on Cloud Computing10.1109/TCC.2024.338965812:2(697-711)Online publication date: Apr-2024
  • (2024)HADES: Hardware-Assisted Distributed Transactions in the Age of Fast Networks and SmartNICs2024 ACM/IEEE 51st Annual International Symposium on Computer Architecture (ISCA)10.1109/ISCA59077.2024.00062(785-800)Online publication date: 29-Jun-2024
  • (2024)HAL: Hardware-assisted Load Balancing for Energy-efficient SNIC-Host Cooperative Computing2024 ACM/IEEE 51st Annual International Symposium on Computer Architecture (ISCA)10.1109/ISCA59077.2024.00051(613-627)Online publication date: 29-Jun-2024
  • (2024)MINOS: Distributed Consistency and Persistency Protocol Implementation & Offloading to SmartNICs2024 IEEE International Symposium on High-Performance Computer Architecture (HPCA)10.1109/HPCA57654.2024.00076(1-17)Online publication date: 2-Mar-2024
  • Show More Cited By

View Options

Get Access

Login options

View options

PDF

View or Download as a PDF file.

PDF

eReader

View online with eReader.

eReader

Media

Figures

Other

Tables

Share

Share

Share this Publication link

Share on social media