Article

SENIC: scalable NIC for end-host rate limiting

Authors:

Sivasankar Radhakrishnan,

Vimalkumar Jeyakumar,

Amin VahdatAuthors Info & Claims

NSDI'14: Proceedings of the 11th USENIX Conference on Networked Systems Design and Implementation

Pages 475 - 488

Published: 02 April 2014 Publication History

Abstract

Rate limiting is an important primitive for managing server network resources. Unfortunately, software-based rate limiting suffers from limited accuracy and high CPU overhead, and modern NICs only support a handful of rate limiters. We present SENIC, a NIC design that can natively support 10s of thousands of rate limiters--100x to 1000x the number available in NICs today. The key idea is that the host CPU only classifies packets, enqueues them in per-class queues in host memory, and specifies rate limits for each traffic class. On the NIC, SENIC maintains class metadata, computes the transmit schedule, and only pulls packets from host memory when they are ready to be transmitted (on a real time basis). We implemented SENIC on NetFPGA, with 1000 rate limiters requiring just 30KB SRAM, and it was able to accurately pace packets. Further, in a memcached benchmark against software rate limiters, SENIC is able to sustain up to 250% higher load, while simultaneously keeping tail latency under 4ms at 90% network utilization.

References

[1]

ALIZADEH, M., ATIKOGLU, B., KABBANI, A., LAKSHMIKANTHA, A., PAN, R., PRABHAKAR, B., AND SEAMAN, M. Data Center Transport Mechanisms: Congestion Control Theory and IEEE Standardization. In 46th Annual Allerton Conference on Communication, Control, and Computing (2008).

[2]

ALIZADEH, M., GREENBERG, A., MALTZ, D. A., PADHYE, J., PATEL, P., PRABHAKAR, B., SENGUPTA, S., AND SRIDHARAN, M. Data Center TCP (DCTCP). In SIGCOMM (2010).

Digital Library

[3]

ALIZADEH, M., KABBANI, A., EDSALL, T., PRABHAKAR, B., VAHDAT, A., AND YASUDA, M. Less Is More: Trading a Little Bandwidth for Ultra-Low Latency in the Data Center. In NSDI (2012).

Digital Library

[4]

BALLANI, H., COSTA, P., KARAGIANNIS, T., AND ROWSTRON, A. Towards Predictable Datacenter Networks. In SIGCOMM (2011).

Digital Library

[5]

BENNETT, J. C., AND ZHANG, H. WF²Q : Worst-case Fair Weighted Fair Queueing. In INFOCOM (1996).

Digital Library

[6]

BENNETT, J. C. R., AND ZHANG, H. Hierarchical Packet Fair Queueing Algorithms. In SIGCOMM (1996).

Digital Library

[7]

BENSON, T., AKELLA, A., AND MALTZ, D. A. Network Traffic Characteristics of Data Centers in the Wild. In IMC (2010).

Digital Library

[8]

CHECCONI, F., RIZZO, L., AND VALENTE, P. QFQ: Efficient Packet Scheduling With Tight Guarantees. In IEEE/ACM Transactions on Networking (June 2013).

Digital Library

[9]

DEMERS, A., KESHAV, S., AND SHENKER, S. Analysis and Simulation of a Fair Queueing Algorithm. In SIGCOMM (1989).

Digital Library

[10]

FLAJSLIK, M., AND ROSENBLUM, M. Network Interface Design for Low Latency Request-Response Protocols. In USENIX ATC (2013).

Digital Library

[11]

Intel 82599 10GbE Controller. http://www.intel. com/content/dam/doc/datasheet/82599-10-gbe-controller-datasheet.pdf.

[12]

JEYAKUMAR, V., ALIZADEH, M., MAZIÈRES, D., PRABHAKAR, B., KIM, C., AND GREENBERG, A. EyeQ: Practical Network Performance Isolation at the Edge. In NSDI (2013).

Digital Library

[13]

KAPOOR, R., PORTER, G., TEWARI, M., VOELKER, G. M., AND VAHDAT, A. Chronos: Predictable Low Latency for Data Center Applications. In SOCC (2012).

Digital Library

[14]

KUMAR, G., KANDULA, S., BODIK, P., AND MENACHE, I. Virtualizing Traffic Shapers for Practical Resource Allocation. In HotCloud (2013).

[15]

LEMON, J. Kqueue - A Generic and Scalable Event Notification Facility. In USENIX ATC (2001).

Digital Library

[16]

LOCKWOOD, J. W., MCKEOWN, N., WATSON, G., GIBB, G., HARTKE, P., NAOUS, J., RAGHURAMAN, R., AND LUO, J. NetFPGA - An Open Platform for Gigabitrate Network Switching and Routing. In IEEE International Conference on Microelectronic Systems Education (2007).

Digital Library

[17]

LU, G., GUO, C., LI, Y., ZHOU, Z., YUAN, T., WU, H., XIONG, Y., GAO, R., AND ZHANG, Y. ServerSwitch: A Programmable and High Performance Platform for Data Center Networks. In NSDI (2011).

Digital Library

[18]

Mellanox Connect-X3. http://www.mellanox.com/related-docs/prod_adapter_cards/ConnectX3_ EN_Card.pdf.

[19]

MOGUL, J. C. TCP Offload Is a Dumb Idea Whose Time Has Come. In HotOS (2003).

Digital Library

[20]

MOGUL, J. C., MUDIGONDA, J., SANTOS, J. R., AND TURNER, Y. The NIC Is the Hypervisor: Bare-Metal Guests in IaaS Clouds. In HotOS (2013).

Digital Library

[21]

MOON, S., REXFORD, J., AND SHIN, K. G. Scalable Hardware Priority Queue Architectures for High-Speed Packet Switches. IEEE Transactions on Computers (Nov. 2000).

Digital Library

[22]

MOSHREF, M., YU, M., SHARMA, A., AND GOVINDAN, R. Scalable Rule Management for Data Centers. In NSDI (2013).

Digital Library

[23]

Myri-10G PCI Express Network Adapter. https://www.myricom.com/products/network-adapters/10g-pcie-8b-2s.html, Retrieved 25 September 2013.

[24]

MYSORE, R. N., PORTER, G., AND VAHDAT, A. Fas-Trak: Enabling Express Lanes in Multi-Tenant Data Centers. In CoNEXT (2013).

[25]

ONGARO, D., RUMBLE, S. M., STUTSMAN, R., OUSTERHOUT, J., AND ROSENBLUM, M. Fast Crash Recovery in RAMCloud. In SOSP (2011).

Digital Library

[26]

OpenFlow Consortium. http:://www.openflow.org.

[27]

PAN, R., BRESLAU, L., PRABHAKAR, B., AND SHENKER, S. Approximate Fairness Through Differential Dropping. SIGCOMM CCR (Apr. 2003).

Digital Library

[28]

PCI-SIG SR-IOV Primer: An Introduction to SR-IOV Technology. http://www.intel.com/content/www/us/en/pci-express/pci-sig-sr-iov-primer-sr-iov-technology-paper.html, Retrieved 25 September 2013.

[29]

PHANISHAYEE, A., KREVAT, E., VASUDEVAN, V., ANDERSEN, D. G., GANGER, G. R., GIBSON, G. A., AND SESHAN, S. Measurement and Analysis of TCP Throughput Collapse in Cluster-based Storage Systems. In USENIX FAST (2008).

Digital Library

[30]

RADHAKRISHNAN, S., JEYAKUMAR, V., KABBANI, A., PORTER, G., AND VAHDAT, A. NicPic: Scalable and Accurate End-Host Rate Limiting. In HotCloud (2013).

[31]

RAMCloud RPC Performance Numbers. https://ramcloud.stanford.edu/wiki/display/ramcloud/RPC+Performance+Numbers, Retrieved 25 September 2013.

[32]

RDMA Aware Networks Programming User Manual. http://www.mellanox.com/related-docs/prod_software/RDMA_Aware_Programming_user_ manual.pdf, Retrieved 25 September 2013.

[33]

REXFORD, J., BONOMI, F., GREENBERG, A., AND WONG, A. A Scalable Architecture for Fair Leaky-Bucket Shaping. In INFOCOM (1997).

Digital Library

[34]

RIZZO, L. netmap: a novel framework for fast packet I/O. In USENIX ATC (2012).

Digital Library

[35]

RODRIGUES, H., SANTOS, J. R., TURNER, Y., SOARES, P., AND GUEDES, D. Gatekeeper: Supporting Bandwidth Guarantees for Multi-tenant Datacenter Networks. In WIOV (2011).

Digital Library

[36]

RUMBLE, S. M., ONGARO, D., STUTSMAN, R., ROSENBLUM, M., AND OUSTERHOUT, J. K. It's Time for Low Latency. In HotOS (2011).

Digital Library

[37]

SHALEV, L., SATRAN, J., BOROVIK, E., AND BENYEHUDA, M. IsoStack: Highly Efficient Network Processing on Dedicated Cores. In USENIX ATC (2010).

Digital Library

[38]

SHIEH, A., KANDULA, S., GREENBERG, A., AND KIM, C. Seawall: Performance Isolation for Cloud Datacenter Networks. In HotCloud (2010).

Digital Library

[39]

SHREEDHAR, M., AND VARGHESE, G. Efficient Fair Queueing Using Deficit Round Robin. In SIGCOMM (1995).

Digital Library

[40]

WILSON, C., BALLANI, H., KARAGIANNIS, T., AND ROWTRON, A. Better Never than Late: Meeting Deadlines in Datacenter Networks. In SIGCOMM (2011).

Digital Library

[41]

Xilinx User Community Forums: ML506 board: Why my DMA IP hangs OS? http://forums.xilinx.com/t5/PCI-Express/ML506-board-Why-my-DMA-IP-hangs-OS/td-p/94298, Retrieved 25 September 2013.

Cited By

Sadok HZhao ZChoung VAtre NBerger DHoe JPanda ASherry JAngel SKasikci BKohler E(2021)We need kernel interposition over the network dataplaneProceedings of the Workshop on Hot Topics in Operating Systems10.1145/3458336.3465281(152-158)Online publication date: 1-Jun-2021
https://dl.acm.org/doi/10.1145/3458336.3465281
Fried JRuan ZOusterhout ABelay ALu SHowell J(2020)CaladanProceedings of the 14th USENIX Conference on Operating Systems Design and Implementation10.5555/3488766.3488782(281-297)Online publication date: 4-Nov-2020
https://dl.acm.org/doi/10.5555/3488766.3488782
Zilberman NBracha GSchzukin GLorch JYu M(2019)StardustProceedings of the 16th USENIX Conference on Networked Systems Design and Implementation10.5555/3323234.3323247(141-159)Online publication date: 26-Feb-2019
https://dl.acm.org/doi/10.5555/3323234.3323247
Show More Cited By

SENIC: scalable NIC for end-host rate limiting
1. General and reference
  1. Cross-computing tools and techniques
2. Social and professional topics
  1. Professional topics
    1. Computing profession

Recommendations

SRVM: Hypervisor Support for Live Migration with Passthrough SR-IOV Network Devices
VEE '16

Single-Root I/O Virtualization (SR-IOV) is a specification that allows a single PCI Express (PCIe) device (ysical function or PF) to be used as multiple PCIe devices (virtual functions or VF). In a virtualization system, each VF can be directly assigned ...
SRVM: Hypervisor Support for Live Migration with Passthrough SR-IOV Network Devices
VEE '16: Proceedings of the12th ACM SIGPLAN/SIGOPS International Conference on Virtual Execution Environments

Single-Root I/O Virtualization (SR-IOV) is a specification that allows a single PCI Express (PCIe) device (ysical function or PF) to be used as multiple PCIe devices (virtual functions or VF). In a virtualization system, each VF can be directly assigned ...
Nosv

nOSV can provide a bare-metal like performance for HPC applications on Cloud.The CPU cores and main memory are not shared among guest VMs of nOSV.Dedicated I/O resources are allocated to I/O sensitive HPC guests.Other virtualization environments can run ...

Comments

Please enable JavaScript to view thecomments powered by Disqus.

Information & Contributors

Information

Published In

cover image ACM Other conferences

NSDI'14: Proceedings of the 11th USENIX Conference on Networked Systems Design and Implementation

April 2014

546 pages

ISBN:9781931971096

Program Chairs:
Ratul Mahajan
Microsoft Research
,
Ion Stoica
University of California at Berkeley

Sponsors

USENIX Assoc: USENIX Assoc

In-Cooperation

Publisher

USENIX Association

United States

Publication History

Published: 02 April 2014

Check for updates

Qualifiers

Article

Contributors

Other Metrics

View Article Metrics

Bibliometrics & Citations

Bibliometrics

Article Metrics

36
Total Citations
View Citations
0
Total Downloads

Downloads (Last 12 months)0
Downloads (Last 6 weeks)0

Reflects downloads up to 01 Oct 2024

Other Metrics

View Author Metrics

Citations

Cited By

Sadok HZhao ZChoung VAtre NBerger DHoe JPanda ASherry JAngel SKasikci BKohler E(2021)We need kernel interposition over the network dataplaneProceedings of the Workshop on Hot Topics in Operating Systems10.1145/3458336.3465281(152-158)Online publication date: 1-Jun-2021
https://dl.acm.org/doi/10.1145/3458336.3465281
Fried JRuan ZOusterhout ABelay ALu SHowell J(2020)CaladanProceedings of the 14th USENIX Conference on Operating Systems Design and Implementation10.5555/3488766.3488782(281-297)Online publication date: 4-Nov-2020
https://dl.acm.org/doi/10.5555/3488766.3488782
Zilberman NBracha GSchzukin GLorch JYu M(2019)StardustProceedings of the 16th USENIX Conference on Networked Systems Design and Implementation10.5555/3323234.3323247(141-159)Online publication date: 26-Feb-2019
https://dl.acm.org/doi/10.5555/3323234.3323247
Stephens BAkella ASwift MLorch JYu M(2019)LoomProceedings of the 16th USENIX Conference on Networked Systems Design and Implementation10.5555/3323234.3323238(33-46)Online publication date: 26-Feb-2019
https://dl.acm.org/doi/10.5555/3323234.3323238
Saeed AZhao YDukkipati NAmmar MZegura EHarras KVahdat ALorch JYu M(2019)EiffelProceedings of the 16th USENIX Conference on Networked Systems Design and Implementation10.5555/3323234.3323237(17-31)Online publication date: 26-Feb-2019
https://dl.acm.org/doi/10.5555/3323234.3323237
He YWu W(2019)Fully Functional Rate Limiter Design on Programmable Hardware SwitchesProceedings of the ACM SIGCOMM 2019 Conference Posters and Demos10.1145/3342280.3342344(159-160)Online publication date: 19-Aug-2019
https://dl.acm.org/doi/10.1145/3342280.3342344
Kumar PDukkipati NLewis NCui YWang YLi CValancius VAdriaens JGribble SFoster NVahdat AWu JHall W(2019)PicNICProceedings of the ACM Special Interest Group on Data Communication10.1145/3341302.3342093(351-366)Online publication date: 19-Aug-2019
https://dl.acm.org/doi/10.1145/3341302.3342093
Shrivastav VWu JHall W(2019)Fast, scalable, and programmable packet scheduler in hardwareProceedings of the ACM Special Interest Group on Data Communication10.1145/3341302.3342090(367-379)Online publication date: 19-Aug-2019
https://dl.acm.org/doi/10.1145/3341302.3342090
Qian KCheng WZhang TRen FWu JHall W(2019)Gentle flow controlProceedings of the ACM Special Interest Group on Data Communication10.1145/3341302.3342065(75-89)Online publication date: 19-Aug-2019
https://dl.acm.org/doi/10.1145/3341302.3342065
Kaufmann AStamler TPeter SSharma NKrishnamurthy AAnderson T(2019)TASProceedings of the Fourteenth EuroSys Conference 201910.1145/3302424.3303985(1-16)Online publication date: 25-Mar-2019
https://dl.acm.org/doi/10.1145/3302424.3303985
Show More Cited By

View Options

View options

Media

Figures

Other

Tables

View Table of Contents