research-article

Public Access

Bayesian factor analysis and performance measurement of the Linux forwarding architecture

Authors:

Patrick CrowleyAuthors Info & Claims

ANCS '18: Proceedings of the 2018 Symposium on Architectures for Networking and Communications Systems

Pages 28 - 40

https://doi.org/10.1145/3230718.3230726

Published: 23 July 2018 Publication History

Abstract

Linux software routers have many configuration options, most of which have received little attention from researchers. Over time, multiqueue NICs, NUMA architectures, and many changes to the kernel have altered the forwarding landscape. Here we investigate (i) allocation of NIC queues to processing cores, (ii) batch sizes at various layers, (iii) receive and transmit packet steering, and (iv) how FIB performance scales. Our experiments focus on forwarding minimum size packets, which maximize the packet processing stress for the kernel, at 10 Gbps. Our investigation uses Bayesian factor analysis, experimental design techniques, and kernel density estimation to robustly measure each factor's effect on the mean packet forwarding rate and mean RTT. This study does not seek bleeding edge performance. Rather, we elucidate key decisions for packet forwarding performance on a widely used platform. Our results show irqbalance is quite volatile, hyperthreading can provide over a 1 MPPS boost in forwarding performance, larger queue sizes do not improve forwarding but can add non-trivial latency (up to 30 ms), and both receive packet steering and low values of rx-usecs can induce receive livelock.

References

[1]

J. Bainbridge and J. Maxwell. Red Hat Enterprise Linux Network Performance Tuning Guide. Available: https://access.redhat.com/sites/default/files/attachments/20150325_network_performance_tuning.pdf, Mar 2015. {Online; accessed 6 December 2016}.

[2]

A. Beifuß, D. Raumer, P. Emmerich, T. M. Runge, F. Wohlfart, B. E. Wolfinger, and G. Carle. A Study of Networking Software Induced Latency. In 2nd IEEE International Conference and Workshops on Networked Systems, NetSys '15, pages 1--8. IEEE, 2015.

[3]

V. Bernat. Performance progression of IPv4 route lookup on Linux. Available: https://vincent.bernat.im/en/blog/2017-performance-progression-ipv4-route-lookup-linux, Jul 2017. {Online; accessed 15 June 2018}.

[4]

A. Bianco, J. M. Finochietto, G. Galante, M. Mellia, and F. Neri. Open-Source PC-Based Software Routers: A Viable Approach to High-Performance Packet Switching, pages 353--366. QoS-IP '05. Springer Berlin Heidelberg, Berlin, Heidelberg, 2005.

Digital Library

[5]

R. Bolla and R. Bruschi. The IP Lookup Mechanism in a Linux Software Router: Performance Evaluation and Optimizations. In 2007 Workshop on High Performance Switching and Routing, pages 1--6, May 2007.

[6]

R. Bolla and R. Bruschi. Linux Software Router: Data Plane Optimization and Performance Evaluation. Academy Publisher Journal of Networks, 2(3):6--17, Jun 2007.

[7]

R. Bolla and R. Bruschi. An effective forwarding architecture for SMP Linux routers. In 2008 4th International Telecommunication Networking Workshop on QoS in Multiservice IP Networks, pages 210--216, Feb 2008.

[8]

R. Bolla and R. Bruschi. PC-Based Software Routers: High Performance and Application Service Support. In Proceedings of the ACM Workshop on Programmable Routers for Extensible Services of Tomorrow, PRESTO '08, pages 27--32, New York, NY, USA, 2008. ACM.

Digital Library

[9]

S. Bradner and J. McQuaid. RFC 2544: Benchmarking Methodology for Network Interconnect Devices, 1999.

Digital Library

[10]

J. Brouer. Packet Per Sec measurements for improving the Linux Kernel network stack. Available: https://netoptimizer.blogspot.com/2014/09/packet-per-sec-measurements-for.html, Sep 2014. {Online; accessed 6 December 2016}.

[11]

J. Brouer. Re: ixgbe: increase default TX ring buffer to 1024. Available: https://patchwork.ozlabs.org/patch/348793/, May 2014. {Online; accessed 9 April 2018}.

[12]

O. Cochard-Labbé. Setting-up a forwarding performance benchmark lab. Available: https://bsdrp.net/documentation/examples/setting_up_a_forwarding_performance_benchmark_lab, Nov 2016. {Online; accessed 6 December 2016}.

[13]

J. Corbet. RPS: Receive Packet Steering. Available: https://lwn.net/Articles/362339/, Nov 2009. {Online; accessed 6 December 2016}.

[14]

G. Dallel. Randomized (Complete) Block Designs. Available: http://www.jerrydallal.com/lhsp/ranblock.htm. {Online; accessed 15 January 2017}.

[15]

J. Demato. Monitoring and Tuning the Linux Network Stack: Receiving Data. Available: http://blog.packagecloud.io/eng/2016/06/22/monitoring-tuning-linux-networking-stack-receiving-data/, Jun 2016. {Online; accessed 6 December 2016}.

[16]

M. Dobrescu, K. Argyraki, and S. Ratnasamy. Toward Predictable Performance in Software Packet-Processing Platforms. In Presented as part of the 9th USENIX Symposium on Networked Systems Design and Implementation, NSDI '12, pages 141--154, San Jose, CA, 2012. USENIX.

Digital Library

[17]

A. Duyck. Re: CPU scheduler to TXQ binding? (ixgbe vs. igb). Available: http://lists.openwall.net/netdev/2014/09/17/67, Sep 2014. {Online; accessed 6 December 2016}.

[18]

P. Emmerich, D. Raumer, F. Wohlfart, and G. Carle. Assessing Soft-and Hardware Bottlenecks in PC-Based Packet Forwarding Systems. In The Fourteenth International Conference on Networks, ICN '15. IARIA, Apr 2015.

[19]

T. Herbert and W. de Bruijn. scaling.txt. Available: https://www.kernel.org/doc/Documentation/networking/scaling.txt. {Online; accessed 6 December 2016}.

[20]

Intel. Data Plane Development Kit. Available: http://dpdk.org. {Online; accessed 11 December 2016}.

[21]

JASP Team. JASP (Version 0.8.6){Computer software}. Available: https://jasp-stats.org/, 2018. {Online; accesed 9 April 2018}.

[22]

D. Kirchner, R. Ferdous, R. L. Cigno, L. Maccari, M. Gallo, D. Perino, and L. Saino. Augustus: A CCN Router for Programmable Networks. In Proceedings of the 3rd ACM Conference on Information-Centric Networking, ACM-ICN '16, pages 31--39, New York, NY, USA, 2016. ACM.

Digital Library

[23]

D. Miller. Re: performance issue of netdev_budget and dev_weight with ixgbe. Available: http://lists.openwall.net/netdev/2009/03/20/117, Mar 2009. {Online; accessed 6 December 2016}.

[24]

I. Pepelnjak and T. Graf. Fast Linux Packet Forwarding with Thomas Graf. Available: http://blog.ipspace.net/2016/10/fast-linux-packet-forwarding-with.html, Oct 2016. {Online; accessed 6 December 2016}.

[25]

J. N. Rouder, R. D. Morey, J. Verhagen, A. R. Swagman, and E.-J. Wagenmakers. Bayesian analysis of factorial designs. Psychological Methods, 22(2):304--321, 2017.

[26]

T. M. Runge, A. Beifufß, and B. E. Wolfinger. Low Latency Network Traffic Processing with Commodity Hardware. In Proceedings of the International Symposium on Performance Evaluation of Computer and Telecommunication Systems, Spects '15, pages 1--8, San Diego, CA, USA, 2015. Society for Computer Simulation International.

Digital Library

[27]

D. Taht. Realtime Response Under Load (RRUL) Specification. Available: https://www.bufferbloat.net/projects/bloat/wiki/RRUL_Spec/. {Online; accessed 9 April 2018}.

[28]

M. Varlese. Performance Analysis, Tuning and Tools on SUSE Linux Enterprise Products. Available: https://www.suse.com/documentation/suse-best-practices/singlehtml/sbp-performance-tuning/sbp-performance-tuning.html, Nov 2017. {Online; accessed 15 June 2018}.

[29]

C. Wiseman, J. Turner, M. Becchi, P. Crowley, J. DeHart, M. Haitjema, S. James, F. Kuhns, J. Lu, J. Parwatikar, R. Patney, M. Wilson, K. Wong, and D. Zar. A Remotely Accessible Network Processor-based Router for Network Experimentation. In Proceedings of the 4th ACM/IEEE Symposium on Architectures for Networking and Communications Systems, ANCS '08, pages 20--29, New York, NY, USA, 2008. ACM.

Digital Library

[30]

Q. Ye and M. MacGregor. Hardware bottleneck evaluation and analysis of a software PC-based router. In 2008 International Symposium on Performance Evaluation of Computer and Telecommunication Systems, pages 480--487. IEEE, June 2008.

Cited By

Bhattacharjee RR RPrasanna Kumar KMV VAthithan GSahadevan A(2021)Scalable flow probe architecture for 100 Gbps+ rates on commodity hardware: Design Considerations and ApproachJournal of Parallel and Distributed Computing10.1016/j.jpdc.2021.04.015Online publication date: May-2021
https://doi.org/10.1016/j.jpdc.2021.04.015
van Bemten ADeric NVarasteh ABlenk ASchmid SKellerer W(2019)Empirical Predictability Study of SDN Switches2019 ACM/IEEE Symposium on Architectures for Networking and Communications Systems (ANCS)10.1109/ANCS.2019.8901878(1-13)Online publication date: Sep-2019
https://doi.org/10.1109/ANCS.2019.8901878

Index Terms

Bayesian factor analysis and performance measurement of the Linux forwarding architecture
1. Mathematics of computing
  1. Probability and statistics
2. Networks
  1. Network components
    1. Intermediate nodes
      1. Routers
    2. Middle boxes / network appliances
  2. Network performance evaluation
    1. Network experimentation
    2. Network measurement

Recommendations

Performance of IP-forwarding of Linux hosts with multiple network interfaces

This paper evaluates and compares the performance of IP-packet forwarding of a Linux host equipped with multiple network interface cards (NICs), namely two receiving NICs and one transmitting NIC. We consider a Linux host with SMP (Symmetric ...
Multicast performance measurement on a high-performance IP backbone

This paper describes an approach to IP multicast performance measurement on the National Science Foundation's very-high-speed Backbone Network Service (vBNS). Using OC-12c ATM attached workstations that act as either multicast senders or receivers ...
Multicast forwarding over ATM: Native approaches

Multicasting is growing in importance as new multimedia applications are devised. Throughout this article, multicasting is understood as the efficient multipoint-to-multipoint transmission of information (in terms of network resource consumption) between ...

Comments

Please enable JavaScript to view thecomments powered by Disqus.

Information & Contributors

Information

Published In

cover image ACM Conferences

ANCS '18: Proceedings of the 2018 Symposium on Architectures for Networking and Communications Systems

July 2018

181 pages

ISBN:9781450359023

DOI:10.1145/3230718

Program Chairs:
Theophilus Benson
Brown University
,
Noa Zilberman
University of Cambridge

Copyright © 2018 ACM.

Permission to make digital or hard copies of all or part of this work for personal or classroom use is granted without fee provided that copies are not made or distributed for profit or commercial advantage and that copies bear this notice and the full citation on the first page. Copyrights for components of this work owned by others than the author(s) must be honored. Abstracting with credit is permitted. To copy otherwise, or republish, to post on servers or to redistribute to lists, requires prior specific permission and/or a fee. Request permissions from [email protected].

Sponsors

Publisher

Association for Computing Machinery

New York, NY, United States

Publication History

Published: 23 July 2018

Permissions

Request permissions for this article.

Request Permissions

Check for updates

Qualifiers

Research-article

Funding Sources

National Science Foundation

Conference

ANCS '18

Sponsor:

ANCS '18: Symposium on Architectures for Networking and Communications Systems

July 23 - 24, 2018

New York, Ithaca

Acceptance Rates

Overall Acceptance Rate 88 of 314 submissions, 28%

Contributors

Other Metrics

View Article Metrics

Bibliometrics & Citations

Bibliometrics

Article Metrics

2
Total Citations
View Citations
402
Total Downloads

Downloads (Last 12 months)46
Downloads (Last 6 weeks)6

Reflects downloads up to 19 Nov 2024

Other Metrics

View Author Metrics

Citations

Cited By

Bhattacharjee RR RPrasanna Kumar KMV VAthithan GSahadevan A(2021)Scalable flow probe architecture for 100 Gbps+ rates on commodity hardware: Design Considerations and ApproachJournal of Parallel and Distributed Computing10.1016/j.jpdc.2021.04.015Online publication date: May-2021
https://doi.org/10.1016/j.jpdc.2021.04.015
van Bemten ADeric NVarasteh ABlenk ASchmid SKellerer W(2019)Empirical Predictability Study of SDN Switches2019 ACM/IEEE Symposium on Architectures for Networking and Communications Systems (ANCS)10.1109/ANCS.2019.8901878(1-13)Online publication date: Sep-2019
https://doi.org/10.1109/ANCS.2019.8901878

View Options

View options

PDF

View or Download as a PDF file.

eReader

View online with eReader.

Login options

Check if you have access through your login credentials or your institution to get full access on this article.

Full Access

Get this Publication

Media

Figures

Other

Tables

View Table of Contents