article

Architectural breakdown of end-to-end latency in a TCP/IP network

Authors:

Parthasarathy Sarangam,

Ram Huggahalli,

Siddharth KulkarniAuthors Info & Claims

International Journal of Parallel Programming, Volume 37, Issue 6

Pages 556 - 571

https://doi.org/10.1007/s10766-009-0109-6

Published: 01 December 2009 Publication History

Abstract

Adoption of the 10GbE Ethernet standard as a high performance interconnect has been impeded by two important performance-oriented considerations: (1) processing requirements of common protocol stacks and (2) end-to-end latency. The overheads of typical software based protocol stacks on CPU utilization and throughput have been well evaluated in several recent studies. We focus on end-to-end latency and present a detailed characterization across typical server system hardware and software stack components. We demonstrate that application level end-to-end one-way latency with a 10GbE connection can be as low as 10 µs for a single isolated request in a standard Linux network stack. The paper analyzes the components of the latency and discusses possible significant variations to the components under realistic conditions. We found that methods that optimize for throughput can significantly compromise Ethernet based latencies. Methods to pursue reducing the minimum latency and controlling the variations are presented.

References

[1]

Foong, A., Huff, T., Hum, H., Patwardhan, J., Regnier, G.: TCP performance re-visited. In: Proceedings of the IEEE International Symposium on Performance of Systems & Software, Austin, Mar. 2003.

[2]

Mogul, J.: TCP offload is a dumb idea whose time has come. In: Proceedings of the 9th Workshop on Hot Topics in Operating Systems (HotOS IX). Usenix Assoc. www.usenix.org/events/hotos03/tech/ full_papers/mogul/mogul.pdf (2003).

[3]

Huggahalli, R., et al.: Direct cache access for high bandwidth network I/O. In: International Symposium on Computer Architecture (ISCA) http://www.cs.wisc.edu~isca2005/papers/02A-02.PDF (2005).

[4]

Regnier, G., et. al.: TCP onloading for data center servers. In: IEEE Computer, Nov. 2004.

Digital Library

[5]

http://vmi.ncsa.uiuc.edu/performance/pmb_lt.php

[6]

http://www.top500.org/stats/28/connfam/

[7]

http://download.qlogic.com/datasheet/53873/SN0058045_00D.pdf

[8]

Fengm, W., Balaji, P., Baron, C., Bhuyan, L.N., Panda, D.K.: Performance characterization of a 10-Gigabit Ethernet TOE; high performance interconnects. In: Proceedings of the 13th Symposium, Aug. 2005. http://nowlab.cse.ohio-state.edu/publications/conf-presentations/2005/balaji-hoti05.pdf

[9]

Express Base Specification Revision, P.C.I.: 2.0. http://www.pcisig.com (2006).

[10]

http://download.intel.com/design/chipsets/datashts/31307103.pdf

[11]

http://linux-net.osdl.org/index.php/NAPI

[12]

IA-32 Intel® architecture software developer's manual volume 3: system programming guide appendix A, June 2005.

[13]

Foong, A., Fung, J., Newell, D.: An in-depth analysis of the impact of processor affinity on network performance. In: Proceedings of the IEEE International Conference Networks, IEEE Press (2004).

[14]

http://www.usenix.org/publications/library/proceedings/als01/full_papers/jamal/jamal.pdf

[15]

http://download.intel.com/design/network/applnots/ap450.pdf

[16]

Hansen, J., Jul, E.: Latency reduction using a polling scheduler. In: Proceedings of the Second Workshop on Cluster-Based Computing, ACM-SIGARCH (2000).

[17]

http://www.ieee802.org/3/ar/public/0503/wadekar_1_0503.pdf

[18]

http://www.microsoft.com/whdc/device/network/NDIS_RSS.mspx

Cited By

Tang WHan YAi TLi GYu BYang X(2024)Yggdrasil: Reducing Network I/O Tax with (CXL-Based) Distributed Shared MemoryProceedings of the 53rd International Conference on Parallel Processing10.1145/3673038.3673138(597-606)Online publication date: 12-Aug-2024
https://dl.acm.org/doi/10.1145/3673038.3673138
Fang TXu LSrisa-an WPatel J(2023)Evaluation of the ProgHW/SW Architectural Design Space of Bandwidth EstimationPassive and Active Measurement10.1007/978-3-031-28486-1_12(257-283)Online publication date: 21-Mar-2023
https://dl.acm.org/doi/10.1007/978-3-031-28486-1_12
Saito SFujimoto KKaneko MShiraga A(2022)Fast Offloading of Accelerator Task over Network with Hardware AssistanceEdge Computing – EDGE 202210.1007/978-3-031-23470-5_1(1-17)Online publication date: 10-Dec-2022
https://dl.acm.org/doi/10.1007/978-3-031-23470-5_1
Show More Cited By

Architectural breakdown of end-to-end latency in a TCP/IP network
1. Networks

Recommendations

An evolutionary approach to improve end-to-end performance in tcp/ip networks
Integrated network interfaces for high-bandwidth TCP/IP
Proceedings of the 2006 ASPLOS Conference

This paper proposes new network interface controller (NIC) designs that take advantage of integration with the host CPU to provide increased flexibility for operating system kernel-based performance optimization.We believe that this approach is more ...
A comparison of mechanisms for improving mobile IP handoff latency for end-to-end TCP
MobiCom '03: Proceedings of the 9th annual international conference on Mobile computing and networking

Handoff latency results in packet losses and severe End-to-End TCP performance degradation as TCP, perceiving these losses as congestion, causes source throttling or retransmission. In order to mitigate these effects, various Mobile IP(v6) extensions ...

Comments

Please enable JavaScript to view thecomments powered by Disqus.

Information & Contributors

Information

Published In

cover image International Journal of Parallel Programming

International Journal of Parallel Programming Volume 37, Issue 6

Special issue on the 19th international symposium on computer architecture and high performance computing (SBAC-PAD 2007)

December 2009

75 pages

ISSN:0885-7458

Issue’s Table of Contents

Publisher

Kluwer Academic Publishers

United States

Publication History

Published: 01 December 2009

Accepted: 21 May 2009

Received: 14 April 2008

Author Tags

Qualifiers

Article

Contributors

Other Metrics

View Article Metrics

Bibliometrics & Citations

Bibliometrics

Article Metrics

22
Total Citations
View Citations
0
Total Downloads

Downloads (Last 12 months)0
Downloads (Last 6 weeks)0

Reflects downloads up to 14 Dec 2024

Other Metrics

View Author Metrics

Citations

Cited By

Tang WHan YAi TLi GYu BYang X(2024)Yggdrasil: Reducing Network I/O Tax with (CXL-Based) Distributed Shared MemoryProceedings of the 53rd International Conference on Parallel Processing10.1145/3673038.3673138(597-606)Online publication date: 12-Aug-2024
https://dl.acm.org/doi/10.1145/3673038.3673138
Fang TXu LSrisa-an WPatel J(2023)Evaluation of the ProgHW/SW Architectural Design Space of Bandwidth EstimationPassive and Active Measurement10.1007/978-3-031-28486-1_12(257-283)Online publication date: 21-Mar-2023
https://dl.acm.org/doi/10.1007/978-3-031-28486-1_12
Saito SFujimoto KKaneko MShiraga A(2022)Fast Offloading of Accelerator Task over Network with Hardware AssistanceEdge Computing – EDGE 202210.1007/978-3-031-23470-5_1(1-17)Online publication date: 10-Dec-2022
https://dl.acm.org/doi/10.1007/978-3-031-23470-5_1
Nayak PKnightly E(2021)uScopeProceedings of the Twenty-second International Symposium on Theory, Algorithmic Foundations, and Protocol Design for Mobile Networks and Mobile Computing10.1145/3466772.3467046(171-180)Online publication date: 26-Jul-2021
https://dl.acm.org/doi/10.1145/3466772.3467046
Jung CKim SYeom IWoo HKim Y(2021)GPU-Ether: GPU-native Packet I/O for GPU Applications on Commodity EthernetIEEE INFOCOM 2021 - IEEE Conference on Computer Communications10.1109/INFOCOM42981.2021.9488699(1-10)Online publication date: 10-May-2021
https://dl.acm.org/doi/10.1109/INFOCOM42981.2021.9488699
Li XCheng WZhang TRen FYang B(2020)Towards Power Efficient High Performance Packet I/OIEEE Transactions on Parallel and Distributed Systems10.1109/TPDS.2019.295774631:4(981-996)Online publication date: 1-Apr-2020
https://dl.acm.org/doi/10.1109/TPDS.2019.2957746
Martinov GLjubimov AMartinova L(2020)From classic CNC systems to cloud-based technology and backRobotics and Computer-Integrated Manufacturing10.1016/j.rcim.2019.10192763:COnline publication date: 1-Jun-2020
https://dl.acm.org/doi/10.1016/j.rcim.2019.101927
Zhang TLiu B(2019)Exposing End-to-End Delay in Software-Defined NetworkingInternational Journal of Reconfigurable Computing10.1155/2019/73639012019Online publication date: 1-Jan-2019
https://dl.acm.org/doi/10.1155/2019/7363901
Krauß DPaus FVahrenkamp NAsfour TAndelfinger PJohansson BJain S(2018)Evaluating and optimizing component-based robot architectures using network simulationProceedings of the 2018 Winter Simulation Conference10.5555/3320516.3320650(1061-1072)Online publication date: 9-Dec-2018
https://dl.acm.org/doi/10.5555/3320516.3320650
Emmerich PRaumer DGallenmüller SWohlfart FCarle G(2018)Throughput and Latency of Virtual Switching with Open vSwitchJournal of Network and Systems Management10.1007/s10922-017-9417-026:2(314-338)Online publication date: 1-Apr-2018
https://dl.acm.org/doi/10.1007/s10922-017-9417-0
Show More Cited By

View Options

View options

Media

Figures

Other

Tables

View Issue’s Table of Contents