research-article

Dissecting Performance of Production QUIC

Authors:

Theophilus A. BensonAuthors Info & Claims

WWW '21: Proceedings of the Web Conference 2021

Pages 1157 - 1168

https://doi.org/10.1145/3442381.3450103

Published: 03 June 2021 Publication History

Abstract

IETF QUIC, the standardized version of Google’s UDP-based layer-4 network protocol, has seen increasing adoption from large Internet companies for its benefits over TCP. Yet despite its rapid adoption, performance analysis of QUIC in production is scarce. Most existing analyses have only used unoptimized open-source QUIC servers on non-tuned kernels: these analyses are unrepresentative of production deployments which raises the question of whether QUIC actually outperforms TCP in practice.

In this paper, we conduct one of the first comparative studies on the performance of QUIC and TCP against production endpoints hosted by Google, Facebook, and Cloudflare under various dimensions: network conditions, workloads, and client implementations.

To understand our results, we create a tool to systematically visualize the root causes of performance differences between the two protocols. Using our tool we make several key observations. First, while QUIC has some inherent advantages over TCP, such as worst-case 1-RTT handshakes, its overall performance is largely determined by the server’s choice of congestion-control algorithm and the robustness of its congestion-control implementation under edge-case network scenarios. Second, we find that some QUIC clients require non-trivial configuration tuning in order to achieve optimal performance. Lastly, we demonstrate that QUIC’s removal of head-of-line (HOL) blocking has little impact on web-page performance in practice. Taken together, our observations illustrate the fact that QUIC’s performance is inherently tied to implementation design choices, bugs, and configurations which implies that QUIC measurements are not always a reflection of the protocol and often do not generalize across deployments.

References

[1]

[n.d.]. . Retrieved September 19, 2020 from https://packages.gentoo.org/packages/net-libs/ngtcp2

[2]

[n.d.]. Chrome-har. Retrieved January 18, 2021 from https://www.npmjs.com/package/chrome-har

[3]

[n.d.]. Implementations. Retrieved September 18, 2020 from https://github.com/quicwg/base-drafts/wiki/Implementations

[4]

[n.d.]. Lighthouse. Retrieved November 26, 2020 from https://developers.google.com/web/tools/lighthouse/

[5]

[n.d.]. netem. Retrieved January 26, 2021 from https://wiki.linuxfoundation.org/networking/netem

[6]

[n.d.]. Puppeteer. Retrieved January 18, 2021 from https://pptr.dev/

[7]

2019. Speed Index. Retrieved November 26, 2020 from https://web.dev/speed-index/

[8]

2020. . Retrieved October 18, 2020 from https://github.com/facebookincubator/mvfst/commit/8a98805d

[9]

M. Allman, V. Paxson, and E. Blanton. 2009. TCP Congestion Control. RFC 5681. RFC Editor. http://www.rfc-editor.org/rfc/rfc5681.txt http://www.rfc-editor.org/rfc/rfc5681.txt.

[10]

João Almeida. 2019. How fast is QUIC protocol and what makes Bolina faster - PT. II. https://blog.codavel.com/how-fast-is-quic-and-what-makes-bolina-faster-pt-ii

[11]

Kornel Lesiński Andrew Galloni. 2019. Parallel streaming of progressive images. Retrieved November 29, 2020 from https://blog.cloudflare.com/parallel-streaming-of-progressive-images/

[12]

P. Biswal and O. Gnawali. 2016. Does QUIC Make the Web Faster?. In 2016 IEEE Global Communications Conference (GLOBECOM). 1–6.

[13]

Neal Cardwell, Yuchung Cheng, C. Stephen Gunn, Soheil Hassas Yeganeh, and Van Jacobson. 2016. BBR: Congestion-Based Congestion Control. ACM Queue 14, September-October (2016), 20 – 53. http://queue.acm.org/detail.cfm?id=3022184

Digital Library

[14]

Gaetano Carlucci, Luca De Cicco, and Saverio Mascolo. 2015. HTTP over UDP: An Experimental Investigation of QUIC. In Proceedings of the 30th Annual ACM Symposium on Applied Computing (Salamanca, Spain) (SAC ’15). Association for Computing Machinery, New York, NY, USA, 609–614. https://doi.org/10.1145/2695664.2695706

Digital Library

[15]

Yuchung Cheng, Neal Cardwell, and Nandita Dukkipati. 2017. RACK: a time-based fast loss detection algorithm for TCP. Internet-Draft draft-ietf-tcpm-rack-02. IETF Secretariat. http://www.ietf.org/internet-drafts/draft-ietf-tcpm-rack-02.txt http://www.ietf.org/internet-drafts/draft-ietf-tcpm-rack-02.txt.

[16]

S. Cook, B. Mathieu, P. Truong, and I. Hamchaoui. 2017. QUIC: Better for what and for whom?. In 2017 IEEE International Conference on Communications (ICC). 1–6.

[17]

Ian Swett David Schinazi, Fan Yang. 2020. Chrome is deploying HTTP/3 and IETF QUIC. Retrieved October 8, 2020 from https://blog.chromium.org/2020/10/chrome-is-deploying-http3-and-ietf-quic.html

[18]

Matt Menke Eric Roman. [n.d.]. NetLog: Chrome’s network logging system. Retrieved January 18, 2021 from https://www.chromium.org/developers/design-documents/network-stack/netlog

[19]

Uday Hiwarale. 2019. How the browser renders a web page? — DOM, CSSOM, and Rendering. Retrieved February 1, 2021 from https://medium.com/jspoint/how-the-browser-renders-a-web-page-dom-cssom-and-rendering-df10531c9969

[20]

Jana Iyengar and Ian Swett. 2020. QUIC Loss Detection and Congestion Control. Internet-Draft draft-ietf-quic-recovery-29. IETF Secretariat. http://www.ietf.org/internet-drafts/draft-ietf-quic-recovery-29.txt http://www.ietf.org/internet-drafts/draft-ietf-quic-recovery-29.txt.

[21]

Jana Iyengar and Martin Thomson. 2020. QUIC: A UDP-Based Multiplexed and Secure Transport. Internet-Draft draft-ietf-quic-transport-32. IETF Secretariat. http://www.ietf.org/internet-drafts/draft-ietf-quic-transport-32.txt http://www.ietf.org/internet-drafts/draft-ietf-quic-transport-32.txt.

[22]

Jana Iyengar and Martin Thomson. 2020. QUIC: A UDP-Based Multiplexed and Secure Transport. Internet-Draft draft-ietf-quic-transport-29. IETF Secretariat. http://www.ietf.org/internet-drafts/draft-ietf-quic-transport-29.txt http://www.ietf.org/internet-drafts/draft-ietf-quic-transport-29.txt.

[23]

Subodh Iyengar. 2018. Moving Fast at Scale: Experience Deploying IETF QUIC at Facebook. In Proceedings of the Workshop on the Evolution, Performance, and Interoperability of QUIC (Heraklion, Greece) (EPIQ’18). Association for Computing Machinery, New York, NY, USA, Keynote. https://doi.org/10.1145/3284850.3322434

Digital Library

[24]

Lu Jun. 2020. Simulate weak network environment using command line under Mac. Retrieved January 26, 2021 from https://programmer.group/simulate-weak-network-environment-using-command-line-under-mac.html

[25]

Arash Molavi Kakhki, Samuel Jero, David Choffnes, Cristina Nita-Rotaru, and Alan Mislove. 2017. Taking a Long Look at QUIC: An Approach for Rigorous Evaluation of Rapidly Evolving Transport Protocols. In Proceedings of the 2017 Internet Measurement Conference (London, United Kingdom) (IMC ’17). Association for Computing Machinery, New York, NY, USA, 290–303. https://doi.org/10.1145/3131365.3131368

Digital Library

[26]

P. K. Kharat, A. Rege, A. Goel, and M. Kulkarni. 2018. QUIC Protocol Performance in Wireless Networks. In 2018 International Conference on Communication and Signal Processing (ICCSP). 0472–0476.

[27]

Adam Langley, Alistair Riddoch, Alyssa Wilk, Antonio Vicente, Charles Krasic, Dan Zhang, Fan Yang, Fedor Kouranov, Ian Swett, Janardhan Iyengar, Jeff Bailey, Jeremy Dorfman, Jim Roskind, Joanna Kulik, Patrik Westin, Raman Tenneti, Robbie Shade, Ryan Hamilton, Victor Vasiliev, Wan-Teh Chang, and Zhongyi Shi. 2017. The QUIC Transport Protocol: Design and Internet-Scale Deployment. In Proceedings of the Conference of the ACM Special Interest Group on Data Communication (Los Angeles, CA, USA) (SIGCOMM ’17). Association for Computing Machinery, New York, NY, USA, 183–196. https://doi.org/10.1145/3098822.3098842

Digital Library

[28]

Jonathan Lipps. [n.d.]. Simulating Different Network Conditions For Virtual Devices. Retrieved November 26, 2020 from https://appiumpro.com/editions/104-simulating-different-network-conditions-for-virtual-devices

[29]

Diego Madariaga, Lucas Torrealba, Javier Madariaga, Javiera Bermúdez, and Javier Bustos-Jiménez. 2020. Analyzing the Adoption of QUIC From a Mobile Development Perspective. In Proceedings of the Workshop on the Evolution, Performance, and Interoperability of QUIC(Virtual Event, USA) (EPIQ ’20). Association for Computing Machinery, New York, NY, USA, 35–41. https://doi.org/10.1145/3405796.3405830

Digital Library

[30]

Robin Marx. 2020. Will HTTP/3 really be faster than HTTP/2? Perhaps. Retrieved October 10, 2020 from https://github.com/rmarx/holblocking-blogpost

[31]

Robin Marx., Tom De Decker., Peter Quax., and Wim Lamotte.2019. Of the Utmost Importance: Resource Prioritization in HTTP/3 over QUIC. In Proceedings of the 15th International Conference on Web Information Systems and Technologies - Volume 1: WEBIST,. INSTICC, SciTePress, 130–143. https://doi.org/10.5220/0008191701300143

[32]

Robin Marx, Joris Herbots, Wim Lamotte, and Peter Quax. 2020. Same Standards, Different Decisions: A Study of QUIC and HTTP/3 Implementation Diversity. In Proceedings of the Workshop on the Evolution, Performance, and Interoperability of QUIC (Virtual Event, USA) (EPIQ ’20). Association for Computing Machinery, New York, NY, USA, 14–20. https://doi.org/10.1145/3405796.3405828

Digital Library

[33]

Robin Marx, Maxime Piraux, Peter Quax, and Wim Lamotte. 2020. Debugging QUIC and HTTP/3 with Qlog and Qvis. In Proceedings of the Applied Networking Research Workshop (Virtual Event, Spain) (ANRW ’20). Association for Computing Machinery, New York, NY, USA, 58–66. https://doi.org/10.1145/3404868.3406663

Digital Library

[34]

M. Mathis, J. Mahdavi, S. Floyd, and A. Romanow. 1996. TCP Selective Acknowledgment Options. RFC 2018. RFC Editor.

[35]

Yang Chi Matt Joras. 2020. How Facebook is bringing QUIC to billions. Retrieved November 12, 2020 from https://engineering.fb.com/2020/10/21/networking-traffic/how-facebook-is-bringing-quic-to-billions/

[36]

Mattt. 2019. Network Link Conditioner. Retrieved November 13, 2020 from https://nshipster.com/network-link-conditioner/

[37]

Patrick Meenan. 2019. Better HTTP/2 Prioritization for a Faster Web. Retrieved November 29, 2020 from https://blog.cloudflare.com/better-http-2-prioritization-for-a-faster-web/

[38]

Javad Nejati and Aruna Balasubramanian. 2020. WProfX: A Fine-Grained Visualization Tool for Web Page Loads. Proc. ACM Hum.-Comput. Interact. 4, EICS, Article 73 (June 2020), 22 pages. https://doi.org/10.1145/3394975

Digital Library

[39]

Kyle Nekritz and Subodh Iyengar. 2017. Building Zero protocol for fast, secure mobile connections. Retrieved August 4, 2020 from https://engineering.fb.com/android/building-zero-protocol-for-fast-secure-mobile-connections/

[40]

K. Nepomuceno, I. N. d. Oliveira, R. R. Aschoff, D. Bezerra, M. S. Ito, W. Melo, D. Sadok, and G. Szabó. 2018. QUIC and TCP: A Performance Evaluation. In 2018 IEEE Symposium on Computers and Communications (ISCC). 00045–00051.

[41]

Minh Nguyen, Hadi Amirpour, Christian Timmerer, and Hermann Hellwagner. 2020. Scalable High Efficiency Video Coding Based HTTP Adaptive Streaming over QUIC. In Proceedings of the Workshop on the Evolution, Performance, and Interoperability of QUIC(Virtual Event, USA) (EPIQ ’20). Association for Computing Machinery, New York, NY, USA, 28–34. https://doi.org/10.1145/3405796.3405829

Digital Library

[42]

Kazuho Oku and Jana Iyengar. 2020. Can QUIC match TCP’s computational efficiency?Retrieved August 4, 2020 from https://www.fastly.com/blog/measuring-quic-vs-tcp-computational-efficiency

[43]

Mohammad Rajiullah, Andra Lutu, Ali Safari Khatouni, Mah-Rukh Fida, Marco Mellia, Anna Brunstrom, Ozgu Alay, Stefan Alfredsson, and Vincenzo Mancuso. 2019. Web Experience in Mobile Networks: Lessons from Two Million Page Visits. In The World Wide Web Conference (San Francisco, CA, USA) (WWW ’19). Association for Computing Machinery, New York, NY, USA, 1532–1543. https://doi.org/10.1145/3308558.3313606

Digital Library

[44]

Jan Rüth, Konrad Wolsing, Klaus Wehrle, and Oliver Hohlfeld. 2019. Perceiving QUIC: Do Users Notice or Even Care?. In Proceedings of the 15th International Conference on Emerging Networking Experiments And Technologies (Orlando, Florida) (CoNEXT ’19). Association for Computing Machinery, New York, NY, USA, 144–150. https://doi.org/10.1145/3359989.3365416

Digital Library

[45]

Darius Saif, Chung-Horng Lung, and Ashraf Matrawy. 2020. An Early Benchmark of Quality of Experience Between HTTP/2 and HTTP/3 using Lighthouse. arxiv:2004.01978 [cs.NI]

[46]

Marten Seemann and Jana Iyengar. 2020. Automating QUIC Interoperability Testing. In Proceedings of the Workshop on the Evolution, Performance, and Interoperability of QUIC (Virtual Event, USA) (EPIQ ’20). Association for Computing Machinery, New York, NY, USA, 8–13. https://doi.org/10.1145/3405796.3405826

Digital Library

[47]

Daniel Stenberg. 2019. Comparison with HTTP/2. Retrieved November 26, 2020 from https://http3-explained.haxx.se/en/h3/h3-h2

[48]

Daniel Stenberg. 2020. QUIC WITH WOLFSSL. Retrieved September 19, 2020 from https://daniel.haxx.se/blog/2020/06/18/quic-with-wolfssl/

[49]

Ian Swett. 2019. From gQUIC to IETF QUIC and Beyond. Retrieved February 4, 2021 from https://mile-high.video/files/mhv2019/pdf/day1/1_11_Swett.pdf

[50]

Sreeni Tellakula. 2020. Comparing HTTP/3 vs. HTTP/2 Performance. Retrieved August 5, 2020 from https://blog.cloudflare.com/http-3-vs-http-2/

[51]

Jack Wallen. 2018. How to enable TCP BBR to improve network speed on Linux. Retrieved October 11, 2020 from https://www.techrepublic.com/article/how-to-enable-tcp-bbr-to-improve-network-speed-on-linux/

[52]

Alex Yu. 2020. Benchmarking QUIC. Retrieved September 15, 2020 from https://medium.com/@the.real.yushuf/benchmarking-quic-1fd043e944c7

[53]

Y. Yu, M. Xu, and Y. Yang. 2017. When QUIC meets TCP: An experimental study. In 2017 IEEE 36th International Performance Computing and Communications Conference (IPCCC). 1–8.

Cited By

Lei GWu JGu KJiang FLi SJiang C(2024)A novel anomaly detection model for secure multipath QUIC communications by jointly using empirical mode decomposition and long short-term memory networksIntelligent Decision Technologies10.3233/IDT-230261(1-22)Online publication date: 6-Jan-2024
https://doi.org/10.3233/IDT-230261
Kempf MGauder NJaeger BZirngibl JCarle G(2024)A Quantum of QUIC: Dissecting Cryptography with Post-Quantum Insights2024 IFIP Networking Conference (IFIP Networking)10.23919/IFIPNetworking62109.2024.10619916(195-203)Online publication date: 3-Jun-2024
https://doi.org/10.23919/IFIPNetworking62109.2024.10619916
Bi PZou YXiao MYu DLi YLiu ZXie QCai JKankanhalli MPrabhakaran BBoll SSubramanian RZheng LSingh VCesar PXie LXu D(2024)LiteQUIC: Improving QoE of Video Streams by Reducing CPU Overhead of QUICProceedings of the 32nd ACM International Conference on Multimedia10.1145/3664647.3681670(7918-7927)Online publication date: 28-Oct-2024
https://dl.acm.org/doi/10.1145/3664647.3681670
Show More Cited By

Recommendations

Congestion control for real-time media over QUIC
EPIQ '21: Proceedings of the 2021 Workshop on Evolution, Performance and Interoperability of QUIC

QUIC is the new transport protocol for the Internet, designed for secure, reliable communication, especially with the web in mind. While multimedia streaming, allowing for some playout delay, has been widely run on top of reliable transport protocols, ...
ECN with QUIC: Challenges in the Wild
IMC '23: Proceedings of the 2023 ACM on Internet Measurement Conference

TCP and QUIC can both leverage ECN to avoid congestion loss and its retransmission overhead. However, both protocols require support of their remote endpoints and it took two decades since the initial standardization of ECN for TCP to reach 80% ECN ...
Poster: QUIC is not Quick Enough over Fast Internet
IMC '23: Proceedings of the 2023 ACM on Internet Measurement Conference

QUIC is a multiplexed transport-layer protocol over UDP and comes with enforced encryption. It is expected to be a game-changer in improving web application performance. Together with the network layer and layers below, UDP, QUIC, and HTTP/3 form a new ...

Comments

Please enable JavaScript to view thecomments powered by Disqus.

Information & Contributors

Information

Published In

cover image ACM Conferences

WWW '21: Proceedings of the Web Conference 2021

April 2021

4054 pages

ISBN:9781450383127

DOI:10.1145/3442381

Editors:
Jure Leskovec
Stanford
,
Marko Grobelnik
Jožef Stefan Institute
,
Marc Najork
Google
,
Jie Tang
Tsinghua University
,
Leila Zia
Wikimedia Foundation

Copyright © 2021 ACM.

Permission to make digital or hard copies of all or part of this work for personal or classroom use is granted without fee provided that copies are not made or distributed for profit or commercial advantage and that copies bear this notice and the full citation on the first page. Copyrights for components of this work owned by others than ACM must be honored. Abstracting with credit is permitted. To copy otherwise, or republish, to post on servers or to redistribute to lists, requires prior specific permission and/or a fee. Request permissions from [email protected]

Sponsors

SIGWEB: ACM Special Interest Group on Hypertext, Hypermedia, and Web

Publisher

Association for Computing Machinery

New York, NY, United States

Publication History

Published: 03 June 2021

Permissions

Request permissions for this article.

Request Permissions

Check for updates

Author Tags

Qualifiers

Research-article
Research
Refereed limited

Conference

WWW '21

Sponsor:

SIGWEB

WWW '21: The Web Conference 2021

April 19 - 23, 2021

Ljubljana, Slovenia

Acceptance Rates

Overall Acceptance Rate 1,899 of 8,196 submissions, 23%

Contributors

Other Metrics

View Article Metrics

Bibliometrics & Citations

Bibliometrics

Article Metrics

26
Total Citations
View Citations
868
Total Downloads

Downloads (Last 12 months)212
Downloads (Last 6 weeks)29

Reflects downloads up to 24 Nov 2024

Other Metrics

View Author Metrics

Citations

Cited By

Lei GWu JGu KJiang FLi SJiang C(2024)A novel anomaly detection model for secure multipath QUIC communications by jointly using empirical mode decomposition and long short-term memory networksIntelligent Decision Technologies10.3233/IDT-230261(1-22)Online publication date: 6-Jan-2024
https://doi.org/10.3233/IDT-230261
Kempf MGauder NJaeger BZirngibl JCarle G(2024)A Quantum of QUIC: Dissecting Cryptography with Post-Quantum Insights2024 IFIP Networking Conference (IFIP Networking)10.23919/IFIPNetworking62109.2024.10619916(195-203)Online publication date: 3-Jun-2024
https://doi.org/10.23919/IFIPNetworking62109.2024.10619916
Bi PZou YXiao MYu DLi YLiu ZXie QCai JKankanhalli MPrabhakaran BBoll SSubramanian RZheng LSingh VCesar PXie LXu D(2024)LiteQUIC: Improving QoE of Video Streams by Reducing CPU Overhead of QUICProceedings of the 32nd ACM International Conference on Multimedia10.1145/3664647.3681670(7918-7927)Online publication date: 28-Oct-2024
https://dl.acm.org/doi/10.1145/3664647.3681670
Mücke JNawrocki MHiesgen RSchmidt TWählisch MVallina-Rodríguez NSuarez-Tángil GLevin DPelsser C(2024)ReACKed QUICer: Measuring the Performance of Instant Acknowledgments in QUIC HandshakesProceedings of the 2024 ACM on Internet Measurement Conference10.1145/3646547.3689022(389-400)Online publication date: 4-Nov-2024
https://dl.acm.org/doi/10.1145/3646547.3689022
Muthuraj NEghbal NLu PVallina-Rodríguez NSuarez-Tángil GLevin DPelsser C(2024)Replication: "Taking a long look at QUIC"Proceedings of the 2024 ACM on Internet Measurement Conference10.1145/3646547.3688453(375-388)Online publication date: 4-Nov-2024
https://dl.acm.org/doi/10.1145/3646547.3688453
Zhang XJin SHe YHassan AMao ZQian FZhang ZChua TNgo CKa-Wei Lee RKumar RLauw H(2024)QUIC is not Quick Enough over Fast InternetProceedings of the ACM Web Conference 202410.1145/3589334.3645323(2713-2722)Online publication date: 13-May-2024
https://dl.acm.org/doi/10.1145/3589334.3645323
Gupta ABartos R(2024)Improving HTTP/3 Quality of Experience with Incremental EPS2024 IEEE International Mediterranean Conference on Communications and Networking (MeditCom)10.1109/MeditCom61057.2024.10621102(365-370)Online publication date: 8-Jul-2024
https://doi.org/10.1109/MeditCom61057.2024.10621102
Zhou MChen YLin SWang XLiu BDing A(2024)Dissecting the Applicability of HTTP/3 in Content Delivery Networks2024 IEEE 44th International Conference on Distributed Computing Systems (ICDCS)10.1109/ICDCS60910.2024.00091(936-946)Online publication date: 23-Jul-2024
https://doi.org/10.1109/ICDCS60910.2024.00091
Gupta ABartos R(2024)Improving Web Content Delivery with HTTP/3 and Non-Incremental EPS2024 33rd International Conference on Computer Communications and Networks (ICCCN)10.1109/ICCCN61486.2024.10637641(1-9)Online publication date: 29-Jul-2024
https://doi.org/10.1109/ICCCN61486.2024.10637641
Gupta ABartos R(2024)Enhancing QoE in HTTP/3 Using EPS Framework2024 International Conference on Computer, Information and Telecommunication Systems (CITS)10.1109/CITS61189.2024.10607991(1-8)Online publication date: 17-Jul-2024
https://doi.org/10.1109/CITS61189.2024.10607991
Show More Cited By

View Options

Login options

Check if you have access through your login credentials or your institution to get full access on this article.

Full Access

Get this Publication

View options

PDF

View or Download as a PDF file.

eReader

View online with eReader.

HTML Format

View this article in HTML Format.

Media

Figures

Other

Tables

View Table of Contents