research-article

B4: experience with a globally-deployed software defined wan

Authors:

Subhasree Mandal,

Leon Poutievski,

Subbaiah Venkata,

Stephen Stuart,

Amin VahdatAuthors Info & Claims

ACM SIGCOMM Computer Communication Review, Volume 43, Issue 4

Pages 3 - 14

https://doi.org/10.1145/2534169.2486019

Published: 27 August 2013 Publication History

Abstract

We present the design, implementation, and evaluation of B4, a private WAN connecting Google's data centers across the planet. B4 has a number of unique characteristics: i) massive bandwidth requirements deployed to a modest number of sites, ii) elastic traffic demand that seeks to maximize average bandwidth, and iii) full control over the edge servers and network, which enables rate limiting and demand measurement at the edge.

These characteristics led to a Software Defined Networking architecture using OpenFlow to control relatively simple switches built from merchant silicon. B4's centralized traffic engineering service drives links to near 100% utilization, while splitting application flows among multiple paths to balance capacity against application priority/demands. We describe experience with three years of B4 production deployment, lessons learned, and areas for future work.

References

[1]

Al-Fares, M., Loukissas, A., and Vahdat, A. A Scalable, Commodity Data Center Network Architecture. In Proc. SIGCOMM (New York, NY, USA, 2008), ACM.

Digital Library

[2]

Allalouf, M., and Shavitt, Y. Centralized and Distributed Algorithms for Routing and Weighted Max-Min Fair Bandwidth Allocation. IEEE/ACM Trans. Networking 16, 5 (2008), 1015--1024.

Digital Library

[3]

Aukia, P., Kodialam, M., Koppol, P. V., Lakshman, T. V., Sarin, H., and Suter, B. RATES: A Server for MPLS Traffic Engineering. IEEE Network Magazine 14, 2 (March 2000), 34--41.

Digital Library

[4]

Awduche, D., Berger, L., Gan, D., Li, T., Srinivasan, V., and Swallow, G. RSVP-TE: Extensions to RSVP for LSP Tunnels. RFC 3209, IETF, United States, 2001.

Digital Library

[5]

Awduche, D., Malcolm, J., Agogbua, J., O'Dell, M., and McManus, J. Requirements for Traffic Engineering Over MPLS. RFC 2702, IETF, 1999.

Digital Library

[6]

Caesar, M., Caldwell, D., Feamster, N., Rexford, J., Shaikh, A., and van der Merwe, K. Design and Implementation of a Routing Control Platform. In Proc. of NSDI (April 2005).

Digital Library

[7]

Casado, M., Freedman, M. J., Pettit, J., Luo, J., McKeown, N., and Shenker, S. Ethane: Taking Control of the Enterprise. In Proc. SIGCOMM (August 2007).

Digital Library

[8]

Casado, M., Garfinkel, T., Akella, A., Freedman, M. J., Boneh, D., McKeown, N., and Shenker, S. SANE: A Protection Architecture for Enterprise Networks. In Proc. of Usenix Security (August 2006).

Digital Library

[9]

Chandra, T. D., Griesemer, R., and Redstone, J. Paxos Made Live: an Engineering Perspective. In Proc. of the ACM Symposium on Principles of Distributed Computing (New York, NY, USA, 2007), ACM, pp. 398--407.

Digital Library

[10]

Choi, T., Yoon, S., Chung, H., Kim, C., Park, J., Lee, B., and Jeong, T. Design and Implementation of Traffic Engineering Server for a Large-Scale MPLS-Based IP Network. In Revised Papers from the International Conference on Information Networking, Wireless Communications Technologies and Network Applications-Part I (London, UK, UK, 2002), ICOIN '02, Springer-Verlag, pp. 699--711.

Digital Library

[11]

Curtis, A. R., Mogul, J. C., Tourrilhes, J., Yalagandula, P., Sharma, P., and Banerjee, S. DevoFlow: Scaling Flow Management for High-Performance Networks. In Proc. SIGCOMM (2011), pp. 254--265.

Digital Library

[12]

Danna, E., Hassidim, A., Kaplan, H., Kumar, A., Mansour, Y., Raz, D., and Segalov, M. Upward Max Min Fairness. In INFOCOM (2012), pp. 837--845.

[13]

Danna, E., Mandal, S., and Singh, A. A Practical Algorithm for Balancing the Max-min Fairness and Throughput Objectives in Traffic Engineering. In Proc. INFOCOM (March 2012), pp. 846--854.

[14]

Elwalid, A., Jin, C., Low, S., and Widjaja, I. MATE: MPLS Adaptive Traffic Engineering. In Proc. IEEE INFOCOM (2001), pp. 1300--1309.

[15]

Farrington, N., Rubow, E., and Vahdat, A. Data Center Switch Architecture in the Age of Merchant Silicon. In Proc. Hot Interconnects (August 2009), IEEE, pp. 93--102.

Digital Library

[16]

Fortz, B., Rexford, J., and Thorup, M. Traffic Engineering with Traditional IP Routing Protocols. IEEE Communications Magazine 40 (2002), 118--124.

Digital Library

[17]

Fortz, B., and Thorup, M. Increasing Internet Capacity Using Local Search. Comput. Optim. Appl. 29, 1 (October 2004), 13--48.

Digital Library

[18]

Greenberg, A., Hamilton, J. R., Jain, N., Kandula, S., Kim, C., Lahiri, P., Maltz, D. A., Patel, P., and Sengupta, S. VL2: A Scalable and Flexible Data Center Network. In Proc. SIGCOMM (August 2009).

Digital Library

[19]

Greenberg, A., Hjalmtysson, G., Maltz, D. A., Myers, A., Rexford, J., Xie, G., Yan, H., Zhan, J., and Zhang, H. A Clean Slate 4D Approach to Network Control and Management. SIGCOMM CCR 35, 5 (2005), 41--54.

Digital Library

[20]

Greenberg, A., Lahiri, P., Maltz, D. A., Patel, P., and Sengupta, S. Towards a Next Generation Data Center Architecture: Scalability and Commoditization. In Proc. ACM workshop on Programmable Routers for Extensible Services of Tomorrow (2008), pp. 57--62.

Digital Library

[21]

Gude, N., Koponen, T., Pettit, J., Pfaff, B., Casado, M., McKeown, N., and Shenker, S. NOX: Towards an Operating System for Networks. In SIGCOMM CCR (July 2008).

Digital Library

[22]

He, J., and Rexford, J. Toward Internet-wide Multipath Routing. IEEE Network Magazine 22, 2 (March 2008), 16--21.

Digital Library

[23]

Hong, C.-Y., Kandula, S., Mahajan, R., Zhang, M., Gill, V., Nanduri, M., and Wattenhofer, R. Have Your Network and Use It Fully Too: Achieving High Utilization in Inter-Datacenter WANs. In Proc. SIGCOMM (August 2013).

Digital Library

[24]

Kandula, S., Katabi, D., Davie, B., and Charny, A. Walking the Tightrope: Responsive Yet Stable Traffic Engineering. In Proc. SIGCOMM (August 2005).

Digital Library

[25]

Kipp, S. Bandwidth Growth and the Next Speed of Ethernet. Proc. North American Network Operators Group (October 2012).

[26]

Koponen, T., Casado, M., Gude, N., Stribling, J., Poutievski, L., Zhu, M., Ramanathan, R., Iwata, Y., Inoue, H., Hama, T., and Shenker, S. Onix: a Distributed Control Platform for Large-scale Production Networks. In Proc. OSDI (2010), pp. 1--6.

Digital Library

[27]

Lakshman, T., Nandagopal, T., Ramjee, R., Sabnani, K., and Woo, T. The Softrouter Architecture. In Proc. HotNets (November 2004).

[28]

McKeown, N., Anderson, T., Balakrishnan, H., Parulkar, G., Peterson, L., Rexford, J., Shenker, S., and Turner, J. OpenFlow: Enabling Innovation in Campus Networks. SIGCOMM CCR 38, 2 (2008), 69--74.

Digital Library

[29]

Medina, A., Taft, N., Salamatian, K., Bhattacharyya, S., and Diot, C. Traffic Matrix Estimation: Existing Techniques and New Directions. In Proc. SIGCOMM (New York, NY, USA, 2002), ACM, pp. 161--174.

Digital Library

[30]

Nascimento, M. R., Rothenberg, C. E., Salvador, M. R., and Magalh\ aes, M. F. QuagFlow: Partnering Quagga with OpenFlow (Poster). In Proc. SIGCOMM (2010), pp. 441--442.

Digital Library

[31]

OpenFlow Specification. http://www.openflow.org/wp/documents/.

[32]

Rothenberg, C. E., Nascimento, M. R., Salvador, M. R., Corrêa, C. N. A., Cunha de Lucena, S., and Raszuk, R. Revisiting Routing Control Platforms with the Eyes and Muscles of Software-defined Networking. In Proc. HotSDN (2012), pp. 13--18.

Digital Library

[33]

Roughan, M., Thorup, M., and Zhang, Y. Traffic Engineering with Estimated Traffic Matrices. In Proc. IMC (2003), pp. 248--258.

Digital Library

[34]

Scoglio, C., Anjali, T., de Oliveira, J. C., Akyildiz, I. F., and UhI, G. TEAM: A Traffic Engineering Automated Manager for DiffServ-based MPLS Networks. Comm. Mag. 42, 10 (October 2004), 134--145.

Digital Library

[35]

Sherwood, R., Gibb, G., Yap, K.-K., Appenzeller, G., Casado, M., McKeown, N., and Parulkar, G. FlowVisor: A Network Virtualization Layer. Tech. Rep. OPENFLOW-TR-2009--1, OpenFlow, October 2009.

[36]

Suchara, M., Xu, D., Doverspike, R., Johnson, D., and Rexford, J. Network Architecture for Joint Failure Recovery and Traffic Engineering. In Proc. ACM SIGMETRICS (2011), pp. 97--108.

Digital Library

[37]

Thaler, D. Multipath Issues in Unicast and Multicast Next-Hop Selection. RFC 2991, IETF, 2000.

Digital Library

[38]

Wang, H., Xie, H., Qiu, L., Yang, Y. R., Zhang, Y., and Greenberg, A. COPE: Traffic Engineering in Dynamic Networks. In Proc. SIGCOMM (2006), pp. 99--110.

Digital Library

[39]

Xu, D., Chiang, M., and Rexford, J. Link-state Routing with Hop-by-hop Forwarding Can Achieve Optimal Traffic Engineering. IEEE/ACM Trans. Netw. 19, 6 (December 2011), 1717--1730.

Digital Library

[40]

Yu, M., Rexford, J., Freedman, M. J., and Wang, J. Scalable flow-based networking with DIFANE. In Proc. SIGCOMM (2010), pp. 351--362.

Digital Library

Cited By

Dinh LQuang PLeguay J(2025)Safe load balancing in software-defined-networkingComputer Communications10.1016/j.comcom.2024.107985229(107985)Online publication date: Jan-2025
https://doi.org/10.1016/j.comcom.2024.107985
Deo KChaudhary KAssaf M(2024)Adaptive quality of service for packet loss reduction using OpenFlow metersPeerJ Computer Science10.7717/peerj-cs.184810(e1848)Online publication date: 4-Apr-2024
https://doi.org/10.7717/peerj-cs.1848
V. Oikonomou NV. Oikonomou DStergiou ELiarokapis D(2024)Comprehensive Analysis of Software-Defined Networking: Evaluating Performance Across Diverse Topologies and Investigating Topology Discovery ProtocolsJournal of Engineering Research and Sciences10.55708/js03070033:7(23-43)Online publication date: Jul-2024
https://doi.org/10.55708/js0307003
Show More Cited By

Index Terms

B4: experience with a globally-deployed software defined wan
1. Computer systems organization
  1. Architectures
    1. Distributed architectures
2. Networks

Recommendations

B4: experience with a globally-deployed software defined wan
SIGCOMM '13: Proceedings of the ACM SIGCOMM 2013 conference on SIGCOMM

We present the design, implementation, and evaluation of B4, a private WAN connecting Google's data centers across the planet. B4 has a number of unique characteristics: i) massive bandwidth requirements deployed to a modest number of sites, ii) elastic ...
A roadmap for traffic engineering in SDN-OpenFlow networks

Software Defined Networking (SDN) is an emerging networking paradigm that separates the network control plane from the data forwarding plane with the promise to dramatically improve network resource utilization, simplify network management, reduce ...
On the multiplexing effects of the ATM traffic
ISCC '95: Proceedings of the IEEE Symposium on Computers and Communications (ISCC'95)

Traffic measurements taken from real data are highly desirable for validating traffic models, designing congestion control and flow control algorithms, implementing communication protocols, and developing switch architectures. This paper investigates ...

Comments

Please enable JavaScript to view thecomments powered by Disqus.

Information & Contributors

Information

Published In

cover image ACM SIGCOMM Computer Communication Review

ACM SIGCOMM Computer Communication Review Volume 43, Issue 4

October 2013

595 pages

ISSN:0146-4833

DOI:10.1145/2534169

Issue’s Table of Contents

SIGCOMM '13: Proceedings of the ACM SIGCOMM 2013 conference on SIGCOMM
August 2013
580 pages
ISBN:9781450320566
DOI:10.1145/2486001
General Chairs:
Dah Ming Chiu
Chinese University of Hong Kong, China
,
Jia Wang
AT&T Labs - Research, USA
,
Program Chairs:
Paul Barford
University of Wisconsin, USA
,
Srinivasan Seshan
Carnegie Mellon University, USA

Copyright © 2013 ACM.

Permission to make digital or hard copies of all or part of this work for personal or classroom use is granted without fee provided that copies are not made or distributed for profit or commercial advantage and that copies bear this notice and the full citation on the first page. Copyrights for components of this work owned by others than ACM must be honored. Abstracting with credit is permitted. To copy otherwise, or republish, to post on servers or to redistribute to lists, requires prior specific permission and/or a fee. Request permissions from [email protected]

Publisher

Association for Computing Machinery

New York, NY, United States

Publication History

Published: 27 August 2013

Published in SIGCOMM-CCR Volume 43, Issue 4

Check for updates

Author Tags

Qualifiers

Research-article

Contributors

Other Metrics

View Article Metrics

Bibliometrics & Citations

Bibliometrics

Article Metrics

1,985
Total Citations
View Citations
13,381
Total Downloads

Downloads (Last 12 months)1,333
Downloads (Last 6 weeks)202

Reflects downloads up to 26 Nov 2024

Other Metrics

View Author Metrics

Citations

Cited By

Dinh LQuang PLeguay J(2025)Safe load balancing in software-defined-networkingComputer Communications10.1016/j.comcom.2024.107985229(107985)Online publication date: Jan-2025
https://doi.org/10.1016/j.comcom.2024.107985
Deo KChaudhary KAssaf M(2024)Adaptive quality of service for packet loss reduction using OpenFlow metersPeerJ Computer Science10.7717/peerj-cs.184810(e1848)Online publication date: 4-Apr-2024
https://doi.org/10.7717/peerj-cs.1848
V. Oikonomou NV. Oikonomou DStergiou ELiarokapis D(2024)Comprehensive Analysis of Software-Defined Networking: Evaluating Performance Across Diverse Topologies and Investigating Topology Discovery ProtocolsJournal of Engineering Research and Sciences10.55708/js03070033:7(23-43)Online publication date: Jul-2024
https://doi.org/10.55708/js0307003
Kumari NKathuria K(2024)Overview of SDN Building Foundations and ApplicationsJournal of Research in Science and Engineering10.53469/jrse.2024.06(07).086:7(43-53)Online publication date: 28-Jul-2024
https://doi.org/10.53469/jrse.2024.06(07).08
Vaghela GSanghani NBorisaniya B(2024)Review on DDoS Attack in Controller Environment of Software Defined NetworkICST Transactions on Scalable Information Systems10.4108/eetsis.582311Online publication date: 24-Jul-2024
https://doi.org/10.4108/eetsis.5823
Kulkarni MGoswami BPaulose JMalakalapalli L(2024)Unlocking the Power of Software-Defined Networking (SDN) in Revolutionizing Network ManagementAdvanced Cyber Security Techniques for Data, Blockchain, IoT, and Network Protection10.4018/979-8-3693-9225-6.ch012(309-336)Online publication date: 4-Oct-2024
https://doi.org/10.4018/979-8-3693-9225-6.ch012
Sharma ABalasubramanian VKamruzzaman J(2024)A Temporal Deep Q Learning for Optimal Load Balancing in Software-Defined NetworksSensors10.3390/s2404121624:4(1216)Online publication date: 14-Feb-2024
https://doi.org/10.3390/s24041216
Stambouli ALogrippo L(2024)Implementation of a Partial-Order Data Security Model for the Internet of Things (IoT) Using Software-Defined Networking (SDN)Journal of Cybersecurity and Privacy10.3390/jcp40300234:3(468-493)Online publication date: 20-Jul-2024
https://doi.org/10.3390/jcp4030023
Fu CWang BWang W(2024)Software-Defined Wide Area Networks (SD-WANs): A SurveyElectronics10.3390/electronics1315301113:15(3011)Online publication date: 30-Jul-2024
https://doi.org/10.3390/electronics13153011
Rzym GDuliński ZChołda P(2024)Dynamic Link Metric Selection for Traffic Aggregation and Multipath Transmission in Software-Defined NetworksApplied Sciences10.3390/app1412531214:12(5312)Online publication date: 19-Jun-2024
https://doi.org/10.3390/app14125312
Show More Cited By

View Options

Login options

Check if you have access through your login credentials or your institution to get full access on this article.

Full Access

Get this Publication

View options

PDF

View or Download as a PDF file.

eReader

View online with eReader.

Media

Figures

Other

Tables

View Issue’s Table of Contents