research-article

A scalable multi-datacenter layer-2 network architecture

Authors:

Ling DingAuthors Info & Claims

SOSR '15: Proceedings of the 1st ACM SIGCOMM Symposium on Software Defined Networking Research

Article No.: 8, Pages 1 - 12

https://doi.org/10.1145/2774993.2775008

Published: 17 June 2015 Publication History

Abstract

Cloud today is evolving towards multi-datacenter deployment, with each datacenter serving customers in different geographical areas. The independence between datacenters, however, prohibits effective inter-datacenter resource sharing and flexible management of the infrastructure. In this paper, we propose WL2, a Software-Defined Networking (SDN) solution to an Internet-scale Layer-2 network across multiple datacenters. In WL2, a logically centralized controller handles control-plane communication and configuration in each datacenter. We achieve scalability in three ways: (1) eliminating Layer-2 broadcast by rerouting control-plane traffic to the controller; (2) introducing a layered addressing scheme for aggregate Layer-2 routing; and (3) creating an overlay abstraction on top of physical topology for fast flow setup. WL2 is fault-tolerant against controller and gateway failures. We deployed and evaluated WL2 in a 2,250-VM testbed across three datacenters. The results indicate high performance and robustness of the system.

References

[1]

Amazon Web Services. http://aws.amazon.com/about-aws/global-infrastructure/.

[2]

Build SDN Agilely: COMPONENT-BASED SOFTWARE DEFINED NETWORKING FRAMEWORK. http://osrg.github.io/ryu/.

[3]

ConnectX-3 Pro - Mellanox. https://www.mellanox.com/related-docs/prod_adapter_cards/PB_ConnectX-3_Pro_Card_EN.pdf.

[4]

Docker - Build, Ship, and Run Any App, Anywhere. https://www.docker.com/.

[5]

Generic Routing Encapsulation (GRE). http://www.ietf.org/rfc/rfc2784.txt.

[6]

Google Compute Engine. https://developers.google.com/compute/.

[7]

Google Compute Engine. https://cloud.google.com/compute/docs/robustsystems.

[8]

Keepalived. http://www.keepalived.org/.

[9]

Overlay Transport Virtualization (OTV) - Cisco. http://www.cisco.com/c/en/us/solutions/data-center-virtualization/overlay-transport-virtualization-otv/index. html.

[10]

RFC 3768 - Virtual Router Redundancy Protocol (VRRP). http://tools.ietf.org/html/rfc3768.

[11]

VXLAN A Framework for Overlaying Virtualized Layer 2 Networks over Layer 3 Networks. http://tools.ietf. org/html/draft-mahalingam-dutt-dcops-vxlan-03.

[12]

M. Al-Fares, A. Loukissas, and A. Vahdat. A scalable, commodity data center network architecture. In Proceedings of the ACM SIGCOMM 2008 Conference on Applications, Technologies, Architectures, and Protocols for Computer Communications, Seattle, WA, USA, August 17--22, 2008, pages 63--74, 2008.

Digital Library

[13]

Amazon Elastic Compute Cloud (EC2). http://aws.amazon.com/ec2/.

[14]

P. Barham, B. Dragovic, K. Fraser, S. Hand, T. Harris, A. Ho, R. Neugebauer, I. Pratt, and A. Warfield. Xen and the art of virtualization. In Proceedings of the nineteenth ACM symposium on Operating systems principles, 2003.

Digital Library

[15]

M. Ben-Yehuda, M. D. Day, Z. Dubitzky, M. Factor, N. Har'El, A. Gordon, A. Liguori, O. Wasserman, and B.-A. Yassour. The turtles project: design and implementation of nested virtualization. In OSDI, 2010.

Digital Library

[16]

R. Bradford, E. Kotsovinos, A. Feldmann, and H. Schiöberg. Live wide-area migration of virtual machines including local persistent state. In Proceedings of the 3rd International Conference on Virtual Execution Environments, 2007.

Digital Library

[17]

C. Clark, K. Fraser, S. Hand, J. G. Hansen, E. Jul, C. Limpach, I. Pratt, and A. Warfield. Live migration of virtual machines. In 2nd Symposium on Networked Systems Design and Implementation (NSDI 2005), May 2--4, 2005, Boston, Massachusetts, USA, Proceedings., 2005.

Digital Library

[18]

A. Greenberg, J. R. Hamilton, N. Jain, S. Kandula, C. Kim, P. Lahiri, D. A. Maltz, P. Patel, and S. Sengupta. Vl2: A scalable and flexible data center network. In SIGCOMM, 2009.

Digital Library

[19]

P. Hunt, M. Konar, F. P. Junqueira, and B. Reed. ZooKeeper: wait-free coordination for internet-scale systems. In USENIX Annual Technical Conference, 2010.

Digital Library

[20]

S. Jain, A. Kumar, S. Mandal, J. Ong, L. Poutievski, A. Singh, S. Venkata, J. Wanderer, J. Zhou, M. Zhu, J. Zolla, U. Hölzle, S. Stuart, and A. Vahdat. B4: Experience with a globally-deployed software defined wan. In SIGCOMM, 2013.

Digital Library

[21]

C. Kim, M. Caesar, and J. Rexford. Floodless in seattle: a scalable ethernet architecture for large enterprises. In SIGCOMM, 2008.

Digital Library

[22]

T. Koponen, K. Amidon, P. Balland, M. Casado, A. Chanda, B. Fulton, I. Ganichev, J. Gross, N. Gude, P. Ingram, E. Jackson, A. Lambeth, R. Lenglet, S.-H. Li, A. Padmanabhan, J. Pettit, B. Pfaff, R. Ramanathan, S. Shenker, A. Shieh, J. Stribling, P. Thakkar, D. Wendlandt, A. Yip, and R. Zhang. Network virtualization in multi-tenant datacenters. In Proceedings of the 11th USENIX Conference on Networked Systems Design and Implementation, NSDI'14, pages 203--216, Berkeley, CA, USA, 2014. USENIX Association.

Digital Library

[23]

T. Koponen, M. Casado, N. Gude, J. Stribling, L. Poutievski, M. Zhu, R. Ramanathan, Y. Iwata, H. Inoue, T. Hama, and S. Shenker. Onix: a distributed control platform for large-scale production networks. In OSDI, 2010.

Digital Library

[24]

L. Lamport. The part-time parliament. ACM Trans. Comput. Syst., 16(2): 133--169, May 1998.

Digital Library

[25]

C. Liu and Y. Mao. Inception: Towards a Nested Cloud Architecture. In 5th USENIX Workshop on Hot Topics in Cloud Computing (HotCloud), 2013.

[26]

N. McKeown, T. Anderson, H. Balakrishnan, G. Parulkar, L. Peterson, J. Rexford, S. Shenker, and J. Turner. Openflow: Enabling innovation in campus networks. SIGCOMM Comput. Commun. Rev., 38(2): 69--74, March 2008.

Digital Library

[27]

R. Mehta. VXLAN Performance Evaluation on VMware vSphere 5.1. Technical report, VMware, 2013.

[28]

Azure: Microsoft's Cloud Platform. http://azure.microsoft.com/.

[29]

A. Myers, T. S. E. Ng, and H. Zhang. Rethinking the Service Model: Scaling Ethernet to a Million Nodes. In HotNets, 2004OA.

[30]

R. Niranjan Mysore, A. Pamboris, N. Farrington, N. Huang, P. Miri, S. Radhakrishnan, V. Subramanya, and A. Vahdat. Portland: A scalable fault-tolerant layer 2 data center network fabric. In SIGCOMM, 2009.

Digital Library

[31]

OpenStack: Open Source Cloud Computing Software. http://openstack.org/.

[32]

B. Pfaff, J. Pettit, K. Amidon, M. Casado, T. Koponen, and S. Shenker. Extending networking into the virtualization layer. In HotNets, 2009.

[33]

B. Stephens, A. Cox, W. Felter, C. Dixon, and J. Carter. Past: Scalable ethernet for data centers. In Proceedings of the 8th International Conference on Emerging Networking Experiments and Technologies, CoNEXT '12, pages 49--60, New York, NY, USA, 2012. ACM.

Digital Library

[34]

A. Tavakoli, M. Casado, T. Koponen, and S. Shenker. Applying NOX to the datacenter. In Eight ACM Workshop on Hot Topics in Networks (HotNets-VIII), HOTNETS '09, New York City, 2009.

[35]

RFC 6326 - Transparent Interconnection of Lots of Links (TRILL) Use of IS-IS. http://tools.ietf.org/html/rfc6326, 2011.

Cited By

Daneshmand VSubratie KFigueiredo R(2024)PolyNet: Cost- and Performance-Aware Multi-Criteria Link Selection in Software-Defined Edge-to-Cloud Overlay Networks2024 IEEE 10th International Conference on Network Softwarization (NetSoft)10.1109/NetSoft60951.2024.10588920(127-135)Online publication date: 24-Jun-2024
https://doi.org/10.1109/NetSoft60951.2024.10588920
Shubbar RAlhisnawi MAbdulhassan AAhamdi M(2021)A Comprehensive Survey on Software-Defined Network ControllersNext Generation of Internet of Things10.1007/978-981-16-0666-3_18(199-231)Online publication date: 15-Jun-2021
https://doi.org/10.1007/978-981-16-0666-3_18
SUBRATIE KADITYA SDANESHMAND VICHIKAWA KFIGUEIREDO R(2020)On the Design and Implementation of IP-over-P2P Overlay Virtual Private NetworksIEICE Transactions on Communications10.1587/transcom.2019CPI0001E103.B:1(2-10)Online publication date: 1-Jan-2020
https://doi.org/10.1587/transcom.2019CPI0001
Show More Cited By

Index Terms

A scalable multi-datacenter layer-2 network architecture
1. Networks
  1. Network protocols
  2. Network services
    1. Network management

Recommendations

Scaling IP multicast on datacenter topologies
CoNEXT '13: Proceedings of the ninth ACM conference on Emerging networking experiments and technologies

IP multicast would reduce significantly both network and server overhead for many datacenter applications' communication. Unfortunately, traditional protocols for managing IP multicast, designed for arbitrary network topologies, do not scale with ...
A Software-Defined Scalable and Autonomous Architecture for Multi-tenancy
IC2E '14: Proceedings of the 2014 IEEE International Conference on Cloud Engineering

Scalability for distributed Data Center Networks (DCNs) has long been a goal of the network research and industrial community. To support dynamically increasing demands from multi-tenants, the network providers have to duplicate or share virtual ...
Live migration of an entire network (and its hosts)
HotNets-XI: Proceedings of the 11th ACM Workshop on Hot Topics in Networks

Live virtual machine (VM) migration can move applications from one location to another without a disruption in service. However, applications often consist of multiple VMs and rely on the state of the underlying network for basic reachability, access ...

Comments

Please enable JavaScript to view thecomments powered by Disqus.

Information & Contributors

Information

Published In

cover image ACM Conferences

SOSR '15: Proceedings of the 1st ACM SIGCOMM Symposium on Software Defined Networking Research

June 2015

226 pages

ISBN:9781450334518

DOI:10.1145/2774993

Program Chairs:
Jennifer Rexford
Princeton
,
Amin Vahdat
Google

Copyright © 2015 ACM.

Permission to make digital or hard copies of all or part of this work for personal or classroom use is granted without fee provided that copies are not made or distributed for profit or commercial advantage and that copies bear this notice and the full citation on the first page. Copyrights for components of this work owned by others than ACM must be honored. Abstracting with credit is permitted. To copy otherwise, or republish, to post on servers or to redistribute to lists, requires prior specific permission and/or a fee. Request permissions from [email protected]

Sponsors

SIGCOMM: ACM Special Interest Group on Data Communication
ONS: Open Networking Summit

In-Cooperation

USENIX Assoc: USENIX Assoc

Publisher

Association for Computing Machinery

New York, NY, United States

Publication History

Published: 17 June 2015

Permissions

Request permissions for this article.

Request Permissions

Check for updates

Author Tags

Qualifiers

Research-article

Funding Sources

AT&T
AFOSR
NSF

Conference

SOSR 2015

Sponsor:

SIGCOMM
ONS

SOSR 2015: ACM SIGCOMM Symposium on SDN Research

June 17 - 18, 2015

California, Santa Clara

Acceptance Rates

SOSR '15 Paper Acceptance Rate 7 of 43 submissions, 16%;

Overall Acceptance Rate 7 of 43 submissions, 16%

Contributors

Other Metrics

View Article Metrics

Bibliometrics & Citations

Bibliometrics

Article Metrics

14
Total Citations
View Citations
631
Total Downloads

Downloads (Last 12 months)9
Downloads (Last 6 weeks)0

Reflects downloads up to 14 Feb 2025

Other Metrics

View Author Metrics

Citations

Cited By

Daneshmand VSubratie KFigueiredo R(2024)PolyNet: Cost- and Performance-Aware Multi-Criteria Link Selection in Software-Defined Edge-to-Cloud Overlay Networks2024 IEEE 10th International Conference on Network Softwarization (NetSoft)10.1109/NetSoft60951.2024.10588920(127-135)Online publication date: 24-Jun-2024
https://doi.org/10.1109/NetSoft60951.2024.10588920
Shubbar RAlhisnawi MAbdulhassan AAhamdi M(2021)A Comprehensive Survey on Software-Defined Network ControllersNext Generation of Internet of Things10.1007/978-981-16-0666-3_18(199-231)Online publication date: 15-Jun-2021
https://doi.org/10.1007/978-981-16-0666-3_18
SUBRATIE KADITYA SDANESHMAND VICHIKAWA KFIGUEIREDO R(2020)On the Design and Implementation of IP-over-P2P Overlay Virtual Private NetworksIEICE Transactions on Communications10.1587/transcom.2019CPI0001E103.B:1(2-10)Online publication date: 1-Jan-2020
https://doi.org/10.1587/transcom.2019CPI0001
Li QHuang NJiang YSinnott RXu M(2020)Scale the Data Plane of Software-Defined Networks: a Lazy Rule Placement Approach2020 IEEE 40th International Conference on Distributed Computing Systems (ICDCS)10.1109/ICDCS47774.2020.00077(366-376)Online publication date: Nov-2020
https://doi.org/10.1109/ICDCS47774.2020.00077
Xie JGuo DQian CLiu LRen BChen H(2019)Validation of Distributed SDN Control Plane Under Uncertain FailuresIEEE/ACM Transactions on Networking10.1109/TNET.2019.291412227:3(1234-1247)Online publication date: 1-Jun-2019
https://dl.acm.org/doi/10.1109/TNET.2019.2914122
Sun JZhang NWo TLiu X(2019)HCNet: An SDN Enabled Virtual Network Management System for Hybrid Clouds2019 IEEE Symposium on Computers and Communications (ISCC)10.1109/ISCC47284.2019.8969569(1-7)Online publication date: Jun-2019
https://doi.org/10.1109/ISCC47284.2019.8969569
Wang YLin Y(2019)Circuit‐based logical layer 2 bridging in software‐defined data center networkingInternational Journal of Communication Systems10.1002/dac.412832:16Online publication date: 5-Aug-2019
https://doi.org/10.1002/dac.4128
Comer DKarandikar R(2018)DCnet: a data centre network architecture that supports live VM migrationIET Networks10.1049/iet-net.2018.5094Online publication date: 25-Oct-2018
https://doi.org/10.1049/iet-net.2018.5094
Wang YLin YChang G(2018)SDN‐based dynamic multipath forwarding for inter–data center networkingInternational Journal of Communication Systems10.1002/dac.384332:1Online publication date: 25-Oct-2018
https://doi.org/10.1002/dac.3843
Goumas GNikas KLakew EKotselidis CAttwood AElmroth EFlouris MFoutris NGoodacre JGrohmann DKarakostas VKoutsourakis PKersten MLujan MRustad EThomson JTomas LVesterkjaer AWebber JZhang YKoziris N(2017)ACTiCLOUD: Enabling the Next Generation of Cloud Applications2017 IEEE 37th International Conference on Distributed Computing Systems (ICDCS)10.1109/ICDCS.2017.252(1836-1845)Online publication date: Jun-2017
https://doi.org/10.1109/ICDCS.2017.252
Show More Cited By

View Options

Login options

Check if you have access through your login credentials or your institution to get full access on this article.

Full Access

Get this Publication

View options

PDF

View or Download as a PDF file.

eReader

View online with eReader.

Figures

Tables

Media

View Table of Conten