Article

Nomad: migrating OS-bypass networks in virtual machines

Authors:

Dhabaleswar PandaAuthors Info & Claims

VEE '07: Proceedings of the 3rd international conference on Virtual execution environments

Pages 158 - 168

https://doi.org/10.1145/1254810.1254833

Published: 13 June 2007 Publication History

Abstract

Virtual machine (VM) technology is experiencing a resurgence due to various benefits including ease of management, security and resource consolidation. Live migration of virtual machines allows transparent movement of OS instances and hosted applications across physical machines. It is one of the most useful features of VM technology because it provides a powerful tool for effective administration of modern cluster environments. Migrating network resources is one of the key problems that need to be addressed in the VM migration process. Existing studies of VM migration have focused on traditional I/O interfaces such as Ethernet. However, modern high-speed interconnects with intelligent NICs pose significantly more challenges as they have additional features including hardware level reliable services and direct I/O accesses. In this paper we present Nomad, a design for migrating modern interconnects with the aforementioned features, focusing on cluster environments running VMs. We introduce a thin namespace virtualization layer to efficiently address location dependent resource handles and a handshake protocol which transparently maintains reliable service semantics during migration. We demonstrate our design by implementing a prototype based on the Xen virtual machine monitor and InfiniBand. Our performance analysis shows that Nomad can achieve efficient migration of network resources, even in environments with stringent communication performance requirements.

References

[1]

RFC 2002: Mobile IP. http://www.ietf.org/rfc/rfc2002.txt.

[2]

R. A. F. Bhoedjang, T. Ruhl, and H. E. Bal. User-Level Network Interface Protocols. IEEE Computer, pages 53--60, November 1998.

Digital Library

[3]

M. Blumrich, C. Dubnicki, E. W. Felten, K. Li, and M. R. Mesarina. Virtual-Memory-Mapped Network Interfaces. In IEEE Micro, pages 21--28, Feb. 1995.

Digital Library

[4]

E. Bugnion, S. Devine, K. Govil, and M. Rosenblum. Disco: Running commodity operating systems on scalable multiprocessors. ACM Transactions on Computer Systems, 15(4):412--447, 1997.

Digital Library

[5]

C. Clark et al. Live Migration of Virtual Machines. In Proceedings of 2nd Symposium on Networked Systems Design and Implementation, 2005.

Digital Library

[6]

B. Dragovic, K. Fraser, S. Hand, T. Harris, A. Ho, I. Pratt, A.Warfield, P. Barham, and R. Neugebauer. Xen and the Art of Virtualization. In Proceedings of the ACM Symposium on Operating Systems Principles, pages 164--177, October 2003.

Digital Library

[7]

D. Dunning, G. Regnier, G. McAlpine, D. Cameron, B. Shubert, F. Berry, A. Merritt, E. Gronke, and C. Dodd. The Virtual Interface Architecture. IEEE Micro, pages 66--76, March/April 1998.

Digital Library

[8]

K. Fraser, S. Hand, R. Neugebauer, I. Pratt, A. Warfield, and M. Williamson. Safe hardware access with the xen virtual machine monitor. In OASIS ASPLOS Workshop, 2004.

[9]

K. Govil, D. Teodosiu, Y. Huang, and M. Rosenblum. Cellular disco: resource management using virtual clusters on shared-memory multiprocessors. ACM Transactions on Computer Systems, 18(3):229--262, 2000.

Digital Library

[10]

InfiniBand Trade Association. InfiniBand Architecture Specification.

[11]

K. Koch. How does ASCI Actually Complete Multi-month 1000-processor Milestone Simulations? In Proceedings of the Conference on High Speed Computing, Gleneden Beach, Oregon, 2002.

[12]

J. Liu, W. Huang, B. Abali, and D. K. Panda. High Performance VMM-Bypass I/O in Virtual Machines. In Proceedings of 2006 USENIX Annual Technical Conference, June 2006.

Digital Library

[13]

J. Liu, J. Wu, S. P. Kini, P. Wyckoff, and D. K. Panda. High Performance RDMA-Based MPI Implementation over InfiniBand. In Proceedings of 17th Annual ACM International Conference on Supercomputing (ICS '03), June 2003.

Digital Library

[14]

Mellanox Technologies. http://www.mellanox.com.

[15]

Myricom, Inc. Myrinet. http://www.myri.com.

[16]

NASA. NAS Parallel Benchmarks. http://www.nas.nasa.gov/Software/NPB/.

[17]

Network-Based Computing Laboratory. MVAPICH: MPI for InfiniBand and other RDMA Interconnects. http://nowlab.cse.ohio-state.edu/projects/mpi-iba/index.html.

[18]

OpenFabrics Alliance. http://www.openfabrics.org.

[19]

S. Osman, D. Subhraveti, G. Su, and J. Nieh. The Design and Implementation of Zap: A System for Migrating Computing Environments. In Proceedings of the 5th Symposium on Operating Systems Design and Implementation (OSDI), 2002.

Digital Library

[20]

S. Pakin, M. Lauria, and A. Chien. High Performance Messaging on Workstations: Illinois Fast Messages (FM). In Proceedings of the Supercomputing, 1995.

Digital Library

[21]

F. Petrini, D. J. Kerbyson, and S. Pakin. The Case of the Missing Supercomputer Performance: Achieving Optimal Performance on the 8,192 Processors of ASCI Q. In Proceedings of SC '03, Washington, DC, USA, 2003.

Digital Library

[22]

I. Pratt and K. Fraser. Arsenic: A User-Accessible Gigabit Ethernet Interface. In INFOCOM, pages 67--76, 2001.

[23]

Quadrics, Ltd. QsNet. http://www.quadrics.com.

[24]

M. Rosenblum and T. Garfinkel. Virtual Machine Monitors: Current Technology and Future Trends. IEEE Computer, May 2005.

Digital Library

[25]

S. M. Hand. Self-Paging in the Nemesis Operating System. In Proceedings 3rd OSDI, pages 73--86, 1999.

Digital Library

[26]

M. Snir, S. Otto, S. Huss-Lederman, D. Walker, and J. Dongarra. MPI?The Complete Reference. Volume 1 - The MPI-1 Core, 2nd edition. The MIT Press, 1998.

Digital Library

[27]

A. Z. Spector. Performing remote operations efficiently on a local computer network. Commun. ACM, 25(4):246--260, 1982.

Digital Library

[28]

G. Su and J. Nieh. Mobile Communication with Virtual Network Address Translation. Technical Report CUCS-003-02, Columbia University, Feb 2002.

[29]

J. Sugerman, G. Venkitachalam, and B. H. Lim. Virtualizing I/O Devices on VMware Workstation's Hosted Virtual Machine Monitor. In Proceedings of USENIX, 2001.

Digital Library

[30]

C. A. Thekkath, H. M. Levy, and E. D. Lazowska. Separating Data and Control Transfer in Distributed Operating Systems. In Proceedings of the Sixth International Conference on Architectural Support for Programming Languages and Operating Systems, pages 2--11, San Jose, California, 1994.

Digital Library

[31]

J. S. Vetter and F. Mueller. Communication Characteristics of Large-Scale Scientific Applications for Contemporary Cluster Architectures. In Proceedings of IEEE International Parallel and Distributed Processing Symposium (IPDPS), April 2002.

Digital Library

[32]

T. von Eicken, A. Basu, V. Buch, and W. Vogels. U-Net: A User-level Network Interface for Parallel and Distributed Computing. In ACM Symposium on Operating Systems Principles, 1995.

Digital Library

[33]

T. von Eicken, D. E. Culler, S. C. Goldstein, and K. E. Schauser. Active Messages: A Mechanism for Integrated Communication and Computation. In International Symposium on Computer Architecture, pages 256--266, 1992.

Digital Library

[34]

W. Huang, J. Liu, B. Abali and D. K. Panda. A Case for High Performance Computing with Virtual Machines. In Proceedings of the 20th ACM International Conference on Supercomputing, 2006.

Digital Library

[35]

C.Waldspurger. Memory resource management in vmware esx server. In Proceedings of the 5th Symposium on Operating Systems Design and Implementation (OSDI), 2002.

Digital Library

[36]

A. Whitaker, M. Shaw, and S. Gribble. Denali: Lightweight virtual machines for distributed and networked applications. In Proceedings of the USENIX Annual Technical Conference, June 2002.

[37]

A. Whitaker, M. Shaw, and S. D. Gribble. Scale and Performance in the Denali Isolation Kernel. In Proceedings of 5th USENIX OSDI, Boston, MA, Dec 2002.

Digital Library

[38]

Xen Wiki. http://wiki.xensource.com/xenwiki/xenbus.

Cited By

Zhang ZXia CLiang CLi JYu CBie TMartin RDan DWang XLiu YGuan H(2024)Un-IOV: Achieving Bare-Metal Level I/O Virtualization Performance for Cloud Usage With Migratability, Scalability and TransparencyIEEE Transactions on Computers10.1109/TC.2024.337558973:7(1655-1668)Online publication date: Jul-2024
https://doi.org/10.1109/TC.2024.3375589
Kadu NJadhav P(2023)High performance computing with inter-virtual-machine communicationJournal of Cyber Security Technology10.1080/23742917.2023.22564328:3(191-203)Online publication date: 14-Sep-2023
https://doi.org/10.1080/23742917.2023.2256432
Devare M(2021)Challenges and Opportunities in High Performance Cloud ComputingResearch Anthology on Architectures, Frameworks, and Integration Strategies for Distributed and Cloud Computing10.4018/978-1-7998-5339-8.ch096(1989-2018)Online publication date: 2021
https://doi.org/10.4018/978-1-7998-5339-8.ch096
Show More Cited By

Index Terms

Nomad: migrating OS-bypass networks in virtual machines
1. Networks
  1. Network properties
    1. Network structure
      1. Network topology types
        Ring networks
        Fiber distributed data interface (FDDI)
  2. Network protocols

Recommendations

Post-copy based live virtual machine migration using adaptive pre-paging and dynamic self-ballooning
VEE '09: Proceedings of the 2009 ACM SIGPLAN/SIGOPS international conference on Virtual execution environments

We present the design, implementation, and evaluation of post-copy based live migration for virtual machines (VMs) across a Gigabit LAN. Live migration is an indispensable feature in today's virtualization technologies. Post-copy migration defers the ...
Cost of Virtual Machine Live Migration in Clouds: A Performance Evaluation
CloudCom '09: Proceedings of the 1st International Conference on Cloud Computing

Virtualization has become commonplace in modern data centers, often referred as "computing clouds". The capability of virtual machine live migration brings benefits such as improved performance, manageability and fault tolerance, while allowing workload ...
Post-copy live migration of virtual machines

We present the design, implementation, and evaluation of post-copy based live migration for virtual machines (VMs) across a Gigabit LAN. Post-copy migration defers the transfer of a VM's memory contents until after its processor state has been sent to ...

Comments

Please enable JavaScript to view thecomments powered by Disqus.

Information & Contributors

Information

Published In

cover image ACM Conferences

VEE '07: Proceedings of the 3rd international conference on Virtual execution environments

June 2007

210 pages

ISBN:9781595936301

DOI:10.1145/1254810

General Chair:
Chandra Krintz
University of California, Santa Barbara
,
Program Chairs:
Steven Hand
University of Cambridge
,
David Tarditi
Microsoft Research

Copyright © 2007 ACM.

Permission to make digital or hard copies of all or part of this work for personal or classroom use is granted without fee provided that copies are not made or distributed for profit or commercial advantage and that copies bear this notice and the full citation on the first page. Copyrights for components of this work owned by others than ACM must be honored. Abstracting with credit is permitted. To copy otherwise, or republish, to post on servers or to redistribute to lists, requires prior specific permission and/or a fee. Request permissions from [email protected]

Sponsors

Publisher

Association for Computing Machinery

New York, NY, United States

Publication History

Published: 13 June 2007

Permissions

Request permissions for this article.

Request Permissions

Check for updates

Author Tags

Qualifiers

Article

Conference

VEE07

Sponsor:

VEE07: International Conference on Virtual Execution Environments

June 13 - 15, 2007

California, San Diego, USA

Acceptance Rates

Overall Acceptance Rate 80 of 235 submissions, 34%

Contributors

Other Metrics

View Article Metrics

Bibliometrics & Citations

Bibliometrics

Article Metrics

57
Total Citations
View Citations
796
Total Downloads

Downloads (Last 12 months)6
Downloads (Last 6 weeks)2

Reflects downloads up to 12 Nov 2024

Other Metrics

View Author Metrics

Citations

Cited By

Zhang ZXia CLiang CLi JYu CBie TMartin RDan DWang XLiu YGuan H(2024)Un-IOV: Achieving Bare-Metal Level I/O Virtualization Performance for Cloud Usage With Migratability, Scalability and TransparencyIEEE Transactions on Computers10.1109/TC.2024.337558973:7(1655-1668)Online publication date: Jul-2024
https://doi.org/10.1109/TC.2024.3375589
Kadu NJadhav P(2023)High performance computing with inter-virtual-machine communicationJournal of Cyber Security Technology10.1080/23742917.2023.22564328:3(191-203)Online publication date: 14-Sep-2023
https://doi.org/10.1080/23742917.2023.2256432
Devare M(2021)Challenges and Opportunities in High Performance Cloud ComputingResearch Anthology on Architectures, Frameworks, and Integration Strategies for Distributed and Cloud Computing10.4018/978-1-7998-5339-8.ch096(1989-2018)Online publication date: 2021
https://doi.org/10.4018/978-1-7998-5339-8.ch096
Fukai TShinagawa TKato K(2021)Live Migration in Bare-Metal CloudsIEEE Transactions on Cloud Computing10.1109/TCC.2018.28489819:1(226-239)Online publication date: 1-Jan-2021
https://doi.org/10.1109/TCC.2018.2848981
Devare M(2019)Challenges and Opportunities in High Performance Cloud ComputingHandbook of Research on the IoT, Cloud Computing, and Wireless Network Optimization10.4018/978-1-5225-7335-7.ch005(85-114)Online publication date: 2019
https://doi.org/10.4018/978-1-5225-7335-7.ch005
Soo WLing TMaw AWin S(2018)Survey on Load-Balancing Methods in 802.11 Infrastructure Mode Wireless Networks for Improving Quality of ServiceACM Computing Surveys10.1145/317286851:2(1-21)Online publication date: 20-Feb-2018
https://dl.acm.org/doi/10.1145/3172868
Zander SWang X(2018)Are We There Yet? IPv6 in Australia and ChinaACM Transactions on Internet Technology10.1145/315837418:3(1-20)Online publication date: 7-Feb-2018
https://dl.acm.org/doi/10.1145/3158374
Zhang WSheng QYao LTaylor KShemshadi AQin Y(2018)A Learning-Based Framework for Improving Querying on Web Interfaces of Curated Knowledge BasesACM Transactions on Internet Technology10.1145/315580618:3(1-20)Online publication date: 5-Feb-2018
https://dl.acm.org/doi/10.1145/3155806
Yao LSheng QWang XZhang WQin Y(2018)Collaborative Location Recommendation by Integrating Multi-dimensional Contextual InformationACM Transactions on Internet Technology10.1145/313443818:3(1-24)Online publication date: 5-Feb-2018
https://dl.acm.org/doi/10.1145/3134438
Farzan AKincaid Z(2017)Strategy synthesis for linear arithmetic gamesProceedings of the ACM on Programming Languages10.1145/31581492:POPL(1-30)Online publication date: 27-Dec-2017
https://dl.acm.org/doi/10.1145/3158149
Show More Cited By

View Options

Get Access

Login options

Check if you have access through your login credentials or your institution to get full access on this article.

Full Access

Get this Publication

View options

PDF

View or Download as a PDF file.

eReader

View online with eReader.

Media

Figures

Other

Tables

View Table of Contents