research-article

Containers and Virtual Machines at Scale: A Comparative Study

Authors:

Prateek Sharma,

Lucas Chaufournier,

Prashant Shenoy,

Y. C. TayAuthors Info & Claims

Middleware '16: Proceedings of the 17th International Middleware Conference

Article No.: 1, Pages 1 - 13

https://doi.org/10.1145/2988336.2988337

Published: 28 November 2016 Publication History

Abstract

Virtualization is used in data center and cloud environments to decouple applications from the hardware they run on. Hardware virtualization and operating system level virtualization are two prominent technologies that enable this. Containers, which use OS virtualization, have recently surged in interest and deployment. In this paper, we study the differences between the two virtualization technologies. We compare containers and virtual machines in large data center environments along the dimensions of performance, manageability and software development.

We evaluate the performance differences caused by the different virtualization technologies in data center environments where multiple applications are running on the same servers (multi-tenancy). Our results show that co-located applications can cause performance interference, and the degree of interference is higher in the case of containers for certain types of workloads. We also evaluate differences in the management frameworks which control deployment and orchestration of containers and VMs. We show how the different capabilities exposed by the two virtualization technologies can affect the management and development of applications. Lastly, we evaluate novel approaches which combine hardware and OS virtualization.

References

[1]

Amazon EC2 Container Service. https://aws.amazon.com/ecs, June 2015.

[2]

Docker. https://www.docker.com/, June 2015.

[3]

Google container engine. https://cloud.google.com/container-engine, June 2015.

[4]

Joyent Public Cloud. https://www.joyent.com, June 2015.

[5]

Kubernetes. https://kubernetes.io, June 2015.

[6]

Linux cgroups. https://www.kernel.org/doc/Documentation/cgroup-v1/cgroups.txt, June 2015.

[7]

Lxc. https://linuxcontainers.org/, June 2015.

[8]

Openstack. https://www.openstack.org, June 2015.

[9]

9p file system interface. http://www.linux-kvm.org/page/9p_virtio, March 2016.

[10]

Bash on ubuntu on windows. https://msdn.microsoft.com/en-us/commandline/wsl/about, 2016.

[11]

Checkpoint Restore in User Space. https://criu.org/, March 2016.

[12]

Cloudstack. https://cloudstack.apache.org/, March 2016.

[13]

Docker Swarm. https://www.docker.com/products/docker-swarm, March 2016.

[14]

Getting and Setting Linux Resource Limits. http://man7.org/linux/man-pages/man2/setrlimit.2.html, March 2016.

[15]

Libvirt Virtualization API. https://libvirt.org, March 2016.

[16]

Linux Kernel Namespaces. https://man7.org/linux/man-pages/man7/namespaces.7.html, March 2016.

[17]

Lxd. https://linuxcontainers.org/lxd/, January 2016.

[18]

Marathon. https://mesosphere.github.io/marathon/, May 2016.

[19]

Unikernels are Unfit for Production. https://www.joyent.com/blog/unikernels-are-unfit-for-production, January 2016.

[20]

Unioning File Systems. https://lwn.net/Articles/327738/, March 2016.

[21]

Vagrant. https://www.vagrantup.com/, March 2016.

[22]

VMware ESX hypervisor. https://www.vmware.com/products/vsphere-hypervisor, March 2016.

[23]

VMware vCenter. https://www.vmware.com/products/vcenter-server, March 2016.

[24]

Windows containers. https://msdn.microsoft.com/virtualization/windowscontainers/containers_welcome, May 2016.

[25]

K. Agarwal, B. Jain, and D. E. Porter. Containing the Hype. In Proceedings of the 6th Asia-Pacific Workshop on Systems, page 8. ACM, 2015.

Digital Library

[26]

P. Barham, B. Dragovic, K. Fraser, S. Hand, T. Harris, A. Ho, R. Neugebauer, I. Pratt, and A. Warfield. Xen and the Art of Virtualization. ACM SIGOPS Operating Systems Review, 37(5):164--177, 2003.

Digital Library

[27]

S. Barker, T. Wood, P. Shenoy, and R. Sitaraman. An Empirical Study of Memory Sharing in Virtual Machines. In USENIX Annual Technical Conference, pages 273--284, 2012.

Digital Library

[28]

J. Beck, D. Comay, L. Ozgur, D. Price, T. Andy, G. Andrew, and S. Blaise. Virtualization and Namespace Isolation in the Solaris Operating System (psarc/2002/174). 2006.

[29]

B. Corrie. VMware Project Bonneville. http://blogs.vmware.com/cloudnative/introducing-project-bonneville/, March 2016.

[30]

R. Dua, A. R. Raja, and D. Kakadia. Virtualization vs Containerization to Support PAAS. In International Conference on Cloud Engineering. IEEE, 2014.

Digital Library

[31]

W. Felter, A. Ferreira, R. Rajamony, and J. Rubio. An Updated Performance Comparison of Virtual Machines and Linux Containers. In Performance Analysis of Systems and Software (ISPASS), 2015 IEEE International Symposium On, pages 171--172.

[32]

M. Gomes Xavier, M. Veiga Neves, F. de Rose, and C. Augusto. A Performance Comparison of Container-based Virtualization Systems for Mapreduce Clusters. In Euromicro International Conference on Parallel, Distributed and Network-Based Processing. IEEE, 2014.

Digital Library

[33]

T. Harter, B. Salmon, R. Liu, A. C. Arpaci-Dusseau, and R. H. Arpaci-Dusseau. Slacker: Fast Distribution with Lazy Docker Containers. In Proceedings of the 14th Usenix Conference on File and Storage Technologies, pages 181--195. USENIX Association, 2016.

Digital Library

[34]

B. Hindman, A. Konwinski, M. Zaharia, A. Ghodsi, A. D. Joseph, R. H. Katz, S. Shenker, and I. Stoica. Mesos: A Platform for Fine-Grained Resource Sharing in the Data Center. In NSDI, volume 11, pages 22--22, 2011.

Digital Library

[35]

J. Hwang, S. Zeng, F. Y. Wu, and T. Wood. A Component-based Performance Comparison of Four Hypervisors. In IFIP/IEEE International Symposium on Integrated Network Management. IEEE.

[36]

Intel. Clear linux. http://clearlinux.org, March 2016.

[37]

Joyent. Lx branded zones. https://wiki.smartos.org/display/DOC/LX+Branded+Zones, June 2015.

[38]

P.-H. Kamp and R. N. Watson. Jails: Confining the Omnipotent Root. In Proceedings of the 2nd International SANE Conference, volume 43, page 116, 2000.

[39]

E. Keller, J. Szefer, J. Rexford, and R. B. Lee. NoHype: Virtualized Cloud Infrastructure without the Virtualization. In ACM SIGARCH Computer Architecture News, volume 38, pages 350--361. ACM, 2010.

Digital Library

[40]

A. Kivity, Y. Kamay, D. Laor, U. Lublin, and A. Liguori. KVM: the Linux Virtual Machine Monitor. In Proceedings of the Linux Symposium, volume 1, pages 225--230, 2007.

[41]

H. A. Lagar-Cavilla, J. A. Whitney, A. M. Scannell, P. Patchin, S. M. Rumble, E. De Lara, M. Brudno, and M. Satyanarayanan. SnowFlock: Rapid Virtual Machine Cloning for Cloud Computing. In EuroSys, April 2009.

Digital Library

[42]

A. Madhavapeddy, R. Mortier, C. Rotsos, D. Scott, B. Singh, T. Gazagnaire, S. Smith, S. Hand, and J. Crowcroft. Unikernels: Library operating systems for the cloud. In Proceedings of the Eighteenth International Conference on Architectural Support for Programming Languages and Operating Systems, ASPLOS '13, pages 461--472, New York, NY, USA, 2013. ACM.

Digital Library

[43]

J. N. Matthews, W. Hu, M. Hapuarachchi, T. Deshane, D. Dimatos, G. Hamilton, M. McCabe, and J. Owens. Quantifying the Performance Isolation Properties of Virtualization Systems. In Workshop on Experimental computer science, page 6. ACM, 2007.

Digital Library

[44]

D. Mishra and P. Kulkarni. Comparative Analysis of Page Cache Provisioning in Virtualized Environments. In Modelling, Analysis & Simulation of Computer and Telecommunication Systems (MASCOTS), 2014 IEEE 22nd International Symposium on, pages 213--222. IEEE, 2014.

Digital Library

[45]

J. Ouyang and J. R. Lange. Preemptable Ticket Spinlocks: Improving Consolidated Performance in the Cloud. In ACM SIGPLAN Notices, volume 48, pages 191--200. ACM, 2013.

Digital Library

[46]

C. Ruiz, E. Jeanvoine, and L. Nussbaum. Performance Evaluation of Containers for HPC. In Euro-Par: Parallel Processing Workshops, 2015.

[47]

P. Sharma and P. Kulkarni. Singleton: System-wide Page Deduplication in Virtual Environments. In Proceedings of the 21st international symposium on High-Performance Parallel and Distributed Computing, pages 15--26. ACM, 2012.

Digital Library

[48]

S. Soltesz, H. Pötzl, M. E. Fiuczynski, A. Bavier, and L. Peterson. Container-based Operating System Virtualization: A Scalable, High-Performance Alternative to Hypervisors. In EuroSys. ACM, 2007.

Digital Library

[49]

C. Tang. Fvd: A High-Performance Virtual Machine Image Format for Cloud. In USENIX Annual Technical Conference, 2011.

Digital Library

[50]

C. A. Waldspurger. Memory Resource Management in VMware ESX Server. SIGOPS Oper. Syst. Rev., 36(SI):181--194, Dec. 2002.

Digital Library

[51]

T. Wood, P. J. Shenoy, A. Venkataramani, and M. S. Yousif. Black-box and Gray-box Strategies for Virtual Machine Migration. In NSDI, volume 7, pages 17--17, 2007.

Digital Library

[52]

M. G. Xavier, M. V. Neves, F. D. Rossi, T. C. Ferreto, T. Lange, and C. A. De Rose. Performance Evaluation of Container-based Virtualization for High Performance Computing Environments. In Euromicro International Conference on Parallel, Distributed and Network-Based Processing. IEEE, 2013.

Digital Library

[53]

I. Zhang, A. Garthwaite, Y. Baskakov, and K. C. Barr. Fast Restore of Checkpointed Memory Using Working Set Estimation. In VEE, 2011.

Digital Library

Cited By

Salcedo-Navarro APeña-Ortiz RClaver JGarcia-Pineda MGutiérrez-Aguado J(2025)Towards GPU-enabled serverless cloud edge platforms for accelerating HEVC video codingCluster Computing10.1007/s10586-024-04692-028:1Online publication date: 1-Feb-2025
https://dl.acm.org/doi/10.1007/s10586-024-04692-0
Urblik LKajati EPapcun PZolotová I(2024)Containerization in Edge Intelligence: A ReviewElectronics10.3390/electronics1307133513:7(1335)Online publication date: 2-Apr-2024
https://doi.org/10.3390/electronics13071335
Tripathi SMishra D(2024)PCLive: Pipelined Restoration of Application Containers for Reduced Service DowntimeProceedings of the 2024 ACM Symposium on Cloud Computing10.1145/3698038.3698545(284-301)Online publication date: 20-Nov-2024
https://dl.acm.org/doi/10.1145/3698038.3698545
Show More Cited By

Containers and Virtual Machines at Scale: A Comparative Study
1. Software and its engineering
  1. Software organization and properties
    1. Contextual software domains

Recommendations

Traffic-sensitive live migration of virtual machines
CCGRID '15: Proceedings of the 15th IEEE/ACM International Symposium on Cluster, Cloud, and Grid Computing

In this paper we address the problem of network contention between the migration traffic and the Virtual Machine (VM) application traffic for the live migration of co-located Virtual Machines. When VMs are migrated with pre-copy, they run at the source ...
Live gang migration of virtual machines
HPDC '11: Proceedings of the 20th international symposium on High performance distributed computing

This paper addresses the problem of simultaneously migrating a group of co-located and live virtual machines (VMs), i.e, VMs executing on the same physical machine. We refer to such a mass simultaneous migration of active VMs as "live gang migration". ...
Optimizing virtual machines using hybrid virtualization

Minimizing virtualization overhead and improving the reliability of virtual machines are challenging when establishing virtual machine cluster. Paravirtualization and hardware-assisted virtualization are two mainstream solutions for modern system ...

Comments

Please enable JavaScript to view thecomments powered by Disqus.

Information & Contributors

Information

Published In

cover image ACM Conferences

Middleware '16: Proceedings of the 17th International Middleware Conference

November 2016

280 pages

ISBN:9781450343008

DOI:10.1145/2988336

Copyright © 2016 ACM.

Permission to make digital or hard copies of all or part of this work for personal or classroom use is granted without fee provided that copies are not made or distributed for profit or commercial advantage and that copies bear this notice and the full citation on the first page. Copyrights for components of this work owned by others than the author(s) must be honored. Abstracting with credit is permitted. To copy otherwise, or republish, to post on servers or to redistribute to lists, requires prior specific permission and/or a fee. Request permissions from [email protected].

Sponsors

ACM: Association for Computing Machinery
USENIX Assoc: USENIX Assoc
IFIP

Publisher

Association for Computing Machinery

New York, NY, United States

Publication History

Published: 28 November 2016

Permissions

Request permissions for this article.

Request Permissions

Check for updates

Qualifiers

Research-article
Research
Refereed limited

Conference

Middleware '16

Sponsor:

ACM
USENIX Assoc

Middleware '16: 17th International Middleware Conference

December 12 - 16, 2016

Trento, Italy

Acceptance Rates

Overall Acceptance Rate 203 of 948 submissions, 21%

Upcoming Conference

MIDDLEWARE '24

25th International Middleware Conference

December 2 - 6, 2024

Hong Kong , Hong Kong

Contributors

Other Metrics

View Article Metrics

Bibliometrics & Citations

Bibliometrics

Article Metrics

157
Total Citations
View Citations
2,935
Total Downloads

Downloads (Last 12 months)457
Downloads (Last 6 weeks)64

Reflects downloads up to 13 Nov 2024

Other Metrics

View Author Metrics

Citations

Cited By

Salcedo-Navarro APeña-Ortiz RClaver JGarcia-Pineda MGutiérrez-Aguado J(2025)Towards GPU-enabled serverless cloud edge platforms for accelerating HEVC video codingCluster Computing10.1007/s10586-024-04692-028:1Online publication date: 1-Feb-2025
https://dl.acm.org/doi/10.1007/s10586-024-04692-0
Urblik LKajati EPapcun PZolotová I(2024)Containerization in Edge Intelligence: A ReviewElectronics10.3390/electronics1307133513:7(1335)Online publication date: 2-Apr-2024
https://doi.org/10.3390/electronics13071335
Tripathi SMishra D(2024)PCLive: Pipelined Restoration of Application Containers for Reduced Service DowntimeProceedings of the 2024 ACM Symposium on Cloud Computing10.1145/3698038.3698545(284-301)Online publication date: 20-Nov-2024
https://dl.acm.org/doi/10.1145/3698038.3698545
Guo JWu DMa CYu HSun GLuo LXu YZhang NSekar VYu MSeneviratne AVeitch D(2024)Poster: A Hybrid Virtual-Real Emulation Platform for Computer Network EducationProceedings of the ACM SIGCOMM 2024 Conference: Posters and Demos10.1145/3672202.3673736(45-47)Online publication date: 4-Aug-2024
https://dl.acm.org/doi/10.1145/3672202.3673736
Huang ZZhang QFan HWu SYu CJin HDeng JGu JTang Z(2024)Multi-Grained Trace Collection, Analysis, and Management of Diverse Container ImagesIEEE Transactions on Computers10.1109/TC.2024.338396673:7(1698-1710)Online publication date: Jul-2024
https://doi.org/10.1109/TC.2024.3383966
Kamieniarz KMazurczyk W(2024)A Comparative Study on the Security of Kubernetes Deployments2024 International Wireless Communications and Mobile Computing (IWCMC)10.1109/IWCMC61514.2024.10592468(0718-0723)Online publication date: 27-May-2024
https://doi.org/10.1109/IWCMC61514.2024.10592468
Müller KGiani MDeubert DMassoth M(2024)Virtualization in Industrial Production - A Survey Focusing on Virtual and Virtualized Industrial Controls2024 IEEE 29th International Conference on Emerging Technologies and Factory Automation (ETFA)10.1109/ETFA61755.2024.10710668(1-7)Online publication date: 10-Sep-2024
https://doi.org/10.1109/ETFA61755.2024.10710668
Azadiabad SKhendek F(2024)Dependability of Network Services in the Context of NFV: A Taxonomy and State of the Art ClassificationJournal of Network and Systems Management10.1007/s10922-024-09810-232:2Online publication date: 26-Mar-2024
https://doi.org/10.1007/s10922-024-09810-2
Bawa SRana PTekchandani R(2024)Migration of containers on the basis of load prediction with dynamic inertia weight based PSO algorithmCluster Computing10.1007/s10586-024-04676-027:10(14585-14609)Online publication date: 1-Dec-2024
https://dl.acm.org/doi/10.1007/s10586-024-04676-0
Olabanji DFitch TMatthew O(2023)Multi-tenancy in Cloud-native Architecture: A Systematic Mapping StudyWSEAS TRANSACTIONS ON COMPUTERS10.37394/23205.2023.22.422(25-43)Online publication date: 7-Mar-2023
https://doi.org/10.37394/23205.2023.22.4
Show More Cited By

View Options

Get Access

Login options

Check if you have access through your login credentials or your institution to get full access on this article.

Full Access

Get this Publication

View options

PDF

View or Download as a PDF file.

eReader

View online with eReader.

Media

Figures

Other

Tables

View Table of Contents