Article

Xen and co.: communication-aware CPU scheduling for consolidated xen-based hosting platforms

Authors:

Sriram Govindan,

Bhuvan Urgaonkar,

Anand SivasubramaniamAuthors Info & Claims

VEE '07: Proceedings of the 3rd international conference on Virtual execution environments

Pages 126 - 136

https://doi.org/10.1145/1254810.1254828

Published: 13 June 2007 Publication History

Abstract

Recent advances in software and architectural support for server virtualization have created interest in using this technology in the design of consolidated hosting platforms. Since virtualization enables easier and faster application migration as well as secure co-location of antagonistic applications, higher degrees of server consolidation are likely to result in such virtualization-based hosting platforms (VHPs). We identify a key shortcoming in existing virtual machine monitors (VMMs) that proves to be an obstacle in operating hosting platforms, such as Internet data centers, under conditions of such high consolidation: CPU schedulers that are agnostic to the communication behavior of modern, multi-tier applications. We develop a new communication-aware CPU scheduling algorithm to alleviate this problem. We implement our algorithm in the Xen VMM and build a prototype VHP on a cluster of servers. Our experimental evaluation with realistic Internet server applications and benchmarks demonstrates the performance/cost benefits and the wide applicability of our algorithms. For example, the TPC-W benchmark exhibited improvements in average response times of up to 35% for a variety of consolidation scenarios. A streaming media server hosted on our prototype VHP was able to satisfactorily service up to 3.5 times as many clients as one running on the default Xen.

References

[1]

M. Adler, Y. Gong, and A. Rosenberg. Optimal Sharing of Bags of Tasks in Heterogeneous Clusters. In Proceedings of the Fifteenth Annual ACM Symposium on Parallel Algorithms and Architectures (SPAA), Scheduling I, pages 1--10. ACM Press, 2003.

Digital Library

[2]

Amazon Elastic Compute Cloud. http://www.nature.com/.

[3]

A. Arpaci-Dusseau. Implicit Coscheduling: Coordinated Scheduling with Implicit Information in Distributed Systems. ACM Transactions on Computer Systems, 19(3):283--331, 2001.

Digital Library

[4]

A. Arpaci-Dusseau and D.E. Culler. Extending Proportional-Share Scheduling to a Network of Workstations. In Proceedings of Parallel and Distributed Processing Techniques and Applications (PDPTA'97), Las Vegas, NV, June 1997.

[5]

G. Banga, P. Druschel, and J. Mogul. Resource Containers: A New Facility for Resource Management in Server Systems. In Proceedings of the Third Symposium on Operating System Design and Implementation (OSDI'99), New Orleans, pages 45--58, February 1999.

Digital Library

[6]

P. Barham, B. Dragovic, K. Fraser, S. Hand, T. Harris, A. Ho, R. Neugebuer, I. Pratt, and A. Warfield. Xen and the Art of Virtulization. In Proceedings of the Nineteenth Symposium on Operating Systems Principles (SOSP), 2003.

Digital Library

[7]

M. Benani and D. Menasce. Resource Allocation for Autonomic Data Centers Using Analytic Performance Models. In Proceedings of IEEE International Conference on Autonomic Computing, Seattle (ICAC--05), WA, June 2005.

Digital Library

[8]

M. Bennani and D. Menacse. Autonomic Virtualized Environments. In Proceedings of the IEEE International Conference on Autonomic and Autonomous Systems (ICAS 2006), Santa Clara, CA, July 2006.

Digital Library

[9]

A. Chandra, M. Adler, P. Goyal, and P. Shenoy. Surplus Fair Scheduling: A Proportional-Share CPU Scheduling Algorithm for Symmetric Multiprocessors. In Proceedings of the Fourth Symposium on Operating System Design and Implementation (OSDI 2000), San Diego, CA, October 2000.

Digital Library

[10]

Y. Chen, A. Das, Q. Wang, A. Sivasubramaniam, R. Harper, and M. Bland. Consolidating Clients on Back-end Servers with Co-location and Frequency Control. In Posthe ACM International Conference on Measurement and Modeling of Computer Systems (SIGMETRICS 2006), June 2006, June 2006.

Digital Library

[11]

C. Clark, K. Fraser, Steven Hand, J. Hansen, E. Jul, C. Limpach, I. Pratt, and A. Warfield. Live Migration of Virtual Machines. In Proceedings of the Second Symposium on Networked Systems Design and Implementation (NSDI'05), May 2005.

Digital Library

[12]

Credit Based Scheduler. http://wiki.xensource.com/xenwiki/CreditScheduler.

[13]

R. Doyle, J. Chase, O. Asad, W. Jin, and Amin Vahdat. Model-Based Resource Provisioning in a Web Service Utility. In Proceedings of the Fourth USITS, March 2003.

Digital Library

[14]

K. J. Duda and D. R. Cheriton. Borrowed-virtual-time (BVT) Scheduling: Supporting Latency-sensitive Threads in a General-purpose Scheduler. In Proceedings of the Seventeenth ACM Symposium on Operating Systems Principles, pages 261--276, New York, NY, USA, 1999. ACM Press.

Digital Library

[15]

R. Goldberg. Survey of Virtual Machine Research. IEEE Computer, pages 34--45, June 1974.

Digital Library

[16]

S. Govindan, A. Nath, A. Das, B. Urgaonkar, and A. Sivasubramaniam. Communication-aware CPU Management in Consolidated Virtualization-based Hosting Platforms. Technical report, Department of Computer Science and Engineering, The Pennsylvania State University, October 2006.

[17]

D. Gupta, L. Cherkasova, R. Gardner, and A. Vahdat. Enforcing Performance Isolation Across Virtual Machines in Xen. In Proceedings of the Seventh International Middleware Conference, Melbourne, Australia, November-December 2006.

Digital Library

[18]

D. Gupta, R. Gardner, and L. Cherkasova. XenMon: QoS Monitoring and Performance Profiling Tool. Technical Report HPL-2005-187, HP Labs, 2005.

[19]

Intel VT. http://www.intel.com/technology/itj/2006/v10i3/foreword.htm.

[20]

The JBoss Application Server. http://www.jboss.org.

[21]

M. B. Jones, D. Rosu, and M. Rosu. CPU Reservations and Time Constraints: Efficient, Predictable Scheduling of Independent Activities. In Proceedings of the Sixteenth ACM Symposium on Operating Systems Principles (SOSP'97), Saint-Malo, France, pages 198--211, December 1997.

Digital Library

[22]

S. T. Leutenegger and M. K. Vernon. The Performance of Multiprogrammed Multiprocessor Scheduling Algorithms. In SIGMETRICS '90: Proceedings of the 1990 ACM SIGMETRICS Conference on Measurement and Modeling of Computer Systems, pages 226--236, 1990.

Digital Library

[23]

J. Liu, W. Huang, B. Abali, and D. K. Panda. High Performance VMM-Bypass I/O in Virtual Machines. In Proceedings of the USENIX Annual Technical Conference (USENIX'06), Boston, MA, May-June 2006.

Digital Library

[24]

A. Menon, A. Cox, and W. Zwaenepoel. Optimizing Network Virtualization in Xen. In Proceedings of the USENIX Annual Technical Conference (USENIX'06), Boston, MA, May 2006.

Digital Library

[25]

MySQL. http://www.mysql.com.

[26]

S. Nagar, A. Banerjee, A. Sivasubramaniam, and C. R. Das. A Closer Look at Co-scheduling Approaches for a Network of Workstations. In SPAA '99: Proceedings of the eleventh annual ACM symposium on Parallel algorithms and architectures, pages 96--105, 1999.

Digital Library

[27]

M. Nelson, B. Lim, and G. Hutchins. Fast Transparent Migration for Virtual Machines. In Proceedings of the 2005 USENIX Annual Technical Conference, pages 391--394, April, 2005.

Digital Library

[28]

J. Nieh and M. Lam. A SMART Scheduler for Multimedia Applications. ACM Transactions on Computer Systems, 21(2):117--163, 2003.

Digital Library

[29]

S. Osman, D. Subhraveti, G. Su, and J. Nieh. The Design and Implementation of Zap: A System for Migrating Computing Environments. In Proceedings of Fifth USENIX Symposium on Operating Systems Design and Implementation, pages 361--376, 2002.

Digital Library

[30]

C. Sapuntzakis, R. Chandra, B. Pfaff, J. Chow, M. S. Lam, and M. Rosenblum. Optimizing the Migration of Virtual Computers. In Proceedings of the 5th Symposium on Operating Systems Design and Implementation, December 2002.

Digital Library

[31]

K. Shen, H. Tang, T. Yang, and L. Chu. Integrated Resource Management for Cluster-based Internet Services. In Proceedings of the Fifth USENIX Symposium on Operating Systems Design and Implementation (OSDI), Boston, MA, December 2002.

Digital Library

[32]

S. Shende, A. Malony, J. Cuny, K. Lindlan, P. Beckman, and S. Karmesin. Portable Profiling and Tracing for Parallel Scientific Applications using C++. In Proceedings of ACM SIGMETRICS Symposium on Parallel and Distributed Tools (SPDT), pages 134--145, August 1998.

Digital Library

[33]

J. E. Smith and R. Nair. Virtual Machines: Architectures, Implementations and Applications. Morgan Kauffmann, New York, 2004.

[34]

W. Smith. TPC--W: Benchmarking An Ecommerce Solution. http://www.tpc.org/information/other/techarticles.asp.

[35]

P. Sobalvarro and W. E. Weihl. Demand-Based Coscheduling of Parallel Jobs on Multiprogrammed Multiprocessors. In IPPS '95: Proceedings of the Workshop on Job Scheduling Strategies for Parallel Processing, pages 106--126, 1995.

Digital Library

[36]

M. S. Squillante, Y. Zhang, A. Sivasubramaniam, N. Gautam, H. Franke, and J. Moreira. Modeling and Analysis of Dynamic Co-scheduling in Parallel and Distributed Environments. In SIGMETRICS '02: Proceedings of the 2002 ACM SIGMETRICS International Conference on Measurement and Modeling of Computer Systems, pages 43--54, 2002.

Digital Library

[37]

NYU TPC-W. http://www.cs.nyu.edu/pdsg/.

[38]

B. Urgaonkar, G. Pacifici, P. Shenoy, M. Spreitzer, and A. Tantawi. An Analytical Model for Multi-tier Internet Services and its Applications. In Proceedings of the ACM International Conference on Measurement and Modeling of Computer Systems (SIGMETRICS 2005), Banff, Canada, June 2005.

Digital Library

[39]

B. Urgaonkar, P. Shenoy, and T. Roscoe. Resource Overbooking and Application Profiling in Shared Hosting Platforms. In Proceedings of the Fifth USENIX Symposium on Operating Systems Design and Implementation (OSDI 2002), Boston, MA, December 2002.

Digital Library

[40]

VMware. http://www.vmware.com/.

[41]

C. Waldspurger. Memory Resource Management in VMWare ESX Server. In Proceedings of the Fifth Symposium on Operating System Design and Implementation (OSDI'02), December 2002.

Digital Library

[42]

C. A. Waldspurger and W. E. Weihl. Lottery Scheduling: Flexible Proportional share Resource Management. In Proceedings of the USENIX Symposium on Operating System Design and Implementation (OSDI'94), November 1994.

Digital Library

[43]

A. Whitaker, M. Shaw, and S. D. Gribble. Scale and Performance in the Denali Isolation Kernel. In Proceedings of the Fifth Symposium on Operating System Design and Implementation (OSDI'02), December 2002.

Digital Library

[44]

XenSource Press Release. http://www.xensource.com/news/pressreleases.html.

[45]

Y. Zhang, H. Franke, J. Moreira, and A. Sivasubramaniam. Improving Parallel Job Scheduling by Combining Gang Scheduling and Backfilling Techniques. In IPDPS '00: Proceedings of the 14th International Symposium on Parallel and Distributed Processing, pages 133--142, 2000.

Digital Library

[46]

Y. Zhang, A. Sivasubramaniam, J. E. Moreira, and H. Franke. A Simulationbased Study of Scheduling Mechanisms for a Dynamic Cluster Environment. In Proceedings of the 11th ACM International Conference on Supercomputing (ICS), pages 100--109, 2000.

Digital Library

Cited By

Jia WZhang JShan JLi JDing XGavrilovska AAltınbüken DBinnig C(2022)Achieving low latency in public edges by hiding workloads mutual interferenceProceedings of the 13th Symposium on Cloud Computing10.1145/3542929.3563459(477-492)Online publication date: 7-Nov-2022
https://dl.acm.org/doi/10.1145/3542929.3563459
Lee JLee HYu H(2022)Fine-Grained I/O Traffic Control Middleware for I/O Fairness in Virtualized SystemIEEE Access10.1109/ACCESS.2022.318773110(73122-73144)Online publication date: 2022
https://doi.org/10.1109/ACCESS.2022.3187731
Yang CShinjo YNagarakatte SBaumann AKasikci B(2020)Obtaining hard real-time performance and rich Linux features in a compounded real-time operating system by a partitioning hypervisorProceedings of the 16th ACM SIGPLAN/SIGOPS International Conference on Virtual Execution Environments10.1145/3381052.3381323(59-72)Online publication date: 17-Mar-2020
https://dl.acm.org/doi/10.1145/3381052.3381323
Show More Cited By

Index Terms

Xen and co.: communication-aware CPU scheduling for consolidated xen-based hosting platforms
1. Networks
  1. Network protocols
2. Software and its engineering
  1. Software organization and properties
    1. Contextual software domains
      1. Operating systems
        Process management
        Scheduling

Recommendations

Xen and Co.: Communication-Aware CPU Management in Consolidated Xen-Based Hosting Platforms

Recent advances in software and architectural support for server virtualization have created interest in using this technology in the design of consolidated hosting platforms. Since virtualization enables easier and faster application migration as well ...
Hybrid CPU Management for Adapting to the Diversity of Virtual Machines

As an important cornerstone for clouds, virtualization plays a vital role in building this emerging infrastructure. Virtual machines (VMs) with a variety of workloads may run simultaneously on a physical machine in the cloud platform. The scheduling ...
Fast Live Cloning of Virtual Machine Based on Xen
HPCC '09: Proceedings of the 2009 11th IEEE International Conference on High Performance Computing and Communications

Virtual Machine (VM) cloning is to create a replica of a source virtual machine (parent virtual machine); the replica, also called child virtual machine, owns exactly the same executing status as parent virtual machine. Fast live cloning guarantees that,...

Comments

Please enable JavaScript to view thecomments powered by Disqus.

Information & Contributors

Information

Published In

cover image ACM Conferences

VEE '07: Proceedings of the 3rd international conference on Virtual execution environments

June 2007

210 pages

ISBN:9781595936301

DOI:10.1145/1254810

General Chair:
Chandra Krintz
University of California, Santa Barbara
,
Program Chairs:
Steven Hand
University of Cambridge
,
David Tarditi
Microsoft Research

Copyright © 2007 ACM.

Permission to make digital or hard copies of all or part of this work for personal or classroom use is granted without fee provided that copies are not made or distributed for profit or commercial advantage and that copies bear this notice and the full citation on the first page. Copyrights for components of this work owned by others than ACM must be honored. Abstracting with credit is permitted. To copy otherwise, or republish, to post on servers or to redistribute to lists, requires prior specific permission and/or a fee. Request permissions from [email protected]

Sponsors

Publisher

Association for Computing Machinery

New York, NY, United States

Publication History

Published: 13 June 2007

Permissions

Request permissions for this article.

Request Permissions

Check for updates

Author Tags

Qualifiers

Article

Conference

VEE07

Sponsor:

VEE07: International Conference on Virtual Execution Environments

June 13 - 15, 2007

California, San Diego, USA

Acceptance Rates

Overall Acceptance Rate 80 of 235 submissions, 34%

Contributors

Other Metrics

View Article Metrics

Bibliometrics & Citations

Bibliometrics

Article Metrics

155
Total Citations
View Citations
1,763
Total Downloads

Downloads (Last 12 months)2
Downloads (Last 6 weeks)0

Reflects downloads up to 12 Nov 2024

Other Metrics

View Author Metrics

Citations

Cited By

Jia WZhang JShan JLi JDing XGavrilovska AAltınbüken DBinnig C(2022)Achieving low latency in public edges by hiding workloads mutual interferenceProceedings of the 13th Symposium on Cloud Computing10.1145/3542929.3563459(477-492)Online publication date: 7-Nov-2022
https://dl.acm.org/doi/10.1145/3542929.3563459
Lee JLee HYu H(2022)Fine-Grained I/O Traffic Control Middleware for I/O Fairness in Virtualized SystemIEEE Access10.1109/ACCESS.2022.318773110(73122-73144)Online publication date: 2022
https://doi.org/10.1109/ACCESS.2022.3187731
Yang CShinjo YNagarakatte SBaumann AKasikci B(2020)Obtaining hard real-time performance and rich Linux features in a compounded real-time operating system by a partitioning hypervisorProceedings of the 16th ACM SIGPLAN/SIGOPS International Conference on Virtual Execution Environments10.1145/3381052.3381323(59-72)Online publication date: 17-Mar-2020
https://dl.acm.org/doi/10.1145/3381052.3381323
Kashyap RVidyarthi D(2019)A Secured Real Time Scheduling Model for Cloud HypervisorCloud Security10.4018/978-1-5225-8176-5.ch026(507-522)Online publication date: 2019
https://doi.org/10.4018/978-1-5225-8176-5.ch026
Li JXue SZhang WMa RQi ZGuan H(2019)When I/O Interrupt Becomes System Bottleneck: Efficiency and Scalability Enhancement for SR-IOV Network VirtualizationIEEE Transactions on Cloud Computing10.1109/TCC.2017.27126867:4(1183-1196)Online publication date: 1-Oct-2019
https://doi.org/10.1109/TCC.2017.2712686
Xu CWang HShea RWang FLiu J(2018)On Multiple Virtual NICs in Cloud Computing: Performance Bottleneck and EnhancementIEEE Systems Journal10.1109/JSYST.2017.274760312:3(2417-2427)Online publication date: Sep-2018
https://doi.org/10.1109/JSYST.2017.2747603
Zhang WHu XLi JGuan H(2018)CoINT: Proactive Coordinator for Avoiding Interruptability Holder Preemption Problem in VSMP EnvironmentIEEE INFOCOM 2018 - IEEE Conference on Computer Communications10.1109/INFOCOM.2018.8485997(477-485)Online publication date: Apr-2018
https://doi.org/10.1109/INFOCOM.2018.8485997
Zhao MWang LLv YXu J(2018)Cross-Layer Optimization for Virtual Machine Resource Management2018 IEEE International Conference on Cloud Engineering (IC2E)10.1109/IC2E.2018.00031(90-98)Online publication date: Apr-2018
https://doi.org/10.1109/IC2E.2018.00031
Li YZhang JChen XLi JDing J(2017)An accurate resource scheduling system for virtual machines based on CPU load monitoring and assessmentCluster Computing10.1007/s10586-017-1344-z21:2(1395-1410)Online publication date: 13-Nov-2017
https://doi.org/10.1007/s10586-017-1344-z
Chen LBirke RSmirni E(2017)State of Practice of Non-self-aware Virtual Machine Management in Cloud Data CentersSelf-Aware Computing Systems10.1007/978-3-319-47474-8_19(555-574)Online publication date: 24-Jan-2017
https://doi.org/10.1007/978-3-319-47474-8_19
Show More Cited By

View Options

Get Access

Login options

Check if you have access through your login credentials or your institution to get full access on this article.

Full Access

Get this Publication

View options

PDF

View or Download as a PDF file.

eReader

View online with eReader.

Media

Figures

Other

Tables

View Table of Contents