research-article

Preserving I/O prioritization in virtualized OSes

Authors:

Francis C. M. LauAuthors Info & Claims

SoCC '17: Proceedings of the 2017 Symposium on Cloud Computing

Pages 269 - 281

https://doi.org/10.1145/3127479.3127484

Published: 24 September 2017 Publication History

Abstract

While virtualization helps to enable multi-tenancy in data centers, it introduces new challenges to the resource management in traditional OSes. We find that one important design in an OS, prioritizing interactive and I/O-bound workloads, can become ineffective in a virtualized OS. Resource multiplexing between multiple tenants breaks the assumption of continuous CPU availability in physical systems and causes two types of priority inversions in virtualized OSes. In this paper, we present xBalloon, a lightweight approach to preserving I/O prioritization. It uses a balloon process in the virtualized OS to avoid priority inversion in both short-term and long-term scheduling. Experiments in a local Xen environment and Amazon EC2 show that xBalloon improves I/O performance in a recent Linux kernel by as much as 136% on network throughput, 95% on disk throughput, and 125x on network tail latency.

References

[1]

Amazon ec2 t2 instances. https://goo.gl/vbCpkV.

[2]

Amazon elastic compute cloud(ec2). http://aws.amazon.com/ec2/.

[3]

Aws instance usage report from rightscale users. https://goo.gl/AjDKiK.

[4]

Cloudsuite: The benchmark suite of cloud services. http://cloudsuite.ch/.

[5]

Credit scheduler cap. https://wiki.xen.org/wiki/Credit_Scheduler#Cap.

[6]

The docker container. https://www.docker.com/.

[7]

Kernel based virtual machine. http://www.linux-kvm.org/.

[8]

Kvm cpu accounting. https://github.com/penberg/linux-kvm/blob/master/kernel/sched/sched.h.

[9]

Memcached. https://memcached.org/.

[10]

Netperf. http://www.netperf.org/.

[11]

Nginx. https://www.nginx.com/.

[12]

Redis. http://redis.io.

[13]

Siege. https://www.joedog.org/siege-home/.

[14]

Sockperf. https://github.com/Mellanox/sockperf.

[15]

Spec cpu 2006 benchmark. https://www.spec.org/cpu2006/.

[16]

The sysbench benchmark suite. https://github.com/akopytov/sysbench.

[17]

Ahn, J., Park, C. H., and Huh, J. Micro-sliced virtual processors to hide the effect of discontinuous cpu availability for consolidated systems. In Proc. of the 47th Annual IEEE/ACM International Symposium on Microarchitecture (MICRO) (2014).

Digital Library

[18]

Barham, P., Dragovic, B., Fraser, K., Hand, S., Harris, T. L., Ho, A., Neugebauer, R., Pratt, I., and Warfield, A. Xen and the art of virtualization. In Proc. of ACM Symposium on Operating Systems Principles (SOSP) (2003).

Digital Library

[19]

Cheng, L., Wang, C., and Lau, F. C. M. PVTCP: towards practical and effective congestion control in virtualized datacenters. In Proc. of IEEE International Conference on Network Protocols (ICNP) (2013).

[20]

Dolan-Gavitt, B., Leek, T., Zhivich, M., Giffin, J., and Lee, W. Virtuoso: Narrowing the semantic gap in virtual machine introspection. In Proceedings of the IEEE Symposium on Security and Privacy (SP) (2011).

Digital Library

[21]

Fu, Y., and Lin, Z. Space traveling across vm: Automatically bridging the semantic gap in virtual machine introspection via online kernel data redirection. In Proceedings of the 2012 IEEE Symposium on Security and Privacy(SP) (2012).

Digital Library

[22]

Gamage, S., Kangarlou, A., Kompella, R. R., and Xu, D. Opportunistic flooding to improve TCP transmit performance in virtualized clouds. In Proceedings of ACM Symposium on Cloud Computing (SoCC) (2011).

Digital Library

[23]

Gamage, S., Xu, C., Kompella, R. R., and Xu, D. vpipe: Piped i/o offloading for efficient data movement in virtualized clouds. In Proceedings of the ACM Symposium on Cloud Computing (SoCC) (2014).

Digital Library

[24]

Gordon, A., Amit, N., Har'El, N., Ben-Yehuda, M., Landau, A., Schuster, A., and Tsafrir, D. Eli: bare-metal performance for i/o virtualization. In Proceedings of the International Conference on Architectural Support for Programming Languages and Operating Systems (ASPLOS) (2012).

Digital Library

[25]

Hwang, J., Ramakrishnan, K., and Wood, T. Netvm: high performance and flexible networking using virtualization on commodity platforms. In Proceedings of the 11th USENIX Conference on Networked Systems Design and Implementation (NSDI) (2014).

[26]

Jones, S. T., Arpaci-Dusseau, A. C., and Arpaci-Dusseau, R. H. Geiger: monitoring the buffer cache in a virtual machine environment. In Proceedings of the 12th International Conference on Architectural Support for Programming Languages and Operating Systems (ASPLOS) (2006).

Digital Library

[27]

Jones, S. T., Arpaci-Dusseau, A. C., Arpaci-Dusseau, R. H., et al. Antfarm: Tracking processes in a virtual machine environment. In USENIX Annual Technical Conference (USENIX ATC) (2006).

[28]

Kang, H., Chen, Y., Wong, J. L., Sion, R., and Wu, J. Enhancement of xen's scheduler for mapreduce workloads. In Proceedings of the 20th international symposium on High performance distributed computing (HPDC) (2011).

Digital Library

[29]

Kangarlou, A., Gamage, S., Kompella, R. R., and Xu, D. vsnoop: Improving tcp throughput in virtualized environments via acknowledgement offload. In Proceedings of the 2010 ACM/IEEE International Conference for High Performance Computing, Networking, Storage and Analysis (SC) (2010).

Digital Library

[30]

Kim, H., Lim, H., Jeong, J., Jo, H., and Lee, J. Task-aware virtual machine scheduling for i/o performance. In Proceedings of the 2009 ACM SIGPLAN/SIGOPS international conference on Virtual execution environments (VEE) (2009).

Digital Library

[31]

Leverich, J., and Kozyrakis, C. Reconciling high server utilization and sub-millisecond quality-of-service. In Proceedings of the Ninth European Conference on Computer Systems (EuroSys) (2014).

Digital Library

[32]

Martins, J., Ahmed, M., Raiciu, C., Olteanu, V., Honda, M., Bifulco, R., and Huici, F. Clickos and the art of network function virtualization. In Proceedings of the 11th USENIX Conference on Networked Systems Design and Implementation (NSDI) (2014).

Digital Library

[33]

Menon, A., Cox, A. L., and Zwaenepoel, W. Optimizing network virtualization in xen. In Proceedings of USENIX Annual Technical Conference (USENLX ATC) (2006).

[34]

Menon, A., Schubert, S., and Zwaenepoel, W. Twindrivers: semi-automatic derivation of fast and safe hypervisor network drivers from guest os drivers. In ACM International Conference on Architectural Support for Programming Languages and Operating Systems (ASPLOS) (2009).

Digital Library

[35]

Menon, A., and Zwaenepoel, W. Optimizing tcp receive performance. In Proceedings of USENIX Annual Technical Conference (USENIX ATC) (2008).

[36]

Ongaro, D., Cox, A. L., and Rixner, S. Scheduling i/o in virtual machine monitors. In Proceedings of the fourth ACM SIGPLAN/SIGOPS international conference on Virtual execution environments(VEE) (2008).

Digital Library

[37]

Popek, G. J., and Goldberg, R. P. Formal requirements for virtualizable third generation architectures. Commun. ACM 17, 7 (1974).

Digital Library

[38]

Saberi, A., Fu, Y., and Lin, Z. Hybrid-bridge: Efficiently bridging the semantic gap in virtual machine introspection via decoupled execution and training memoization. In Proceedings of the 21st Annual Network and Distributed System Security Symposium (NDSS) (2014).

[39]

Shim, H., and Lee, S.-M. Cfs-v: I/o demand-driven vm scheduler in kvm. Software R&D Center, Samsung Electronics (2014).

[40]

Suo, K., Rao, J., Cheng, L., and C. M. Lau, F. Time capsule: Tracing packet latency across different layers in virtualized systems. In Proceedings of the 7th ACM SIGOPS Asia-Pacific Workshop on Systems (APSys) (2016).

Digital Library

[41]

VMware. Vmware horizon view architecture planning 6.0. In VMware Technical White Paper (2014).

[42]

Waldspurger, C., and Rosenblum, M. I/O virtualization. Commun. ACM 55, 1 (Jan. 2012).

Digital Library

[43]

Waldspurger, C. A. Memory resource management in vmware ESX server. In Proceedings of the Symposium on Operating System Design and Implementation (OSDI) (2002).

[44]

Wang, G., and Ng, T. S. E. The impact of virtualization on network performance of amazon ec2 data center. In Proceedings of the 29th Conference on Information Communications (INFOCOM) (2010).

Digital Library

[45]

Xu, C., Gamage, S., Lu, H., Kompella, R., and Xu, D. vturbo: Accelerating virtual machine i/o processing using designated turbo-sliced core. In Proceedings of the 2013 USENIX Conference on Annual Technical Conference (ATC) (2013).

[46]

Xu, C., Gamage, S., Rao, P. N., Kangarlou, A., Kompella, R. R., and Xu, D. vSlicer: latency-aware virtual machine scheduling via differentiated-frequency cpu slicing. In Proceedings of the international symposium on High-Performance Parallel and Distributed Computing (HPDC) (2012).

Digital Library

[47]

Xu, Y., Bailey, M., Noble, B., and Jahanian, F. Small is better: Avoiding latency traps in virtualized data centers. In Proceedings of the 4th annual Symposium on Cloud Computing (SoCC) (2013).

Digital Library

Cited By

Suo KVu LIslam MDhar NNguyen THe SWu XLo DGamess E(2024)A Systematic Investigation of Hardware and Software in Electric Vehicular PlatformProceedings of the 2024 ACM Southeast Conference10.1145/3603287.3651203(9-17)Online publication date: 18-Apr-2024
https://dl.acm.org/doi/10.1145/3603287.3651203
Wang LYang JZhai JYang G(2024)Optimizing I/O Performance Through Effective vCPU Scheduling Interference ManagementIEEE Transactions on Parallel and Distributed Systems10.1109/TPDS.2023.332929835:12(2315-2330)Online publication date: Dec-2024
https://doi.org/10.1109/TPDS.2023.3329298
Xing TXiong CYe CWei QPicorel JBarbalace A(2023)Maximizing VMs' IO Performance on Overcommitted CPUs with FairnessProceedings of the 2023 ACM Symposium on Cloud Computing10.1145/3620678.3624649(93-108)Online publication date: 30-Oct-2023
https://dl.acm.org/doi/10.1145/3620678.3624649
Show More Cited By

Index Terms

Preserving I/O prioritization in virtualized OSes
1. General and reference
  1. Cross-computing tools and techniques
    1. Performance
2. Software and its engineering
  1. Software organization and properties
    1. Contextual software domains
      1. Operating systems

Recommendations

Scheduler activations for interference-resilient SMP virtual machine scheduling
Middleware '17: Proceedings of the 18th ACM/IFIP/USENIX Middleware Conference

The wide adoption of SMP virtual machines (VMs) and resource consolidation present challenges to efficiently executing multi-threaded programs in the cloud. An important problem is the semantic gaps between the guest OS and the hypervisor. The well-...
Transparently bridging semantic gap in CPU management for virtualized environments

Consolidated environments are progressively accommodating diverse and unpredictable workloads in conjunction with virtual desktop infrastructure and cloud computing. Unpredictable workloads, however, aggravate the semantic gap between the virtual ...
Performance Analysis of Network I/O Workloads in Virtualized Data Centers

Server consolidation and application consolidation through virtualization are key performance optimizations in cloud-based service delivery industry. In this paper, we argue that it is important for both cloud consumers and cloud providers to understand ...

Comments

Please enable JavaScript to view thecomments powered by Disqus.

Information & Contributors

Information

Published In

cover image ACM Conferences

SoCC '17: Proceedings of the 2017 Symposium on Cloud Computing

September 2017

672 pages

ISBN:9781450350280

DOI:10.1145/3127479

Copyright © 2017 ACM.

Permission to make digital or hard copies of all or part of this work for personal or classroom use is granted without fee provided that copies are not made or distributed for profit or commercial advantage and that copies bear this notice and the full citation on the first page. Copyrights for components of this work owned by others than ACM must be honored. Abstracting with credit is permitted. To copy otherwise, or republish, to post on servers or to redistribute to lists, requires prior specific permission and/or a fee. Request permissions from [email protected]

Sponsors

Publisher

Association for Computing Machinery

New York, NY, United States

Publication History

Published: 24 September 2017

Permissions

Request permissions for this article.

Request Permissions

Check for updates

Author Tags

Qualifiers

Research-article

Funding Sources

Hong Kong Research Grants Council Collaborative Research Fund
U.S. National Science Foundation

Conference

SoCC '17

Sponsor:

SoCC '17: ACM Symposium on Cloud Computing

September 24 - 27, 2017

California, Santa Clara

Acceptance Rates

Overall Acceptance Rate 169 of 722 submissions, 23%

Contributors

Other Metrics

View Article Metrics

Bibliometrics & Citations

Bibliometrics

Article Metrics

11
Total Citations
View Citations
395
Total Downloads

Downloads (Last 12 months)8
Downloads (Last 6 weeks)2

Reflects downloads up to 23 Nov 2024

Other Metrics

View Author Metrics

Citations

Cited By

Suo KVu LIslam MDhar NNguyen THe SWu XLo DGamess E(2024)A Systematic Investigation of Hardware and Software in Electric Vehicular PlatformProceedings of the 2024 ACM Southeast Conference10.1145/3603287.3651203(9-17)Online publication date: 18-Apr-2024
https://dl.acm.org/doi/10.1145/3603287.3651203
Wang LYang JZhai JYang G(2024)Optimizing I/O Performance Through Effective vCPU Scheduling Interference ManagementIEEE Transactions on Parallel and Distributed Systems10.1109/TPDS.2023.332929835:12(2315-2330)Online publication date: Dec-2024
https://doi.org/10.1109/TPDS.2023.3329298
Xing TXiong CYe CWei QPicorel JBarbalace A(2023)Maximizing VMs' IO Performance on Overcommitted CPUs with FairnessProceedings of the 2023 ACM Symposium on Cloud Computing10.1145/3620678.3624649(93-108)Online publication date: 30-Oct-2023
https://dl.acm.org/doi/10.1145/3620678.3624649
Lee KLee KPark HHwang JYoo C(2023)Autothrottle: Satisfying Network Performance Requirements for ContainersIEEE Transactions on Cloud Computing10.1109/TCC.2022.318639711:2(2096-2109)Online publication date: 1-Apr-2023
https://doi.org/10.1109/TCC.2022.3186397
Mvondo DBarbalace ATchana AMuller G(2021)Tell me when you are sleepy and what may wake you up!Proceedings of the ACM Symposium on Cloud Computing10.1145/3472883.3487013(562-569)Online publication date: 1-Nov-2021
https://dl.acm.org/doi/10.1145/3472883.3487013
Lei JMunikar MSuo KLu HRao JBarbalace ABhatotia PAlvisi LCadar C(2021)Parallelizing packet processing in container overlay networksProceedings of the Sixteenth European Conference on Computer Systems10.1145/3447786.3456241(261-276)Online publication date: 21-Apr-2021
https://dl.acm.org/doi/10.1145/3447786.3456241
Suo KShi YHung CBobbie PHung CHong JBechini ASong E(2021)Quantifying context switch overhead of artificial intelligence workloads on the cloud and edgesProceedings of the 36th Annual ACM Symposium on Applied Computing10.1145/3412841.3441993(1182-1189)Online publication date: 22-Mar-2021
https://dl.acm.org/doi/10.1145/3412841.3441993
Gong XCao DLi YLiu XLi YZhang JLi T(2021)A Thread Level SLO-Aware I/O Framework for Embedded VirtualizationIEEE Transactions on Parallel and Distributed Systems10.1109/TPDS.2020.302604232:3(500-513)Online publication date: 1-Mar-2021
https://doi.org/10.1109/TPDS.2020.3026042
Suo KSon JCheng DChen WBaidya S(2021)Tackling Cold Start of Serverless Applications by Efficient and Adaptive Container Runtime Reusing2021 IEEE International Conference on Cluster Computing (CLUSTER)10.1109/Cluster48925.2021.00018(433-443)Online publication date: Sep-2021
https://doi.org/10.1109/Cluster48925.2021.00018
Jia WShan JLi TShang XCui HDing XGavrilovska AZadok E(2020)vSMT-IOProceedings of the 2020 USENIX Conference on Usenix Annual Technical Conference10.5555/3489146.3489176(449-463)Online publication date: 15-Jul-2020
https://dl.acm.org/doi/10.5555/3489146.3489176
Show More Cited By

View Options

Login options

Check if you have access through your login credentials or your institution to get full access on this article.

Full Access

Get this Publication

View options

PDF

View or Download as a PDF file.

eReader

View online with eReader.

Media

Figures

Other

Tables

View Table of Contents