short-paper

Shoot4U: Using VMM Assists to Optimize TLB Operations on Preempted vCPUs

Authors:

Jiannan Ouyang,

Haoqiang ZhengAuthors Info & Claims

VEE '16: Proceedings of the12th ACM SIGPLAN/SIGOPS International Conference on Virtual Execution Environments

Pages 17 - 23

https://doi.org/10.1145/2892242.2892245

Published: 25 March 2016 Publication History

Abstract

Virtual Machine based approaches to workload consolidation, as seen in IaaS cloud as well as datacenter platforms, have long had to contend with performance degradation caused by synchronization primitives inside the guest environments. These primitives can be affected by virtual CPU preemptions by the host scheduler that can introduce delays that are orders of magnitude longer than those primitives were designed for. While a significant amount of work has focused on the behavior of spinlock primitives as a source of these performance issues, spinlocks do not represent the entirety of synchronization mechanisms that are susceptible to scheduling issues when running in a virtualized environment. In this paper we address the virtualized performance issues introduced by TLB shootdown operations. Our profiling study, based on the PARSEC benchmark suite, has shown that up to 64% of a VM's CPU time can be spent on TLB shootdown operations under certain workloads. In order to address this problem, we present a paravirtual TLB shootdown scheme named Shoot4U. Shoot4U completely eliminates TLB shootdown preemptions by invalidating guest TLB entries from the VMM and allowing guest TLB shootdown operations to complete without waiting for remote virtual CPUs to be scheduled. Our performance evaluation using the PARSEC benchmark suite demonstrates that Shoot4U can reduce benchmark runtime by up to 85% compared an unmodified Linux kernel, and up to 44% over a state-of-the-art paravirtual TLB shootdown scheme.

References

[1]

Linux Control Groups (cgroups). https://www.kernel.org/doc/Documentation/cgroups/cgroups.txt.

[2]

Gartner Says Efficient Data Center Design Can Lead to 300 Percent Capacity Growth in 60 Percent Less Space. http://www.gartner.com/newsroom/id/1472714.

[3]

ktap: A lightweight script-based dynamic tracing tool for Linux. http://www.ktap.org/.

[4]

KVM Paravirt Remote Flush TLB. https://lwn.net/Articles/500188/.

[5]

The PARSEC Benchmark Suite. http://parsec.cs.princeton.edu/.

[6]

perf: Linux Profiling with Performance Counters. https://perf.wiki.kernel.org/.

[7]

Sysbench. https://github.com/akopytov/sysbench.

[8]

Vmware(r) vsere(tm): The cpu scheduler in vmware esx(r) 4.1. Technical report, VMware, Inc, 2010.

[9]

le]Barroso13L. A. Barroso, J. Clidaras, and U. Hölzle. The Datacenter as a Computer: An Introduction to the Design of Warehouse-Scale Machines. Synthesis Lectures on Computer Architecture, 2013.

Digital Library

[10]

X. Ding, P. B. Gibbons, M. A. Kozuch, and J. Shan. Gleaner: Mitigating the Blocked-Waiter Wakeup Problem for Virtualized Multicore Applications. In Proc. 2014 USENIX Conference on USENIX Annual Technical Conference (USENIX ATC), iladelia, PA, June 2014. USENIX Association. URL https://www.usenix.org/conference/atc14/technical-sessions/presentation/ding.

[11]

T. Friebel. How to Deal with Lock-Holder Preemption. Presented at the Xen Summit North America, 2008.

[12]

J. Kaplan, W. Forrest, and N. Kindler. Revolutionizing Data Center Energy Efficiency. Technical report, McKinsey & Company, 2008.

[13]

H. Kim, S. Kim, J. Jeong, J. Lee, and S. Maeng. Demand-based Coordinated Scheduling for SMP VMs. In Proc. International Conference on Architectural Support for Programming Languages and Operating Systems (ASPLOS), 2013.

Digital Library

[14]

D. Lo, L. Cheng, R. Govindaraju, P. Ranganathan, and C. Kozyrakis. Heracles: Improving Resource Efficiency at Scale. In Proc. of the 42nd Annual International Symposium on Computer Architecture (ISCA), ISCA '15, 2015. 10.1145/2749469.2749475. URL http://doi.acm.org/10.1145/2749469.2749475.

Digital Library

[15]

J. Ousterhout. Scheduling Techniques for Concurrent Systems. In Proc. 3rd International Conference on Distributed Computing Systems, 1982.

[16]

J. Ouyang and J. R. Lange. Preemptable Ticket Spinlocks: Improving Consolidated Performance in the Cloud. In Proc. 9th ACM SIGPLAN/SIGOPS International Conference on Virtual Execution Environments (VEE), 2013.

Digital Library

[17]

K. Raghavendra and J. Fitzhardinge. Paravirtualized ticket spinlocks, May 2012. URL http://lwn.net/Articles/495597/.

[18]

C. Reiss, A. Tumanov, G. R. Ganger, R. H. Katz, and M. A. Kozuch. Heterogeneity and Dynamicity of Clouds at Scale: Google Trace Analysis. In Proc. 3rd ACM Symposium on Cloud Computing (SoCC), 2012. ISBN 978--1--4503--1761-0. 10.1145/2391229.2391236. URL http://doi.acm.org/10.1145/2391229.2391236.

Digital Library

[19]

R. v. Riel. Directed yield for pause loop exiting, 2011. URL http://lwn.net/Articles/424960/.

[20]

O. Sukwong and H. S. Kim. Is Co-scheduling Too Expensive for SMP VMs? In Proc. 6th European Conference on Computer Systems (EuroSys), 2011.

Digital Library

[21]

V. Uhlig, J. LeVasseur, E. Skoglund, and U. Dannowski. Towards Scalable Multiprocessor Virtual Machines. In Proc. 3rd conference on Virtual Machine Research And Technology Symposium, 2004.

Digital Library

[22]

C. Weng, Q. Liu, L. Yu, and M. Li. Dynamic Adaptive Scheduling for Virtual Machines. In Proc. 20th International Symposium on High Performance Parallel and Distributed Computing (HPDC), 2011.

Digital Library

[23]

L. Zhang, Y. Chen, Y. Dong, and C. Liu. Lock-Visor: An Efficient Transitory Co-scheduling for MP Guest. In Proc. 41st International Conference on Parallel Processing (ICPP), 2012.

Digital Library

Cited By

Ishiguro KYasuno NAublin PKono K(2023)Revisiting VM-Agnostic KVM vCPU Scheduler for Mitigating Excessive vCPU SpinningIEEE Transactions on Parallel and Distributed Systems10.1109/TPDS.2023.329768834:10(2615-2628)Online publication date: Oct-2023
https://doi.org/10.1109/TPDS.2023.3297688
Ishiguro KYasuno NAublin PKono KTitzer BXu HZhang I(2021)Mitigating excessive vCPU spinning in VM-agnostic KVMProceedings of the 17th ACM SIGPLAN/SIGOPS International Conference on Virtual Execution Environments10.1145/3453933.3454020(139-152)Online publication date: 7-Apr-2021
https://dl.acm.org/doi/10.1145/3453933.3454020
Schildermans SShan JAerts KJackrel JDing X(2021)Virtualization Overhead of Multithreading in X86 State-of-the-Art & Remaining ChallengesIEEE Transactions on Parallel and Distributed Systems10.1109/TPDS.2021.306470932:10(2557-2570)Online publication date: 1-Oct-2021
https://doi.org/10.1109/TPDS.2021.3064709
Show More Cited By

Index Terms

Shoot4U: Using VMM Assists to Optimize TLB Operations on Preempted vCPUs
1. Software and its engineering
  1. Software organization and properties
    1. Contextual software domains
      1. Operating systems
        Memory management
        Virtual memory
        Process management
        Multithreading
        Scheduling
      2. Software infrastructure
        Virtual machines

Recommendations

Performance Implications of Extended Page Tables on Virtualized x86 Processors
VEE '16: Proceedings of the12th ACM SIGPLAN/SIGOPS International Conference on Virtual Execution Environments

Managing virtual memory is an expensive operation, and becomes even more expensive on virtualized servers. Process- ing TLB misses on a virtualized x86 server requires a two-dimensional page walk that can have 6x more page table lookups, hence 6x more ...
Shoot4U: Using VMM Assists to Optimize TLB Operations on Preempted vCPUs
VEE '16

Virtual Machine based approaches to workload consolidation, as seen in IaaS cloud as well as datacenter platforms, have long had to contend with performance degradation caused by synchronization primitives inside the guest environments. These primitives ...
DiDi: Mitigating the Performance Impact of TLB Shootdowns Using a Shared TLB Directory
PACT '11: Proceedings of the 2011 International Conference on Parallel Architectures and Compilation Techniques

Translation Look aside Buffers (TLBs) are ubiquitously used in modern architectures to cache virtual-to-physical mappings and, as they are looked up on every memory access, are paramount to performance scalability. The emergence of chip-multiprocessors (...

Comments

Please enable JavaScript to view thecomments powered by Disqus.

Information & Contributors

Information

Published In

cover image ACM Conferences

VEE '16: Proceedings of the12th ACM SIGPLAN/SIGOPS International Conference on Virtual Execution Environments

March 2016

186 pages

ISBN:9781450339476

DOI:10.1145/2892242

General Chair:
Vishakha Gupta-Cledat
Intel Labs
,
Program Chairs:
Donald E. Porter
Stony Brook University
,
Vivek Sarkar
Rice University

ACM SIGPLAN Notices Volume 51, Issue 7
VEE '16
July 2016
167 pages
ISSN:0362-1340
EISSN:1558-1160
DOI:10.1145/3007611
Editor:
Matthew Fluet
Issue’s Table of Contents

Copyright © 2016 ACM.

Permission to make digital or hard copies of all or part of this work for personal or classroom use is granted without fee provided that copies are not made or distributed for profit or commercial advantage and that copies bear this notice and the full citation on the first page. Copyrights for components of this work owned by others than ACM must be honored. Abstracting with credit is permitted. To copy otherwise, or republish, to post on servers or to redistribute to lists, requires prior specific permission and/or a fee. Request permissions from [email protected]

Sponsors

Publisher

Association for Computing Machinery

New York, NY, United States

Publication History

Published: 25 March 2016

Permissions

Request permissions for this article.

Request Permissions

Check for updates

Author Tags

Qualifiers

Short-paper

Conference

VEE '16

Sponsor:

VEE '16: 12th ACM SIGPLAN/SIGOPS International Conference on Virtual Execution Environments

April 2 - 3, 2016

Georgia, Atlanta, USA

Acceptance Rates

VEE '16 Paper Acceptance Rate 10 of 29 submissions, 34%;

Overall Acceptance Rate 80 of 235 submissions, 34%

Contributors

Other Metrics

View Article Metrics

Bibliometrics & Citations

Bibliometrics

Article Metrics

16
Total Citations
View Citations
452
Total Downloads

Downloads (Last 12 months)9
Downloads (Last 6 weeks)2

Reflects downloads up to 14 Nov 2024

Other Metrics

View Author Metrics

Citations

Cited By

Ishiguro KYasuno NAublin PKono K(2023)Revisiting VM-Agnostic KVM vCPU Scheduler for Mitigating Excessive vCPU SpinningIEEE Transactions on Parallel and Distributed Systems10.1109/TPDS.2023.329768834:10(2615-2628)Online publication date: Oct-2023
https://doi.org/10.1109/TPDS.2023.3297688
Ishiguro KYasuno NAublin PKono KTitzer BXu HZhang I(2021)Mitigating excessive vCPU spinning in VM-agnostic KVMProceedings of the 17th ACM SIGPLAN/SIGOPS International Conference on Virtual Execution Environments10.1145/3453933.3454020(139-152)Online publication date: 7-Apr-2021
https://dl.acm.org/doi/10.1145/3453933.3454020
Schildermans SShan JAerts KJackrel JDing X(2021)Virtualization Overhead of Multithreading in X86 State-of-the-Art & Remaining ChallengesIEEE Transactions on Parallel and Distributed Systems10.1109/TPDS.2021.306470932:10(2557-2570)Online publication date: 1-Oct-2021
https://doi.org/10.1109/TPDS.2021.3064709
Kim TPark CHuh JAhn J(2020)Reconciling Time Slice Conflicts of Virtual Machines With Dual Time Slice for CloudsIEEE Transactions on Parallel and Distributed Systems10.1109/TPDS.2020.299325231:10(2453-2465)Online publication date: 1-Oct-2020
https://doi.org/10.1109/TPDS.2020.2993252
Schildermans SAerts KShan JDing X(2020)Ptlbmalloc2: Reducing TLB Shootdowns with High Memory Efficiency2020 IEEE Intl Conf on Parallel & Distributed Processing with Applications, Big Data & Cloud Computing, Sustainable Computing & Communications, Social Computing & Networking (ISPA/BDCloud/SocialCom/SustainCom)10.1109/ISPA-BDCloud-SocialCom-SustainCom51426.2020.00036(76-83)Online publication date: Dec-2020
https://doi.org/10.1109/ISPA-BDCloud-SocialCom-SustainCom51426.2020.00036
Yan ZVeselý JCox GBhattacharjee A(2018)Hardware Translation Coherence for Virtualized SystemsACM SIGOPS Operating Systems Review10.1145/3273982.327398852:1(57-70)Online publication date: 28-Aug-2018
https://dl.acm.org/doi/10.1145/3273982.3273988
Ahn JPark CHeo THuh JOliveira RFelber PHu Y(2018)Accelerating critical OS services in virtualized systems with flexible micro-sliced coresProceedings of the Thirteenth EuroSys Conference10.1145/3190508.3190521(1-14)Online publication date: 23-Apr-2018
https://dl.acm.org/doi/10.1145/3190508.3190521
Kilic ODoddamani SBhat ABagdi HGopalan K(2018)Overcoming Virtualization Overheads for Large-vCPU Virtual Machines2018 IEEE 26th International Symposium on Modeling, Analysis, and Simulation of Computer and Telecommunication Systems (MASCOTS)10.1109/MASCOTS.2018.00042(369-380)Online publication date: Sep-2018
https://doi.org/10.1109/MASCOTS.2018.00042
Yan ZVeselý JCox GBhattacharjee A(2017)Hardware Translation Coherence for Virtualized SystemsACM SIGARCH Computer Architecture News10.1145/3140659.308021145:2(430-443)Online publication date: 24-Jun-2017
https://dl.acm.org/doi/10.1145/3140659.3080211
Amit NWei MTu C(2017)HypercallbacksACM SIGOPS Operating Systems Review10.1145/3139645.313965451:1(54-59)Online publication date: 11-Sep-2017
https://dl.acm.org/doi/10.1145/3139645.3139654
Show More Cited By

View Options

Get Access

Login options

Check if you have access through your login credentials or your institution to get full access on this article.

Full Access

Get this Publication

View options

PDF

View or Download as a PDF file.

eReader

View online with eReader.

Media

Figures

Other

Tables

View Table of Contents