research-article

A Comprehensive Implementation and Evaluation of Direct Interrupt Delivery

Authors:

Michael Ferdman,

Tzi-cker ChiuehAuthors Info & Claims

VEE '15: Proceedings of the 11th ACM SIGPLAN/SIGOPS International Conference on Virtual Execution Environments

Pages 1 - 15

https://doi.org/10.1145/2731186.2731189

Published: 14 March 2015 Publication History

Abstract

As the performance overhead associated with CPU and memory virtualization becomes largely negligible, research efforts are directed toward reducing the I/O virtualization overhead, which mainly comes from two sources: DMA set-up and payload copy, and interrupt delivery. The advent of SRIOV and MRIOV effectively reduces the DMA-related virtualization overhead to a minimum. Therefore, the last battleground for minimizing virtualization overhead is how to directly deliver every interrupt to its target VM without involving the hypervisor.

This paper describes the design, implementation, and evaluation of a KVM-based direct interrupt delivery system called DID. DID delivers interrupts from SRIOV devices, virtual devices, and timers to their target VMs directly, completely avoiding VM exits. Moreover, DID does not require any modifications to the VM's operating system and preserves the correct priority among interrupts in all cases. We demonstrate that DID reduces the number of VM exits by a factor of 100 for I/O-intensive workloads, decreases the interrupt invocation latency by 80%, and improves the throughput of a VM running Memcached by a factor of 3.

References

[1]

Enabling Optimized Interrupt/APIC Virtualization in KVM. KVM Forum 2012.

[2]

Fio - Flexible I/O Tester. http://freecode.com/projects/fio.

[3]

Introduction of AMD Advanced Virtual Interrupt Controller. XenSummit 2012.

[4]

Jailhouse Partitioning Hypervisor. https://github.com/siemens/jailhouse.

[5]

Memcached: memory object caching system. http://memcached.org/.

[6]

Secure virtual machine architecture reference manual. AMD.

[7]

Single-Root I/O Virtualization and Sharing Specification, Revision 1.0, PCI-SIG.

[8]

SIPp: traffic generator for the SIP protocol. http://sipp.sourceforge.net/.

[9]

Sippy B2BUA. http://www.b2bua.org/.

[10]

virtio- and vhost-net need for speed performance challenges. KVM Forum 2010.

[11]

Intel 64 Architecture x2APIC Specification, Intel Corporation, 2008.

[12]

Single-Root I/O Virtualization and Sharing Specification, Revision 1.0, PCI-SIG, 2008.

[13]

Keith Adams and Ole Agesen. A comparison of software and hardware techniques for x86 virtualization. In ACM ASPLOS'06.

Digital Library

[14]

Ole Agesen, Jim Mattson, Radu Rugina, and Jeffrey Sheldon. Software techniques for avoiding hardware virtualization exits. In USENIX Annual Technical Conference, pages 373--385, 2012.

Digital Library

[15]

Muli Ben-Yehuda, Michael D Day, Zvi Dubitzky, Michael Factor, Nadav Har'El, Abel Gordon, Anthony Liguori, Orit Wasserman, and Ben-Ami Yassour. The turtles project: Design and implementation of nested virtualization. In OSDI, volume 10, pages 423--436, 2010.

Digital Library

[16]

Christoffer Dall and Jason Nieh. Kvm/arm: Experiences building the linux arm hypervisor. 2013.

[17]

Michael Ferdman, Almutaz Adileh, Onur Kocberber, Stavros Volos, Mohammad Alisafaee, Djordje Jevdjic, Cansu Kaynak, Adrian Daniel Popescu, Anastasia Ailamaki, and Babak Falsafi. Clearing the Clouds: A Study of Emerging Scale-out Workloads on Modern Hardware. volume 40, pages 37--48. ACM, 2012.

Digital Library

[18]

Abel Gordon, Nadav Amit, Nadav Har'El, Muli Ben-Yehuda, Alex Landau, Assaf Schuster, and Dan Tsafrir. Eli: bare-metal performance for i/o virtualization. ACM SIGARCH Computer Architecture News, 40(1):411--422, 2012.

Digital Library

[19]

Nadav Har'El, Abel Gordon, Alex Landau, Muli Ben-Yehuda, Avishay Traeger, and Razya Ladelsky. Efficient and scalable paravirtual i/o system. In USENIX Annual Technical Conference, pages 231--242, 2013.

Digital Library

[20]

R. Hiremane. Intel Virtualization Technology for Directed I/O (Intel VT-d). Technology@ Intel Magazine, 2007.

[21]

DPDK Intel. Intel data plane development kit.

[22]

Eric Keller, Jakub Szefer, Jennifer Rexford, and Ruby B Lee. Nohype: virtualized cloud infrastructure without the virtualization. In ACM SIGARCH Computer Architecture News, volume 38, pages 350--361. ACM, 2010.

Digital Library

[23]

Jan Kiszka. Towards linux as a real-time hypervisor. RTLWS11, 2009.

[24]

Avi Kivity, Yaniv Kamay, Dor Laor, Uri Lublin, and Anthony Liguori. kvm: the linux virtual machine monitor. In Proceedings of the Linux Symposium, volume 1, pages 225--230, 2007.

[25]

Kaushik Kumar Ram, Jose Renato Santos, Yoshio Turner, Alan L Cox, and Scott Rixner. Achieving 10 gb/s using safe and transparent network interface virtualization. In Proceedings of the 2009 ACM SIGPLAN/SIGOPS international conference on Virtual execution environments, pages 61--70. ACM, 2009.

Digital Library

[26]

Rusty Russell. virtio: towards a de-facto standard for virtual i/o devices. SIGOPS Oper. Syst. Rev. 2008.

Digital Library

[27]

Jose Renato Santos, Yoshio Turner, G John Janakiraman, and Ian Pratt. Bridging the gap between software and hardware techniques for i/o virtualization. In USENIX Annual Technical Conference, pages 29--42, 2008.

Digital Library

[28]

Quinn O Snell, Armin R Mikler, and John L Gustafson. Netpipe: A network protocol independent performance evaluator. In IASTED International Conference on Intelligent Information Management and Systems, volume 6. Washington, DC, USA), 1996.

[29]

Jakub Szefer, Eric Keller, Ruby B Lee, and Jennifer Rexford. Eliminating the hypervisor attack surface for a more secure cloud. In Proceedings of the 18th ACM conference on Computer and communications security, pages 401--412. ACM, 2011.

Digital Library

[30]

Ajay Tirumala, Feng Qin, Jon Dugan, Jim Ferguson, and Kevin Gibbs. Iperf: The tcp/udp bandwidth measurement tool. http://dast.nlanr.net/Projects, 2005.

[31]

Hitachi Tomoki Sekiyama, Yokohama Research Lab. Improvement of real-time performance of kvm.

[32]

Cheng-Chun Tu, Chao-tang Lee, and Tzi-cker Chiueh. Secure i/o device sharing among virtual machines on multiple hosts. In Proceedings of the 40th Annual International Symposium on Computer Architecture, pages 108--119. ACM, 2013.

Digital Library

[33]

Rich Uhlig, Gil Neiger, Dion Rodgers, Amy L Santoni, Fernando CM Martins, Andrew V Anderson, Steven M Bennett, Alain Kagi, Felix H Leung, and Larry Smith. Intel virtualization technology. Computer, 38(5):48--56, 2005.

Digital Library

[34]

Rafal Wojtczuk and Joanna Rutkowska. Following the white rabbit: Software attacks against intel vt-d technology.

Cited By

Yasukata KTazaki HAublin PAamodt TJerger NSwift M(2023)Exit-Less, Isolated, and Shared Access for Virtual MachinesProceedings of the 28th ACM International Conference on Architectural Support for Programming Languages and Operating Systems, Volume 310.1145/3582016.3582042(224-237)Online publication date: 25-Mar-2023
https://dl.acm.org/doi/10.1145/3582016.3582042
Hu XLi JMa RGuan H(2022)ES2: Building an Efficient and Responsive Event Path for I/O VirtualizationIEEE Transactions on Cloud Computing10.1109/TCC.2020.296966010:2(1358-1372)Online publication date: 1-Apr-2022
https://doi.org/10.1109/TCC.2020.2969660
Tian KZhang YKang LZhao YDong YGavrilovska AZadok E(2020)coIOMMUProceedings of the 2020 USENIX Conference on Usenix Annual Technical Conference10.5555/3489146.3489178(479-492)Online publication date: 15-Jul-2020
https://dl.acm.org/doi/10.5555/3489146.3489178
Show More Cited By

Index Terms

A Comprehensive Implementation and Evaluation of Direct Interrupt Delivery
1. Hardware
  1. Communication hardware, interfaces and storage

Recommendations

A Comprehensive Implementation and Evaluation of Direct Interrupt Delivery
VEE '15

As the performance overhead associated with CPU and memory virtualization becomes largely negligible, research efforts are directed toward reducing the I/O virtualization overhead, which mainly comes from two sources: DMA set-up and payload copy, and ...
ELI: bare-metal performance for I/O virtualization
ASPLOS XVII: Proceedings of the seventeenth international conference on Architectural Support for Programming Languages and Operating Systems

Direct device assignment enhances the performance of guest virtual machines by allowing them to communicate with I/O devices without host involvement. But even with device assignment, guests are still unable to approach bare-metal performance, because ...
ELI: bare-metal performance for I/O virtualization
ASPLOS '12

Direct device assignment enhances the performance of guest virtual machines by allowing them to communicate with I/O devices without host involvement. But even with device assignment, guests are still unable to approach bare-metal performance, because ...

Comments

Please enable JavaScript to view thecomments powered by Disqus.

Information & Contributors

Information

Published In

cover image ACM Conferences

VEE '15: Proceedings of the 11th ACM SIGPLAN/SIGOPS International Conference on Virtual Execution Environments

March 2015

238 pages

ISBN:9781450334501

DOI:10.1145/2731186

General Chair:
Ada Gavrilovska
Georgia Tech
,
Program Chairs:
Angela Demke Brown
University of Toronto
,
Bjarne Steensgaard
Microsoft

ACM SIGPLAN Notices Volume 50, Issue 7
VEE '15
July 2015
221 pages
ISSN:0362-1340
EISSN:1558-1160
DOI:10.1145/2817817
Editor:
Andy Gill
University of Kansas, Lawrence, KS
Issue’s Table of Contents

Copyright © 2015 ACM.

Permission to make digital or hard copies of all or part of this work for personal or classroom use is granted without fee provided that copies are not made or distributed for profit or commercial advantage and that copies bear this notice and the full citation on the first page. Copyrights for components of this work owned by others than ACM must be honored. Abstracting with credit is permitted. To copy otherwise, or republish, to post on servers or to redistribute to lists, requires prior specific permission and/or a fee. Request permissions from [email protected]

Sponsors

Publisher

Association for Computing Machinery

New York, NY, United States

Publication History

Published: 14 March 2015

Permissions

Request permissions for this article.

Request Permissions

Check for updates

Author Tags

Qualifiers

Research-article

Conference

VEE '15

Sponsor:

VEE '15: 11th ACM SIGPLAN/SIGOPS International Conference on Virtual Execution Environments

March 14 - 15, 2015

Istanbul, Turkey

Acceptance Rates

VEE '15 Paper Acceptance Rate 16 of 50 submissions, 32%;

Overall Acceptance Rate 80 of 235 submissions, 34%

Contributors

Other Metrics

View Article Metrics

Bibliometrics & Citations

Bibliometrics

Article Metrics

32
Total Citations
View Citations
899
Total Downloads

Downloads (Last 12 months)45
Downloads (Last 6 weeks)7

Reflects downloads up to 21 Nov 2024

Other Metrics

View Author Metrics

Citations

Cited By

Yasukata KTazaki HAublin PAamodt TJerger NSwift M(2023)Exit-Less, Isolated, and Shared Access for Virtual MachinesProceedings of the 28th ACM International Conference on Architectural Support for Programming Languages and Operating Systems, Volume 310.1145/3582016.3582042(224-237)Online publication date: 25-Mar-2023
https://dl.acm.org/doi/10.1145/3582016.3582042
Hu XLi JMa RGuan H(2022)ES2: Building an Efficient and Responsive Event Path for I/O VirtualizationIEEE Transactions on Cloud Computing10.1109/TCC.2020.296966010:2(1358-1372)Online publication date: 1-Apr-2022
https://doi.org/10.1109/TCC.2020.2969660
Tian KZhang YKang LZhao YDong YGavrilovska AZadok E(2020)coIOMMUProceedings of the 2020 USENIX Conference on Usenix Annual Technical Conference10.5555/3489146.3489178(479-492)Online publication date: 15-Jul-2020
https://dl.acm.org/doi/10.5555/3489146.3489178
Narayanan VHuang YTan GJaeger TBurtsev ANagarakatte SBaumann AKasikci B(2020)Lightweight kernel isolation with virtualization and VM functionsProceedings of the 16th ACM SIGPLAN/SIGOPS International Conference on Virtual Execution Environments10.1145/3381052.3381328(157-171)Online publication date: 17-Mar-2020
https://dl.acm.org/doi/10.1145/3381052.3381328
Cheng KDoddamani SChiueh TLi YGopalan KNagarakatte SBaumann AKasikci B(2020)DirectvisorProceedings of the 16th ACM SIGPLAN/SIGOPS International Conference on Virtual Execution Environments10.1145/3381052.3381317(45-58)Online publication date: 17-Mar-2020
https://dl.acm.org/doi/10.1145/3381052.3381317
Lim JNieh JLarus JCeze LStrauss K(2020)Optimizing Nested Virtualization Performance Using Direct Virtual HardwareProceedings of the Twenty-Fifth International Conference on Architectural Support for Programming Languages and Operating Systems10.1145/3373376.3378467(557-574)Online publication date: 9-Mar-2020
https://dl.acm.org/doi/10.1145/3373376.3378467
Vilanova LAmit NEtsion YManne SHunter HAltman E(2019)Using SMT to accelerate nested virtualizationProceedings of the 46th International Symposium on Computer Architecture10.1145/3307650.3322261(750-761)Online publication date: 22-Jun-2019
https://dl.acm.org/doi/10.1145/3307650.3322261
Li JXue SZhang WMa RQi ZGuan H(2019)When I/O Interrupt Becomes System Bottleneck: Efficiency and Scalability Enhancement for SR-IOV Network VirtualizationIEEE Transactions on Cloud Computing10.1109/TCC.2017.27126867:4(1183-1196)Online publication date: 1-Oct-2019
https://doi.org/10.1109/TCC.2017.2712686
Auernhammer FArndt R(2018)XIVEIBM Journal of Research and Development10.1147/JRD.2018.284559962:4-5(5:1-5:10)Online publication date: 1-Jul-2018
https://dl.acm.org/doi/10.1147/JRD.2018.2845599
Zhao SDing X(2018)FIMCEACM Transactions on Privacy and Security10.1145/319518121:3(1-30)Online publication date: 21-May-2018
https://dl.acm.org/doi/10.1145/3195181
Show More Cited By

View Options

Login options

Check if you have access through your login credentials or your institution to get full access on this article.

Full Access

Get this Publication

View options

PDF

View or Download as a PDF file.

eReader

View online with eReader.

Media

Figures

Other

Tables

View Table of Contents