research-article

IBMon: monitoring VMM-bypass capable InfiniBand devices using memory introspection

Authors:

Ada Gavrilovska,

Karsten SchwanAuthors Info & Claims

HPCVirt '09: Proceedings of the 3rd ACM Workshop on System-level Virtualization for High Performance Computing

Pages 25 - 32

https://doi.org/10.1145/1519138.1519142

Published: 31 March 2009 Publication History

Abstract

Active monitoring of virtual machine (VM) behaviors and their utilization of different resource types is critical for effective management of high end cluster machines, data centers, and cloud computing infrastructures. Unfortunately, for reasons of performance, certain types of existing and future devices support capabilities that provide VMs with direct access to device resources, thereby bypassing the virtual machine monitor (VMM). This presents significant challenges to the VMM due to its resulting inability to assess VMdevice interactions.

This paper describes a monitoring utility, IBMon, which enables the asynchronous monitoring of virtualized Infini-Band devices -- a sample VMM-bypass device heavily used in the HPC community. In the absence of adequate hardware supported monitoring information, IBMon uses dynamic memory introspection techniques to infer information regarding the VM-device interactions. Experimental results demonstrate that IBMon can asynchronously monitor VMM-bypass operations with acceptable accuracy, and negligible overheads, including for larger number of VMs, and for VMs with dynamic behavior patterns.

References

[1]

S. Agarwala and K. Schwan. SysProf: Online Distributed Behavior Diagnosis through Fine-grain System Monitoring. In ICDCS, 2006.

Digital Library

[2]

I. Cohen, S. Zhang, M. Goldszmidt, J. Symons, T. Kelly, and A. Fox. Capturing, indexing, clustering and retrieving system history. In SOSP, 2005.

Digital Library

[3]

T. Garfinkel and M. Rosenblum. A virtual machine introspection based architecture for intrusion detection. In Proceedings of the 2003 Network and Distributed System Symposium, 2003.

[4]

W. Huang, J. Liu, and D. Panda. A Case for High Performance Computing with Virtual Machines. In ICS, 2006.

Digital Library

[5]

N. L. P. Jr., T. Fraser, J. Molina, and W. A. Arbaugh. Copilot -a coprocessor-based kernel runtime integrity monitor. 2004.

[6]

M. Kesavan, A. Ranadive, A. Gavrilovska, and K. Schwan. Active CoordinaTion (ACT) - Towards Effectively Managing Virtualized Multicore Clouds. In Cluster 2008, 2008.

[7]

S. Kumar, V. Talwar, P. Ranganathan, R. Nathuji, and K. Schwan. M-Channels and M-Brokers: Coordinated Management in Virtualized Systems. In Workshop on Managed Multi-Core Systems, in conjunction with HPDC'08, 2008.

[8]

J. Liu, W. Huang, B. Abali, and D. K. Panda. High Performance VMM-Bypass I/O in Virtual Machines. In ATC, 2006.

Digital Library

[9]

Myricom, Inc. Myrinet. www.myri.com.

[10]

R. Nathuji and K. Schwan. VirtualPower: Coordinated Power Management in Virtualized Enterprise Systems. In SOSP, 2007.

Digital Library

[11]

OpenFabrics Software Stack - OFED 1.1. www.openfabrics.org/.

[12]

B. D. Payne, M. D. P. de Carbone, and W. Lee. Secure and Flexible Monitoring of Virtual Machines. In Computer Security Applications Conference, 2007.

[13]

I. Pratt, K. Fraser, S. Hand, C. Limpach, A. Warfield, D. Magenheimer, J. Nakajima, and A. Mallick. Xen 3.0 and the Art of Virtualization. In Ottawa Linux Symposium, 2005.

[14]

Quadrics, Ltd. QsNet. www.quadrics.com.

[15]

H. Raj and K. Schwan. High Performance and Scalable I/O Virtualization via Self-Virtualized Devices. In HPDC, 2007.

Digital Library

[16]

K. K. Ram, J. R. Santos, Y. Turner, A. L. Cox, and S. Rixner. Achieving 10Gbps using safe and transparent network interface virtualization. In VEE, 2009.

Digital Library

[17]

A. Ranadive, M. Kesavan, A. Gavrilovska, and K. Schwan. Performance Implications of Virtualizing Multicore Cluster Machines. In Workshop on HPC System Virtualization, in conjunction with Eurosys'08, 2008.

Digital Library

[18]

RDMA-enabled NIC. www.rdmaconsortium.org.

[19]

J. Sugerman, G. Venkitachalam, and B. H. Lim. Virtualizing I/O Devices on VMware Workstation's Hosted Virtual Machine Monitor. In USENIX, 2001.

Digital Library

[20]

B. Urgaonkar and P. Shenoy. Sharc: Managing CPU and Network Bandwidth in Shared Clusters. In IPDPS, 2004.

Digital Library

[21]

Virtual Machine Device Queues. www.intel.com/network-/connectivity/vtc_vmdq.html.

[22]

VMSafe. www.vmware.com/technology/security/vmsafe.html.

[23]

The VMWare ESX Server. http://www.vmware.com/products/esx/.

[24]

T.Wood, P. Shenoy, A. Venkataramani, and M. Yousif. Blackbox and Gray-box Strategies for Virtual Machine Migration. In NSDI, 2007.

Digital Library

[25]

Xen Credit Scheduler. wiki.xensource.com/xenwiki/CreditScheduler.

Cited By

Weng CLiu QLi KZou D(2016)CloudMon: Monitoring Virtual Machines in CloudsIEEE Transactions on Computers10.1109/TC.2016.2560809(1-1)Online publication date: 2016
https://doi.org/10.1109/TC.2016.2560809
Suneja SIsci Cde Lara EBala V(2015)Exploring VM IntrospectionACM SIGPLAN Notices10.1145/2817817.273119650:7(133-146)Online publication date: 14-Mar-2015
https://dl.acm.org/doi/10.1145/2817817.2731196
Suneja SIsci Cde Lara EBala VGavrilovska ABrown ASteensgaard B(2015)Exploring VM IntrospectionProceedings of the 11th ACM SIGPLAN/SIGOPS International Conference on Virtual Execution Environments10.1145/2731186.2731196(133-146)Online publication date: 14-Mar-2015
https://dl.acm.org/doi/10.1145/2731186.2731196
Show More Cited By

IBMon: monitoring VMM-bypass capable InfiniBand devices using memory introspection
1. General and reference
  1. Cross-computing tools and techniques
2. Software and its engineering
  1. Software organization and properties
    1. Software system structures

Recommendations

SRVM: Hypervisor Support for Live Migration with Passthrough SR-IOV Network Devices
VEE '16

Single-Root I/O Virtualization (SR-IOV) is a specification that allows a single PCI Express (PCIe) device (ysical function or PF) to be used as multiple PCIe devices (virtual functions or VF). In a virtualization system, each VF can be directly assigned ...
SRVM: Hypervisor Support for Live Migration with Passthrough SR-IOV Network Devices
VEE '16: Proceedings of the12th ACM SIGPLAN/SIGOPS International Conference on Virtual Execution Environments

Single-Root I/O Virtualization (SR-IOV) is a specification that allows a single PCI Express (PCIe) device (ysical function or PF) to be used as multiple PCIe devices (virtual functions or VF). In a virtualization system, each VF can be directly assigned ...
Enabling Instantaneous Relocation of Virtual Machines with a Lightweight VMM Extension
CCGRID '10: Proceedings of the 2010 10th IEEE/ACM International Conference on Cluster, Cloud and Grid Computing

We are developing an efficient resource management system with aggressive virtual machine (VM) relocation among physical nodes in a data center. Existing live migration technology, however, requires a long time to change the execution host of a VM, it ...

Comments

Please enable JavaScript to view thecomments powered by Disqus.

Information & Contributors

Information

Published In

cover image ACM Conferences

HPCVirt '09: Proceedings of the 3rd ACM Workshop on System-level Virtualization for High Performance Computing

March 2009

42 pages

ISBN:9781605584652

DOI:10.1145/1519138

General Chair:
Stephen L. Scott
Oak Ridge National Laboratory
,
Program Chair:
Geoffroy Vallée
Oak Ridge National Laboratory

Copyright © 2009 ACM.

Permission to make digital or hard copies of all or part of this work for personal or classroom use is granted without fee provided that copies are not made or distributed for profit or commercial advantage and that copies bear this notice and the full citation on the first page. Copyrights for components of this work owned by others than ACM must be honored. Abstracting with credit is permitted. To copy otherwise, or republish, to post on servers or to redistribute to lists, requires prior specific permission and/or a fee. Request permissions from [email protected]

Sponsors

SIGOPS: ACM Special Interest Group on Operating Systems

Publisher

Association for Computing Machinery

New York, NY, United States

Publication History

Published: 31 March 2009

Permissions

Request permissions for this article.

Request Permissions

Check for updates

Qualifiers

Research-article

Conference

EuroSys '09

Sponsor:

SIGOPS

EuroSys '09: Fourth EuroSys Conference 2009

March 31, 2009

Nuremburg, Germany

Upcoming Conference

EuroSys '25

Sponsor:
sigops

Twentieth European Conference on Computer Systems

March 30 - April 3, 2025

Rotterdam , Netherlands

Contributors

Other Metrics

View Article Metrics

Bibliometrics & Citations

Bibliometrics

Article Metrics

13
Total Citations
View Citations
385
Total Downloads

Downloads (Last 12 months)6
Downloads (Last 6 weeks)1

Reflects downloads up to 12 Nov 2024

Other Metrics

View Author Metrics

Citations

Cited By

Weng CLiu QLi KZou D(2016)CloudMon: Monitoring Virtual Machines in CloudsIEEE Transactions on Computers10.1109/TC.2016.2560809(1-1)Online publication date: 2016
https://doi.org/10.1109/TC.2016.2560809
Suneja SIsci Cde Lara EBala V(2015)Exploring VM IntrospectionACM SIGPLAN Notices10.1145/2817817.273119650:7(133-146)Online publication date: 14-Mar-2015
https://dl.acm.org/doi/10.1145/2817817.2731196
Suneja SIsci Cde Lara EBala VGavrilovska ABrown ASteensgaard B(2015)Exploring VM IntrospectionProceedings of the 11th ACM SIGPLAN/SIGOPS International Conference on Virtual Execution Environments10.1145/2731186.2731196(133-146)Online publication date: 14-Mar-2015
https://dl.acm.org/doi/10.1145/2731186.2731196
Li KZheng HWu JDu X(2015)Virtual machine placement in cloud systems through migration processInternational Journal of Parallel, Emergent and Distributed Systems10.1080/17445760.2014.93195730:5(393-410)Online publication date: 1-Sep-2015
https://dl.acm.org/doi/10.1080/17445760.2014.931957
Suneja SIsci CBala Vde Lara EMummert T(2014)Non-intrusive, out-of-band and out-of-the-box systems monitoring in the cloudACM SIGMETRICS Performance Evaluation Review10.1145/2637364.259200942:1(249-261)Online publication date: 16-Jun-2014
https://dl.acm.org/doi/10.1145/2637364.2592009
Suneja SIsci CBala Vde Lara EMummert TSanghavi SShakkottai SLelarge MSchroeder B(2014)Non-intrusive, out-of-band and out-of-the-box systems monitoring in the cloudThe 2014 ACM international conference on Measurement and modeling of computer systems10.1145/2591971.2592009(249-261)Online publication date: 16-Jun-2014
https://dl.acm.org/doi/10.1145/2591971.2592009
Li KZheng HWu J(2013)Migration-based virtual machine placement in cloud systems2013 IEEE 2nd International Conference on Cloud Networking (CloudNet)10.1109/CloudNet.2013.6710561(83-90)Online publication date: Nov-2013
https://doi.org/10.1109/CloudNet.2013.6710561
Ranadive AGavrilovska ASchwan K(2013)Distributed resource exchange: Virtualized resource management for SR-IOV InfiniBand clusters2013 IEEE International Conference on Cluster Computing (CLUSTER)10.1109/CLUSTER.2013.6702653(1-8)Online publication date: Sep-2013
https://doi.org/10.1109/CLUSTER.2013.6702653
(2011)ReferencesAttaining High Performance Communications10.1201/b10249-17(367-405)Online publication date: 7-Feb-2011
https://doi.org/10.1201/b10249-17
Ranadive AGavrilovska ASchwan K(2011)ResourceExchangeProceedings of the 2011 IEEE International Conference on Cluster Computing10.1109/CLUSTER.2011.14(45-53)Online publication date: 26-Sep-2011
https://dl.acm.org/doi/10.1109/CLUSTER.2011.14
Show More Cited By

View Options

Get Access

Login options

Check if you have access through your login credentials or your institution to get full access on this article.

Full Access

Get this Publication

View options

PDF

View or Download as a PDF file.

eReader

View online with eReader.

Media

Figures

Other

Tables

View Table of Contents