Nothing Special   »   [go: up one dir, main page]

skip to main content
10.1145/3620665.3640358acmconferencesArticle/Chapter ViewAbstractPublication PagesasplosConference Proceedingsconference-collections
research-article
Open access

Direct Memory Translation for Virtualized Clouds

Published: 27 April 2024 Publication History

Abstract

Virtual memory translation has become a key performance bottleneck of memory-intensive workloads in virtualized cloud environments. On the x86 architecture, a nested translation needs to sequentially fetch up to 24 page table entries (PTEs). This paper presents Direct Memory Translation (DMT), a hardware-software extension for x86-based virtual memory that minimizes translation overhead while maintaining backward compatibility with x86. In DMT, the OS manages last-level PTEs in a contiguous physical memory region, termed Translation Entry Areas (TEAs). DMT establishes a direct mapping from each virtual page in a Virtual Memory Area (VMA) to the corresponding PTE in a TEA. Since processes manage memory with a handful of major VMAs, the mapping can be maintained per VMA and effectively stored in a few dedicated registers. DMT further optimizes virtualized memory translation via guest-host cooperation by directly allocating guest TEAs in physical memory, bypassing intermediate virtualization layers. DMT is inherently scalable---it takes one, two, and three memory references in native, virtualized, and nested virtualized setups. Its scalability enables hardware-assisted translation for nested virtualization. Our evaluation shows that DMT significantly speeds up page walks by an average of 1.58x (1.65x with THP) in a virtualized setup, resulting in 1.20x (1.14x with THP) speedup of application execution on average.

References

[1]
Adams, K., and Agesen, O. A Comparison of Software and Hardware Techniques for x86 Virtualization. In Proceedings of the 26th ACM International Conference on Architectural Support for Programming Languages and Operating Systems (ASPLOS'06) (Oct. 2006).
[2]
Agesen, O., Garthwaite, A., Sheldon, J., and Subrahmanyam, P. The Evolution of an x86 Virtual Machine Monitor. ACM SIGOPS Operating Systems Review (Dec. 2010), 3--18.
[3]
Ahn, J., Jin, S., and Huh, J. Revisiting Hardware-Assisted Page Walks for Virtualized Systems. In Proceedings of the 39th Annual International Symposium on Computer Architecture (ISCA'12) (June 2012).
[4]
Ainsworth, S., and Jones, T. M. Compendia: Reducing Virtual-Memory Costs via Selective Densification. In Proceedings of the 2021 ACM SIGPLAN International Symposium on Memory Management (ISMM'21) (June 2021).
[5]
Alam, H., Zhang, T., Erez, M., and Etsion, Y. Do-It-Yourself Virtual Memory Translation. In Proceedings of the 44th Annual International Symposium on Computer Architecture (ISCA'17) (June 2017).
[6]
Alverti, C., Psomadakis, S., Karakostas, V., Gandhi, J., Nikas, K., Goumas, G., and Koziris, N. Enhancing and Exploiting Contiguity for Fast Memory Virtualization. In Proceedings of the 47th Annual International Symposium on Computer Architecture (ISCA'20) (June 2020).
[7]
Azure, M. Nested Virtualization in Azure. https://azure.microsoft.com/en-us/blog/nested-virtualization-in-azure/, July 2017.
[8]
Balasubramonian, R., Kahng, A. B., Muralimanohar, N., Shafiee, A., and Srinivas, V. CACTI 7: New Tools for Interconnect Exploration in Innovative Off-Chip Memories. ACM Transactions on Architecture and Code Optimization (TACO) 14, 2 (June 2017), 1--25.
[9]
Barr, T. W., Cox, A. L., and Rixner, S. Translation Caching: Skip, Don't Walk (the Page Table). In Proceedings of the 37th Annual International Symposium on Computer Architecture (ISCA'10) (June 2010).
[10]
Basu, A., Gandhi, J., Chang, J., Hill, M. D., and Swift, M. M. Efficient Virtual Memory for Big Memory Servers. In Proceedings of the 40th Annual International Symposium on Computer Architecture (ISCA'13) (June 2013).
[11]
Ben-Yehuda, M., Day, M. D., Dubitzky, Z., Factor, M., Har'El, N., Gordon, A., Liguori, A., Wasserman, O., and Yassour, B.-A. The Turtles Project: Design and Implementation of Nested Virtualization. In Proceedings of the 9th USENIX Symposium on Operating Systems Design and Implementation (OSDI'10) (Oct. 2010).
[12]
Bender, M. A., Bhattacharjee, A., Conway, A., Farach-Colton, M., Johnson, R., Kannan, S., Kuszmaul, W., Mukherjee, N., Porter, D., Tagliavini, G., Vorobyeva, J., and West, E. Paging and the Address-Translation Problem. In Proceedings of the 33rd ACM Symposium on Parallelism in Algorithms and Architectures (SPAA'21) (July 2021).
[13]
Bienia, C., Kumar, S., Singh, J. P., and Li, K. The parsec benchmark suite: Characterization and architectural implications. In Proceedings of the 17th international conference on Parallel architectures and compilation techniques (PACT '08) (2008).
[14]
Cloud, G. About nested virtualization. https://cloud.google.com/compute/docs/instances/nested-virtualization/overview, Dec. 2023.
[15]
Cox, G., and Bhattacharjee, A. Efficient Address Translation for Architectures with Multiple Page Sizes. In Proceedings of the Twenty-Second International Conference on Architectural Support for Programming Languages and Operating Systems (ASPLOS'17) (Apr. 2017).
[16]
Devices, A. M. AMD64 Architecture Programmer's Manual: Volumes 1-5. https://www.amd.com/content/dam/amd/en/documents/processor-tech-docs/programmer-references/40332.pdf, June 2023.
[17]
Du, Y., Zhou, M., Childers, B. R., Mossé, D., and Melhem, R. Supporting Superpages in Non-Contiguous Physical Memory. In Proceedings of the 21st IEEE International Symposium on High Performance Computer Architecture (HPCA-21) (Feb. 2015).
[18]
DynamoRIO. Dynamic Instrumentation Tool Platform. https://github.com/DynamoRIO/dynamorio, Dec. 2023.
[19]
Gandhi, J., Basu, A., Hill, M. D., and Swift, M. M. Efficient Memory Virtualization: Reducing Dimensionality of Nested Page Walks. In Proceedings of the 47th Annual IEEE/ACM International Symposium on Microarchitecture (MICRO-47) (Dec. 2014).
[20]
Gandhi, J., Hill, M. D., and Swift, M. M. Agile Paging: Exceeding the Best of Nested and Shadow Paging. In Proceedings of the 43rd Annual International Symposium on Computer Architecture (ISCA'16) (June 2016).
[21]
Google. TCMalloc. https://github.com/google/tcmalloc, 2023.
[22]
Gorman, M. Understanding the Linux Virtual Memory Manager. Prentice Hall, Apr. 2004.
[23]
Gosakan, K., Han, J., Kuszmaul, W., Mubarek, I. N., Mukherjee, N., Sriram, K., Tagliavini, G., West, E., Bender, M. A., Bhattacharjee, A., Conway, A., Farach-Colton, M., Gandhi, J., Johnson, R., Kannan, S., and Porter, D. E. Mosaic Pages: Big TLB Reach with Small Pages. In Proceedings of the 28th ACM International Conference on Architectural Support for Programming Languages and Operating Systems (ASPLOS'23) (Mar. 2023).
[24]
Graph 500 Steering Committee. Graph 500 Benchmark Specification. https://graph500.org/?page_id=12, 2017.
[25]
Gupta, S., Bhattacharyya, A., Oh, Y., Bhattacharjee, A., Falsafi, B., and Payer, M. Rebooting Virtual Memory with Midgard. In Proceedings of the 48th Annual International Symposium on Computer Architecture (ISCA'21) (June 2021).
[26]
Haria, S., Hill, M. D., and Swift, M. M. Devirtualizing Memory in Heterogeneous Systems. In Proceedings of the 23rd International Conference on Architectural Support for Programming Languages and Operating Systems (ASPLOS'18) (Mar. 2018).
[27]
HPC Challenge Benchmark. RandomAccess: GUPS (Giga Updates Per Second). https://hpcchallenge.org/projectsfiles/hpcc/RandomAccess.html, 2022.
[28]
Huang, H., Lai, J., Rao, J., Lu, H., Hou, W., Su, H., Xu, Q., Zhong, J., Zeng, J., Wang, X., He, Z., Han, W., Liu, J., Ma, T., and Wu, S. PVM: Efficient Shadow Paging for Deploying Secure Containers in Cloud-native Environment. In Proceedings of the 29th Symposium on Operating Systems Principles (SOSP'23) (Oct. 2023).
[29]
Hunter, A., Kennelly, C., Turner, P., Gove, D., Moseley, T., and Ranganathan, P. Beyond malloc efficiency to fleet efficiency: a hugepage-aware memory allocator. In Proceedings of the 15th USENIX Symposium on Operating Systems Design and Implementation (OSDI'21) (July 2021).
[30]
Intel. 5-Level Paging and 5-Level EPT White Paper. https://software.intel.com/content/www/us/en/develop/download/5-level-paging-and-5-level-ept-white-paper.html, May 2017.
[31]
Intel. Intel Xeon Gold 6138 Processor 27.5M Cache 2.00 GHz Product Specifications. https://ark.intel.com/content/www/us/en/ark/products/120476/intel-xeon-gold-6138-processor-27-5m-cache-2-00-ghz.html, 2017.
[32]
Intel. Intel® 64 and IA-32 Architectures Developer's Manual: Vol. 3C. https://www.intel.com/content/www/us/en/developer/articles/technical/intel-sdm.html, Dec. 2023.
[33]
Jaleel, A., Ebrahimi, E., and Duncan, S. DUCATI: High-Performance Address Translation by Extending TLB Reach of GPU-Accelerated Systems. ACM Transactions on Architecture and Code Optimization (TACO) 16, 1 (Mar. 2019), 1--24.
[34]
Jia, W., Zhang, J., Shan, J., and Ding, X. Making Dynamic Page Coalescing Effective on Virtualized Clouds. In Proceedings of the 18th European Conference on Computer Systems (EuroSys'23) (May 2023).
[35]
Jia, W., Zhang, J., Shan, J., Du, Y., Ding, X., and Xu, T. HugeGPT: Storing Guest Page Tables on Host Huge Pages to Accelerate Address Translation. In Proceedings of the 32nd International Conference on Parallel Architectures and Compilation Techniques (PACT'23) (Oct. 2023).
[36]
Kadekodi, R., Kadekodi, S., Ponnapalli, S., Shirwadkar, H., Ganger, G. R., Kolli, A., and Chidambaram, V. WineFS: a hugepage-aware file system for persistent memory that ages gracefully. In Proceedings of the 28th ACM Symposium on Operating Systems Principles (SOSP'21) (Oct. 2021).
[37]
Kanellopoulos, K., Bera, R., Stojiljkovic, K., Bostanci, F. N., Firtina, C., Ausavarungnirun, R., Kumar, R., Hajinazar, N., Sadrosadati, M., Vijaykumar, N., and Mutlu, O. Utopia: Efficient Address Translation using Hybrid Virtual-to-Physical Address Mapping. In Proceedings of the 56th Annual IEEE/ACM International Symposium on Microarchitecture (MICRO-56) (Dec. 2023).
[38]
Karakostas, V., Gandhi, J., Ayar, F., Cristal, A., Hill, M. D., McKinley, K. S., Nemirovsky, M., Swift, M. M., and Ünsal, O. Redundant Memory Mappings for Fast Access to Large Memories. In Proceedings of the 42nd Annual International Symposium on Computer Architecture (ISCA'15) (June 2015).
[39]
Kwon, O., Lee, Y., and Hong, S. Pinning Page Structure Entries to Last-Level Cache for Fast Address Translation. IEEE Access 10 (Oct. 2022), 114552--114565.
[40]
Kwon, Y., Yu, H., Peter, S., Rossbach, C. J., and Witchel, E. Coordinated and Efficient Huge Page Management with Ingens. In Proceedings of the 12th USENIX Symposium on Operating Systems Design and Implementation (OSDI'16) (Nov. 2016).
[41]
Levy, H. M., and Lipman, P. H. Virtual Memory Management in the VAX/VMS Operating System. IEEE Computer 15, 3 (Mar. 1982), 35--41.
[42]
Lim, J. T., and Nieh, J. Optimizing Nested Virtualization Performance Using Direct Virtual Hardware. In Proceedings of the Twenty-Fifth International Conference on Architectural Support for Programming Languages and Operating Systems (ASPLOS'20) (Mar. 2020).
[43]
Maas, M., Kennelly, C., Nguyen, K., Gove, D., McKinley, K. S., and Turner, P. Adaptive Huge-Page Subrelease for Non-moving Memory Allocators in Warehouse-Scale Computers. In Proceedings of the 2021 ACM SIGPLAN International Symposium on Memory Management (ISMM'21) (June 2021).
[44]
Marathe, Y., Gulur, N., Ryoo, J. H., Song, S., and John, L. K. CSALT: Context Switch Aware Large TLB. In Proceedings of the 50th Annual IEEE/ACM International Symposium on Microarchitecture (MICRO-50) (Oct. 2017).
[45]
Margaritov, A., Ustiugov, D., Bugnion, E., and Grot, B. Prefetched Address Translation. In Proceedings of the 52nd Annual IEEE/ACM International Symposium on Microarchitecture (MICRO-52) (Oct. 2019).
[46]
Margaritov, A., Ustiugov, D., Shahab, A., and Grot, B. PTEMagnet: Fine-grained Physical Memory Reservation for Faster Page Walks in Public Clouds. In Proceedings of the 26th ACM International Conference on Architectural Support for Programming Languages and Operating Systems (ASPLOS'21) (Apr. 2021).
[47]
Mazumdar, C., Mitra, P., and Basu, A. Dead Page and Dead Block Predictors: Cleaning TLBs and Caches Together. In Proceedings of the 27th IEEE International Symposium on High Performance Computer Architecture (HPCA-27) (Feb. 2021).
[48]
Memcached. memcached - a distributed memory object caching system. https://memcached.org, 2023.
[49]
Microsoft. Frequently Asked Questions about Windows Subsystem for Linux. https://learn.microsoft.com/en-us/windows/wsl/faq, 2023.
[50]
Microsoft. Virtualization-based Security (VBS). https://learn.microsoft.com/en-us/windows-hardware/design/device-experiences/oem-vbs, Mar. 2023.
[51]
Mitosis Project. Mitosis Workload Btree. https://github.com/mitosis-project/mitosis-workload-btree, 2020.
[52]
Molnar, I. Performance Counters for Linux. https://lwn.net/Articles/337493/, June 2009.
[53]
Navarro, J., Iyer, S., and Cox, A. Practical, Transparent Operating System Support for Superpages. In Proceedings of the 5th Symposium on Operating Systems Design and Implementation (OSDI'02) (Dec. 2002).
[54]
Pagh, R., and Rodler, F. F. Cuckoo Hashing. Journal of Algorithms 51, 2 (May 2004), 122--144.
[55]
Panwar, A., Bansal, S., and Gopinath, K. HawkEye: Efficient Fine-grainedOS Support for Huge Pages. In Proceedings of the Twenty-Fourth International Conference on Architectural Support for Programming Languages and Operating Systems (ASPLOS'19) (Apr. 2019).
[56]
Panwar, A., Prasad, A., and Gopinath, K. Making Huge Pages Actually Useful. In Proceedings of the Twenty-Third International Conference on Architectural Support for Programming Languages and Operating Systems (ASPLOS'18) (Mar. 2018).
[57]
Park, C. H., Cha, S., Kim, B., Kwon, Y., Black-Schaffer, D., and Huh, J. Perforated Page: Supporting Fragmented Memory Allocation for Large Pages. In Proceedings of the 47th Annual International Symposium on Computer Architecture (ISCA'20) (May 2020).
[58]
Park, C. H., Heo, T., Jeong, J., and Huh, J. Hybrid TLB Coalescing: Improving TLB Translation Coverage under Diverse Fragmented Memory Allocations. In Proceedings of the 44th Annual International Symposium on Computer Architecture (ISCA'17) (June 2017).
[59]
Park, C. H., Vougioukas, I., Sandberg, A., and Black-Schaffer, D. Every Walk's a Hit: Making Page Walks Single-Access Cache Hits. In Proceedings of the 27th ACM International Conference on Architectural Support for Programming Languages and Operating Systems (ASPLOS'22) (Feb. 2022).
[60]
Pham, B., Bhattacharjee, A., Eckert, Y., and Loh, G. H. Increasing TLB Reach by Exploiting Clustering in Page Translations. In Proceedings of the 20th IEEE International Symposium on High Performance Computer Architecture (HPCA-20) (Feb. 2014).
[61]
Pham, B., Vaidyanathan, V., Jaleel, A., and Bhattacharjee, A. CoLT: Coalesced Large-Reach TLBs. In Proceedings of the 45th IEEE/ACM International Symposium on Microarchitecture (MICRO-45) (Dec. 2012).
[62]
Redis. http://redis.io/, 2023.
[63]
Ryoo, J. H., Gulur, N., Song, S., and John, L. K. Rethinking TLB Designs in Virtualized Environments: A Very Large Part-of-Memory TLB. In Proceedings of the 44th Annual International Symposium on Computer Architecture (ISCA'17) (June 2017).
[64]
Skarlatos, D., Kokolis, A., Xu, T., and Torrellas, J. Elastic Cuckoo Page Tables: Rethinking Virtual Memory Translation for Parallelism. In Proceedings of the Twenty-Fifth International Conference on Architectural Support for Programming Languages and Operating Systems (ASPLOS'20) (Mar. 2020).
[65]
Stojkovic, J., Mantri, N., Skarlatos, D., Xu, T., and Torrellas, J. Memory-Efficient Hashed Page Tables. In Proceedings of the 29th IEEE International Symposium on High-Performance Computer Architecture (HPCA-29) (Feb. 2023).
[66]
Stojkovic, J., Skarlatos, D., Kokolis, A., Xu, T., and Torrellas, J. Parallel Virtualized Memory Translation with Nested Elastic Cuckoo Page Tables. In Proceedings of the 27th ACM International Conference on Architectural Support for Programming Languages and Operating Systems (ASPLOS'22) (Feb. 2022).
[67]
Suchy, B., Campanoni, S., Hardavellas, N., and Dinda, P. CARAT: A Case for Virtual Memory through Compiler- and Runtime-Based Address Translation. In Proceedings of the 41st ACM SIGPLAN Conference on Programming Language Design and Implementation (PLDI'20) (June 2020).
[68]
Suchy, B., Ghosh, S., Kersnar, D., Chai, S., Huang, Z., Nelson, A., Cuevas, M., Bernat, A., Chaudhary, G., Hardavellas, N., Campanoni, S., and Dinda, P. CARAT CAKE: Replacing Paging via Compiler/Kernel Cooperation. In Proceedings of the 27th ACM International Conference on Architectural Support for Programming Languages and Operating Systems (ASPLOS'22) (Feb. 2022).
[69]
Talluri, M., Hill, M. D., and Khalidi, Y. A. A New Page Table for 64-bit Address Space. In Proceedings of the 15th ACM Symposium on Operating Systems Principles (SOSP'95) (Dec. 1995).
[70]
Teabe, B., Yuhala, P., Tchana, A., Hermenier, F., Hagimont, D., and Muller, G. (No)Compromis: Paging Virtualization Is Not a Fatality. In Proceedings of the 17th ACM SIGPLAN/SIGOPS International Conference on Virtual Execution Environments (VEE'21) (Apr. 2021).
[71]
Tramm, J. R., Siegel, A. R., Islam, T., and Schulz, M. XSBench - The Development and Verification of a Performance Abstraction for Monte Carlo Reactor Analysis. In Proceedings of the International Conference on Physics of Reactors (PHYSOR 2014) (Sept. 2014).
[72]
VMware. Performance Evaluation of Intel EPT Hardware Assist. https://www.vmware.com/pdf/Perf_ESX_Intel-EPT-eval.pdf, 2009.
[73]
VMware. Support for running ESXi as a nested virtualization solution. https://kb.vmware.com/s/article/2009916, Nov. 2023.
[74]
Waldspurger, C. A. Memory Resource Management in VMware ESX Server. In Proceedings of the 5th Symposium on Operating Systems Design and Implementation (OSDI'02) (Dec. 2002).
[75]
Wang, X., Zang, J., Wang, Z., Luo, Y., and Li, X. Selective Hardware/Software Memory Virtualization. In Proceedings of the 7th ACM SIGPLAN/SIGOPS International Conference on Virtual Execution Environments (VEE'11) (Mar. 2011).
[76]
WikiChip. Intel Xeon Gold 6138. https://en.wikichip.org/wiki/intel/xeon_gold/6138, 2019.
[77]
Wrenger, L., Rommel, F., Halbuer, A., Dietrich, C., and Lohmann, D. LLFree: Scalable and Optionally-Persistent Page-Frame Allocation. In Proceedings of the 2023 USENIX Annual Technical Conference (USENIX ATC'23) (July 2023).
[78]
x86 cpus' Guide. Intel Xeon Gold 6138. https://www.x86-guide.net/en/cpu/Intel-Xeon-Gold-6138-cpu-no6271.html, 2023.
[79]
Yan, Z., Lustig, D., Nellans, D., and Bhattacharjee, A. Nimble Page Management for Tiered Memory Systems. In Proceedings of the 24th International Conference on Architectural Support for Programming Languages and Operating Systems (ASPLOS'19) (Apr. 2019).
[80]
Yaniv, I., and Tsafrir, D. Hash, Don't Cache (the Page Table). In Proceedings of the 2016 ACM SIGMETRICS International Conference on Measurement and Modeling of Computer Systems (SIGMETRICS'16) (June 2016).
[81]
Ye, H. Introduction to 5-Level Paging in 3rd Gen Intel Xeon Scalable Processors with Linux. https://lenovopress.lenovo.com/lp1468.pdf, May 2021.
[82]
Yip, M. Running Hyper-V on Amazon EC2 Bare Metal Instances. https://aws.amazon.com/blogs/compute/running-hyper-v-on-amazon-ec2-bare-metal-instances/, Aug. 2018.
[83]
Zhang, F., Chen, J., Chen, H., and Zang, B. CloudVisor: Retrofitting Protection of Virtual Machines in Multi-tenant Cloud with Nested Virtualization. In Proceedings of the Twenty-Third ACM Symposium on Operating Systems Principles (SOSP'11) (Oct. 2011).
[84]
Zhang, L., Speight, E., Rajamony, R., and Lin, J. Enigma: Architectural and Operating System Support for Reducing the Impact of Address Translation. In Proceedings of the 24th ACM International Conference on Supercomputing (ICS'10) (June 2010).
[85]
Zhao, K., Xue, K., Wang, Z., Schatzberg, D., Yang, L., Manousis, A., Weiner, J., Van Riel, R., Sharma, B., Tang, C., and Skarlatos, D. Contiguitas: The Pursuit of Physical Memory Contiguity in Data-centers. In Proceedings of the 50th Annual International Symposium on Computer Architecture (ISCA'23) (June 2023).

Recommendations

Comments

Please enable JavaScript to view thecomments powered by Disqus.

Information & Contributors

Information

Published In

cover image ACM Conferences
ASPLOS '24: Proceedings of the 29th ACM International Conference on Architectural Support for Programming Languages and Operating Systems, Volume 2
April 2024
1299 pages
ISBN:9798400703850
DOI:10.1145/3620665
Permission to make digital or hard copies of all or part of this work for personal or classroom use is granted without fee provided that copies are not made or distributed for profit or commercial advantage and that copies bear this notice and the full citation on the first page. Copyrights for components of this work owned by others than the author(s) must be honored. Abstracting with credit is permitted. To copy otherwise, or republish, to post on servers or to redistribute to lists, requires prior specific permission and/or a fee. Request permissions from [email protected].

Sponsors

In-Cooperation

Publisher

Association for Computing Machinery

New York, NY, United States

Publication History

Published: 27 April 2024

Check for updates

Badges

Author Tags

  1. cloud computing
  2. virtualization
  3. virtual memory
  4. address translation

Qualifiers

  • Research-article

Conference

ASPLOS '24

Acceptance Rates

Overall Acceptance Rate 535 of 2,713 submissions, 20%

Upcoming Conference

Contributors

Other Metrics

Bibliometrics & Citations

Bibliometrics

Article Metrics

  • 0
    Total Citations
  • 873
    Total Downloads
  • Downloads (Last 12 months)873
  • Downloads (Last 6 weeks)140
Reflects downloads up to 29 Nov 2024

Other Metrics

Citations

View Options

View options

PDF

View or Download as a PDF file.

PDF

eReader

View online with eReader.

eReader

Login options

Media

Figures

Other

Tables

Share

Share

Share this Publication link

Share on social media