Logical Memory Pools: Flexible and Local Disaggregated Memory
Pages 25 - 32
Abstract
We propose logical memory pools, a memory disaggregation architecture for the emerging Compute Express Link (CXL) technology in datacenters. The key idea is to create a memory pool by carving out parts of the local memory in each server, rather than using a physical memory pool that is separate from servers. Logical pools provide significant benefits over physical pools, namely, lower cost, support for near-memory computing without extra hardware, and flexibility on designating whether memory is part of the memory pool or not. We demonstrate that logical pools can execute workloads that are unfeasible in physical pools, and that its faster access leads to better performance. Realizing logical memory pools poses five major challenges, which we believe can be overcome. Given the benefits of logical pools, we believe the CXL community should refocus efforts on logical, rather than physical memory pools.
References
[1]
Anant Agarwal, Ricardo Bianchini, David Chaiken, Kirk L Johnson, David Kranz, John Kubiatowicz, Beng-Hong Lim, Kenneth Mackenzie, and Donald Yeung. 1995. The MIT Alewife machine: Architecture and performance. ACM SIGARCH Computer Architecture News 23, 2 (1995), 2--13.
[2]
Emmanuel Amaro, Christopher Branner-Augmon, Zhihong Luo, Amy Ousterhout, Marcos K Aguilera, Aurojit Panda, Sylvia Ratnasamy, and Scott Shenker. 2020. Can far memory improve job throughput?. In European Conference on Computer Systems. 1--16.
[3]
Cristiana Amza, Alan L. Cox, Shandya Dwarkadas, Pete Keleher, Honghui Lu, Ramakrishnan Rajamony, Weimin Yu, and Willy Zwaenepoel. 1996. TreadMarks: Shared Memory Computing on Networks of Workstations. IEEE Computer 29, 2 (Feb. 1996), 18--28.
[4]
J. K. Bennett, J. B. Carter, and W. Zwaenepoel. 1990. Munin: Distributed Shared Memory Based on Type-specific Memory Coherence. In ACM Symposium on Principles and Practice of Parallel Programming. 168--176.
[5]
Qingchao Cai, Wentian Guo, Hao Zhang, Divyakant Agrawal, Gang Chen, Beng Chin Ooi, Kian-Lee Tan, Yong Meng Teo, and Sheng Wang. 2018. Efficient distributed memory management with RDMA and caching. 11, 11 (2018), 1604--1617.
[6]
Irina Calciu, Dave Dice, Yossi Lev, Victor Luchangco, Virendra J Marathe, and Nir Shavit. 2013. NUMA-aware reader-writer locks. In ACM Symposium on Principles and Practice of Parallel Programming. 157--166.
[7]
CCIX Consortium. Accessed 2023/01/26. CCIX. (Accessed 2023/01/26). https://www.ccixconsortium.com/wp-content/uploads/2019/11/CCIX-White-Paper-Rev111219.pdf.
[8]
cxl [n. d.]. Compute Express Link (CXL). ([n. d.]). https://www.computeexpresslink.org.
[9]
Rafael Lourenco de Lima Chehab, Antonio Paolillo, Diogo Behrens, Ming Fu, Hermann Härtig, and Haibo Chen. 2021. Clof: A compositional lock framework for multi-level NUMA systems. In ACM Symposium on Operating Systems Principles. 851--865.
[10]
Dave Dice and Alex Kogan. 2019. Compact NUMA-aware locks. In Proceedings of the Fourteenth EuroSys Conference 2019. 1--15.
[11]
David Dice, Virendra J Marathe, and Nir Shavit. 2012. Lock cohorting: a general technique for designing NUMA locks. ACM SIGPLAN Notices 47, 8(2012), 247--256.
[12]
Aleksandar Dragojević, Dushyanth Narayanan, Miguel Castro, and Orion Hodson. 2014. FaRM: Fast Remote Memory. In Symposium on Networked Systems Design and Implementation. 401--414.
[13]
Aleksandar Dragojević, Dushyanth Narayanan, Ed Nightingale, Matthew Renzelmann, Alex Shamis, Anirudh Badam, and Miguel Castro. 2015. No compromises: distributed transactions with consistency, availability, and performance. In ACM Symposium on Operating Systems Principles. 54--70.
[14]
Philip Werner Frey and Gustavo Alonso. 2009. Minimizing the hidden cost of RDMA. In 2009 29th IEEE International Conference on Distributed Computing Systems. IEEE, 553--560.
[15]
Vasilis Gavrielatos, Antonios Katsarakis, Arpit Joshi, Nicolai Oswald, Boris Grot, and Vijay Nagarajan. 2018. Scale-out ccNUMA: Exploiting skew with strongly consistent caching. In European Conference on Computer Systems. 1--15.
[16]
genz [n. d.]. Gen-Z consortium. ([n. d.]). https://genzconsortium.org.
[17]
Donghyun Gouk, Sangwon Lee, Miryeong Kwon, and Myoungsoo Jung. 2022. Direct Access, High-Performance Memory Disaggregation with DirectCXL. In USENIX Annual Technical Conference.
[18]
Juncheng Gu, Youngmoon Lee, Yiwen Zhang, Mosharaf Chowdhury, and Kang G. Shin. 2017. Efficient Memory Disaggregation with INFINISWAP. In Symposium on Networked Systems Design and Implementation. 649--667.
[19]
Intel. Accessed 2023/01/26. Intel Rack Scale Architecture. (Accessed 2023/01/26). https://www.intel.com/content/dam/www/public/us/en/documents/white-papers/rack-scale-design-architecture-white-paper.pdf.
[20]
Intel. Accessed 2023/06/29. Intel Xeon Processor Scalable Family Technical Overview. (Accessed 2023/06/29). https://www.intel.com/content/www/us/en/developer/articles/technical/xeon-processor-scalable-family-technical-overview.html.
[21]
Nusrat Sharmin Islam, Dipti Shankar, Xiaoyi Lu, Md Wasi-Ur-Rahman, and Dhabaleswar K Panda. 2015. Accelerating I/O performance of big data analytics on HPC clusters through RDMA-based key-value store. In International Conference on Parallel Processing. 280--289.
[22]
Anuj Kalia, Michael Kaminsky, and David G. Andersen. 2014. Using RDMA Efficiently for Key-Value Services. In ACM Special Interest Group on Data Communications.
[23]
Anuj Kalia, Michael Kaminsky, and David G Andersen. 2016. FaSST: Fast, Scalable and Simple Distributed Transactions with Two-Sided (RDMA) Datagram RPCs. In Symposium on Operating Systems Design and Implementation. 185--201.
[24]
Sanidhya Kashyap, Irina Calciu, Xiaohe Cheng, Changwoo Min, and Taesoo Kim. 2019. Scalable and practical locking with shuffling. In ACM Symposium on Operating Systems Principles. 586--599.
[25]
James Laudon and Daniel Lenoski. 1997. The SGI Origin: a ccNUMA highly scalable server. ACM SIGARCH Computer Architecture News 25, 2 (1997), 241--251.
[26]
Seung-seob Lee, Yanpeng Yu, Yupeng Tang, Anurag Khandelwal, Lin Zhong, and Abhishek Bhattacharjee. 2021. Mind: In-network memory management for disaggregated data centers. In ACM Symposium on Operating Systems Principles. 488--504.
[27]
Huaicheng Li, Daniel S. Berger, Stanko Novakovic, Lisa Hsu, Dan Ernst, Pantea Zardoshti, Monish Shah, Samir Rajadnya, Scott Lee, Ishwar Agarwal, Mark D. Hill, Marcus Fontoura, and Ricardo Bianchini. 2023. Pond: CXL-Based Memory Pooling Systems for Cloud Platforms. In ACM International Conference on Architectural Support for Programming Languages and Operating Systems.
[28]
Kai Li and Paul Hudak. 1989. Memory Coherence in Shared Virtual Memory Systems. ACM Transactions on Computer Systems 7, 4 (Nov. 1989), 321--359.
[29]
Kevin Lim, Jichuan Chang, Trevor Mudge, Parthasarathy Ranganathan, Steven K. Reinhardt, and Thomas F. Wenisch. 2009. Disaggregated Memory for Expansion and Sharing in Blade Servers. In International Symposium on Computer Architecture.
[30]
Ming Liu. 2023. Fabric-Centric Computing. In Workshop on Hot Topics in Operating Systems. 118--126.
[31]
Youyou Lu, Jiwu Shu, Youmin Chen, and Tao Li. 2017. Octopus: an RDMA-enabled Distributed Persistent Memory File System. In USENIX Annual Technical Conference.
[32]
Shaonan Ma, Teng Ma, Kang Chen, and Yongwei Wu. 2022. A Survey of Storage Systems in the RDMA Era. IEEE Transactions on Parallel and Distributed Systems (2022).
[33]
Christopher Mitchell, Yifeng Geng, and Jinyang Li. 2013. Using One-Sided RDMA Reads to Build a Fast, CPU-Efficient Key-Value Store. In USENIX Annual Technical Conference.
[34]
Jacob Nelson, Brandon Holt, Brandon Myers, Preston Briggs, Luis Ceze, Simon Kahan, and Mark Oskin. 2015. Latency-tolerant Software Distributed Shared Memory. In USENIX Annual Technical Conference. 291--305.
[35]
Stanko Novakovic, Alexandros Daglis, Edouard Bugnion, Babak Falsafi, and Boris Grot. 2014. Scale-out NUMA. In ACM International Conference on Architectural Support for Programming Languages and Operating Systems. 3--18.
[36]
Stanko Novakovic, Alexandros Daglis, Edouard Bugnion, Babak Falsafi, and Boris Grot. 2016. The case for RackOut: Scalable data serving using rack-scale systems. In ACM Symposium on Cloud Computing. 182--195.
[37]
Stanko Novakovic, Yizhou Shan, Aasheesh Kolli, Michael Cui, Yiying Zhang, Haggai Eran, Boris Pismenny, Liran Liss, Michael Wei, Dan Tsafrir, and Marcos K. Aguilera. 2019. Storm: a fast transactional dataplane for remote data structures. In ACM International Conference on Systems and Storage. 97--108.
[38]
Yifan Qiao, Chenxi Wang, Zhenyuan Ruan, Adam Belay, Qingda Lu, Yiying Zhang, Miryung Kim, and Guoqing Harry Xu. 2023. Hermit: Low-Latency, High-Throughput, and Transparent Remote Memory via Feedback-Directed Asynchrony. In Symposium on Networked Systems Design and Implementation. 181--198.
[39]
Zhenyuan Ruan, Malte Schwarzkopf, Marcos K Aguilera, and Adam Belay. 2020. AIFM: High-performance, application-integrated far memory. In Symposium on Operating Systems Design and Implementation. 315--332.
[40]
André Ryser, Alberto Lerner, Alex Forencich, and Philippe Cudré-Mauroux. 2022. D-RDMA: Bringing Zero-Copy RDMA to Database Systems. In Conference on Innovative Data Systems Research.
[41]
Daniel J. Scales, Kourosh Gharachorloo, and Chandramohan A. Thekkath. 1996. Shasta: A Low Overhead, Software-only Approach for Supporting Fine-grain Shared Memory. In ACM International Conference on Architectural Support for Programming Languages and Operating Systems. 174--185.
[42]
Ioannis Schoinas, Babak Falsafi, Alvin R. Lebeck, Steven K. Reinhardt, James R. Larus, and David A. Wood. 1994. Fine-grain Access Control for Distributed Shared Memory. In ACM International Conference on Architectural Support for Programming Languages and Operating Systems. 297--306.
[43]
Yizhou Shan, Yutong Huang, Yilun Chen, and Yiying Zhang. 2018. LegoOS: A Disseminated, Distributed OS for Hardware Resource Disaggregation. In Symposium on Operating Systems Design and Implementation.
[44]
Yan Sun, Yifan Yuan, Zeduo Yu, Reese Kuper, Ipoom Jeong, Ren Wang, and Nam Sung Kim. 2023. Demystifying CXL Memory with Genuine CXL-Ready Systems and Devices. arXiv preprint arXiv:2303.15375 (2023).
[45]
Chenxi Wang, Yifan Qiao, Haoran Ma, Shi Liu, Wenguang Chen, Ravi Netravali, Miryung Kim, and Guoqing Harry Xu. 2023. Canvas: Isolated and Adaptive Swapping for Multi-Applications on Remote Memory. In Symposium on Networked Systems Design and Implementation. 161--179.
[46]
Hao Wang, Sreeram Potluri, Devendar Bureddy, Carlos Rosales, and Dhabaleswar K Panda. 2013. GPU-aware MPI on RDMA-enabled clusters: Design, implementation and evaluation. IEEE Transactions on Parallel and Distributed Systems 25, 10 (2013), 2595--2605.
[47]
Jian Yang, Joseph Izraelevitz, and Steven Swanson. 2019. Orion: A Distributed File System for Non-Volatile Main Memory and RDMA-Capable Networks. In USENIX Conference on File and Storage Technologies, Vol. 19. 221--234.
[48]
Erfan Zamanian, Xiangyao Yu, Michael Stonebraker, and Tim Kraska. 2019. Rethinking Database High Availability with RDMA Networks. Proceedings of the VLDB Endowment 12, 11 (2019), 1637--1650.
[49]
Yang Zhou, Hassan Wassel, Sihang Liu, Jiaqi Gao, James Mickens, Minlan Yu, Chris Kennelly, Paul Jack Turner, David E Culler, Hank Levy, and Amin Vahdat. 2022. Carbink: Fault-tolerant Far Memory. In Symposium on Operating Systems Design and Implementation.
Index Terms
- Logical Memory Pools: Flexible and Local Disaggregated Memory
Recommendations
CrypTag: Thwarting Physical and Logical Memory Vulnerabilities using Cryptographically Colored Memory
ASIA CCS '21: Proceedings of the 2021 ACM Asia Conference on Computer and Communications SecurityMemory vulnerabilities are a major threat to many computing systems. To effectively thwart spatial and temporal memory vulnerabilities, full logical memory safety is required. However, current mitigation techniques for memory safety are either too ...
Write-once-memory-code phase change memory
DATE '14: Proceedings of the conference on Design, Automation & Test in EuropeThis paper describes a write-once-memory-code phase change memory (WOM-code PCM) architecture for next-generation non-volatile memory applications. Specifically, we address the long latency of the write operation in PCM --- attributed to PCM SET --- by ...
Enabling Hybrid PCM Memory System with Inherent Memory Management
RACS '16: Proceedings of the International Conference on Research in Adaptive and Convergent SystemsReplacing the traditional volatile main memory, e.g., DRAM, with a non-volatile phase change memory (PCM) has become a possible solution to reduce the energy consumption of computing systems. To further reduce the bit cost of PCM, the development trend ...
Comments
Please enable JavaScript to view thecomments powered by Disqus.Information & Contributors
Information
Published In
November 2023
306 pages
ISBN:9798400704154
DOI:10.1145/3626111
Copyright © 2023 ACM.
Permission to make digital or hard copies of all or part of this work for personal or classroom use is granted without fee provided that copies are not made or distributed for profit or commercial advantage and that copies bear this notice and the full citation on the first page. Copyrights for components of this work owned by others than the author(s) must be honored. Abstracting with credit is permitted. To copy otherwise, or republish, to post on servers or to redistribute to lists, requires prior specific permission and/or a fee. Request permissions from [email protected].
Sponsors
Publisher
Association for Computing Machinery
New York, NY, United States
Publication History
Published: 28 November 2023
Check for updates
Qualifiers
- Research-article
- Research
- Refereed limited
Conference
HotNets '23
Sponsor:
HotNets '23: The 22nd ACM Workshop on Hot Topics in Networks
November 28 - 29, 2023
MA, Cambridge, USA
Acceptance Rates
Overall Acceptance Rate 110 of 460 submissions, 24%
Contributors
Other Metrics
Bibliometrics & Citations
Bibliometrics
Article Metrics
- 0Total Citations
- 337Total Downloads
- Downloads (Last 12 months)320
- Downloads (Last 6 weeks)25
Reflects downloads up to 14 Dec 2024
Other Metrics
Citations
View Options
Login options
Check if you have access through your login credentials or your institution to get full access on this article.
Sign in