RDMP-KV: Designing remote direct memory persistence based key-value stores with PMEM
Article No.: 52, Pages 1 - 4
Abstract
Byte-addressable persistent memory (PMEM) can be directly manipulated by Remote Direct Memory Access (RDMA) capable networks. However, existing studies to combine RDMA and PMEM can not deliver the desired performance due to their PMEM-oblivious communication protocols. In this paper, we propose novel PMEM-aware RDMA-based communication protocols for persistent key-value stores, referred to as Remote Direct Memory Persistence based Key-Value stores (RDMP-KV). RDMP-KV employs a hybrid 'server-reply/server-bypass' approach to 'durably' store individual key-value objects on PMEM-equipped servers. RDMP-KV's runtime can easily adapt to existing (server-assisted durability) and emerging (appliance durability) RDMA-capable interconnects, while ensuring server scalability through a lightweight consistency scheme. Performance evaluations show that RDMP-KV can improve the server-side performance with different persistent key-value storage architectures by up to 22x, as compared with PMEM-oblivious RDMA-'Server-Reply' protocols. Our evaluations also show that RDMP-KV outperforms a distributed PMEM-based filesystem by up to 65% and a recent RDMA-to-PMEM framework by up to 71%.
References
[1]
An introduction to pmemobj (part 1) - accessing the persistent memory. http://pmem.io/2015/06/13/accessing-pmem.html, 2015.
[2]
Redis. https://redis.io/, 2017.
[3]
Redis, enhanced to use PMDK's libpmemobj. https://github.com/pmem/redis, 2017.
[4]
pmemkv. https://github.com/pmem/pmemkv, 2018.
[5]
Apache Crail. http://crail.incubator.apache.org/, 2019.
[6]
Apache Ignite. https://ignite.apache.org/, 2019.
[7]
Persistent Memory Replication Over Traditional RDMA. https://software.intel.com/content/www/us/en/develop/articles/persistent-memory-replication-over-traditional-rdma-part-1-understanding-remote-persistent.html, 2019.
[8]
B. Atikoglu, Y. Xu, E. Frachtenberg, S. Jiang, and M. Paleczny. Workload Analysis of a Large-scale Key-value Store. SIGMETRICS Perform. Eval. Rev., 40(1):53--64, June 2012.
[9]
K. A. Bailey, P. Hornyack, L. Ceze, S. D. Gribble, and H. M. Levy. Exploring Storage Class Memory with Key Value Stores. In Proceedings of the 1st Workshop on Interactions of NVM/FLASH with Operating Systems and Workloads, page 4. ACM, 2013.
[10]
B. F. Cooper, A. Silberstein, E. Tam, R. Ramakrishnan, and R. Sears. Benchmarking Cloud Serving Systems with YCSB. In The Proceedings of the ACM Symposium on Cloud Computing (SoCC '10), Indianapolis, Indiana, June 2010.
[11]
C. Douglas. RDMA with PMEM. http://www.snia.org/sites/default/files/SDC15presentations/persistant_mem/ChetDouglas_RDMA_with_PM.pdf, 2015.
[12]
A. Dragojević, D. Narayanan, M. Castro, and O. Hodson. FaRM: Fast Remote Memory. In 11th USENIX Symposium on Networked Systems Design and Implementation (NSDI 14), pages 401--414, Seattle, WA, 2014. USENIX Association.
[13]
S. R. Dulloor, S. Kumar, A. Keshavamurthy, P. Lantz, D. Reddy, R. Sankaran, and J. Jackson. System Software for Persistent Memory. In Proceedings of the Ninth European Conference on Computer Systems, page 15. ACM, 2014.
[14]
H. Huang, K. Huang, L. You, and L. Huang. Forca: Fast and Atomic Remote Direct Access to Persistent Memory. 2018 IEEE 36th International Conference on Computer Design (ICCD), pages 246--249, 2018.
[15]
J. Huang, K. Schwan, and M. K. Qureshi. NVRAM-aware Logging in Transaction Systems. Proceedings of the VLDB Endowment, 8(4), 2014.
[16]
Y. Huang, M. Pavlovic, V. Marathe, M. Seltzer, T. Harris, and S. Byan. Closing the Performance Gap Between Volatile and Persistent Key-Value Stores Using Cross-Referencing Logs. In 2018 USENIX Annual Technical Conference (USENIX ATC 18), pages 967--979, 2018.
[17]
Intel. PMDK: Persistent Memory Development Kit. https://github.com/pmem/pmdk/, 2019.
[18]
N. S. Islam, M. Wasi-ur Rahman, X. Lu, and D. K. Panda. High Performance Design for HDFS with Byte-Addressability of NVM and RDMA. In Proceedings of the 2016 International Conference on Supercomputing, ICS '16, pages 8:1--8:14, New York, NY, USA, 2016. ACM.
[19]
J. Jose, H. Subramoni, M. Luo, M. Zhang, J. Huang, M. Wasi-ur Rahman, N. S. Islam, X. Ouyang, H. Wang, S. Sur, and D. K. Panda. Memcached Design on High Performance RDMA Capable Interconnects. In Proceedings of the 2011 International Conference on Parallel Processing, ICPP, Washington, DC, USA, 2011.
[20]
A. Kalia, M. Kaminsky, and D. G. Andersen. Using RDMA Efficiently for Key-Value Services. In Proceeding of SIGCOMM '14, Aug 2014.
[21]
E. Kültürsay, M. Kandemir, A. Sivasubramaniam, and O. Mutlu. Evaluating STT-RAM as an Energy-Efficient Main Memory Alternative. In 2013 IEEE International Symposium on Performance Analysis of Systems and Software (ISPASS), pages 256--267. IEEE, 2013.
[22]
Y. Lu, J. Shu, Y. Chen, and T. Li. Octopus: an RDMA-enabled Distributed Persistent Memory File System. In 2017 USENIX Annual Technical Conference (USENIX ATC 17), pages 773--785, Santa Clara, CA, 2017. USENIX Association.
[23]
Micron. 3D XPoint technology. 2019.
[24]
C. Mitchell, Y. Geng, and J. Li. Using One-Sided RDMA Reads to Build a Fast, CPU-Efficient Key-Value Store. In Proceeding of USENIX Annual Technical Conference (USENIX ATC '13), Jun 2013.
[25]
S. Pelley, T. F. Wenisch, B. T. Gold, and B. Bridge. Storage management in the NVRAM era. Proceedings of the VLDB Endowment, 7(2):121--132, 2013.
[26]
redis. How fast is Redis? https://redis.io/topics/benchmarks.
[27]
Y. Shan, S.-Y. Tsai, and Y. Zhang. Distributed Shared Persistent Memory. In Proceedings of the 2017 Symposium on Cloud Computing, pages 323--337. ACM, 2017.
[28]
D. Shankar, X. Lu, N. Islam, M. Wasi-Ur-Rahman, and D. K. Panda. High-Performance Hybrid Key-Value Store on Modern Clusters with RDMA Interconnects and SSDs: Non-blocking Extensions, Designs, and Benefits. In 2016 IEEE International Parallel and Distributed Processing Symposium (IPDPS), pages 393--402, May 2016.
[29]
M. Su, M. Zhang, K. Chen, Z. Guo, and Y. Wu. RFP: When RPC is Faster Than Server-Bypass with RDMA. In Proceedings of the Twelfth European Conference on Computer Systems, EuroSys '17, pages 1--15, 2017.
[30]
T. Talpey. Remote Access to Ultra-Low-Latency Storage - SNIA. https://www.snia.org/sites/default/files/SDC15presentations/persistantmem/Talpey-Remote_Access_Storage.pdf, 2015.
[31]
T. Talpey. RDMA Persistent Memory Extensions. In 15th Annual Open Fabrics Alliance Workshop, 2019.
[32]
H. Volos, A. J. Tack, and M. M. Swift. Mnemosyne: Lightweight Persistent Memory. In ACM SIGARCH Computer Architecture News, volume 39, pages 91--104. ACM, 2011.
[33]
T. Wang, R. Johnson, and I. Pandis. Query Fresh: Log Shipping on Steroids. Proc. VLDB Endow., 11--4:406--419, December 2017.
[34]
Y. Wang, L. Zhang, J. Tan, M. Li, Y. Gao, X. Guerin, X. Meng, and S. Meng. HydraDB: A Resilient RDMA-driven Key-value Middleware for In-memory Cluster Computing. In Proceedings of the International Conference for High Performance Computing, Networking, Storage and Analysis, SC'15, 2015.
[35]
H. S. Wong, S. Raoux, S. Kim, J. Liang, J. P. Reifenberg, B. Rajendran, M. Asheghi, and K. E. Goodson. Phase Change Memory. Proceedings of the IEEE, 98(12):2201--2227, Dec 2010.
[36]
J. Yang, J. Izraelevitz, and S. Swanson. FileMR: Rethinking RDMA Networking for Scalable Persistent Memory. In 17th USENIX Symposium on Networked Systems Design and Implementation (NSDI 20), pages 111--125, 2020.
[37]
Y. Zhang, J. Yang, A. Memaripour, and S. Swanson. Mojim: A Reliable and Highly-Available Non-Volatile Memory System. In ACM SIGARCH Computer Architecture News, volume 43, pages 3--18. ACM, 2015.
Recommendations
Using RDMA efficiently for key-value services
SIGCOMM '14: Proceedings of the 2014 ACM conference on SIGCOMMThis paper describes the design and implementation of HERD, a key-value system designed to make the best use of an RDMA network. Unlike prior RDMA-based key-value systems, HERD focuses its design on reducing network round trips while using efficient ...
FlatLSM: Write-Optimized LSM-Tree for PM-Based KV Stores
The Log-Structured Merge Tree (LSM-Tree) is widely used in key-value (KV) stores because of its excwrite performance. But LSM-Tree-based KV stores still have the overhead of write-ahead log and write stall caused by slow L0 flush and L0-L1 compaction. New ...
Using RDMA efficiently for key-value services
SIGCOMM'14This paper describes the design and implementation of HERD, a key-value system designed to make the best use of an RDMA network. Unlike prior RDMA-based key-value systems, HERD focuses its design on reducing network round trips while using efficient ...
Comments
Please enable JavaScript to view thecomments powered by Disqus.Information & Contributors
Information
Published In
November 2020
1454 pages
ISBN:9781728199986
- General Chair:
- Christine Cuicchi,
- Program Chairs:
- Irene Qualters,
- William Kramer
Sponsors
In-Cooperation
- IEEE CS
Publisher
IEEE Press
Publication History
Published: 09 November 2020
Check for updates
Author Tags
Qualifiers
- Research-article
Conference
SC '20
Sponsor:
SC '20: The International Conference for High Performance Computing, Networking, Storage and Analysis
November 9 - 19, 2020
Georgia, Atlanta
Acceptance Rates
Overall Acceptance Rate 1,516 of 6,373 submissions, 24%
Upcoming Conference
Contributors
Other Metrics
Bibliometrics & Citations
Bibliometrics
Article Metrics
- 0Total Citations
- 245Total Downloads
- Downloads (Last 12 months)16
- Downloads (Last 6 weeks)1
Reflects downloads up to 23 Nov 2024
Other Metrics
Citations
View Options
Login options
Check if you have access through your login credentials or your institution to get full access on this article.
Sign in