research-article

DComp: Efficient Offload of LSM-tree Compaction with Data Processing Units

Authors:

Peng XuAuthors Info & Claims

ICPP '23: Proceedings of the 52nd International Conference on Parallel Processing

Pages 233 - 243

https://doi.org/10.1145/3605573.3605633

Published: 13 September 2023 Publication History

Abstract

LSM-based Key-value stores suffer from sub-optimal performance due to their slow and heavy background compactions. The compaction overhead shifts to the CPU as the storage performance continuously increases. This paper further reveals that data-intensive compression in compaction consumes a significant portion of CPU power. Moreover, the multi-threaded compactions cause substantial CPU contention during high-load periods. Based on the above observations, we propose fine-grained dynamical compaction offloading by leveraging the modern Data Processing Unit (DPU) to alleviate the CPU overhead. To achieve this, we first employ dedicated hardware-based accelerators on the DPU to speed up the compression in compactions. We then leverage the Arm cores on the DPU to meet the burst CPU requirements to reduce resource contention. We integrate our DPU-offloaded compaction with RocksDB and evaluate it with NVIDIA’s latest Bluefield-2 DPU on a real system. The evaluation shows that the DPU is an effective solution to solve the CPU bottleneck of compaction. The results show that compaction performance is accelerated by 2.86 to 4.03 times, system write and read throughput is improved by up to 3.2 times and 1.4 times respectively, and host CPU contention is effectively reduced compared to the fine-tuned CPU-only baseline.

References

[1]

2023. Facebook. RocksDB, a persistent key-value store for fast storage enviroments. http://rocksdb.org/.

[2]

2023. Google. LevelDB. https://github.com/google/leveldb.

[3]

2023. Intel. DPDK: Data Plane Development Kit.https://www.dpdk.org/.

[4]

2023. Intel. Infrastructure Processing Unit.https://www.intel.com.au/content/www/au/en/products/network-io/smartnic.html.

[5]

2023. Marvell. OCTEON 10 DPU.https://www.marvell.com/products/data-processing-units.html.

[6]

2023. Nividia. Bluefield-2 DPU.https://network.nvidia.com/files/doc-2020/pb-bluefield-2-dpu.pdf.

[7]

2023. NVM Express over Fabrics Specification. https://nvmexpress.org/developers/nvme-of-specification/.

[8]

Abutalib Aghayev, Sage Weil, Michael Kuchnik, Mark Nelson, Gregory R Ganger, and George Amvrosiadis. 2019. File systems unfit as distributed storage backends: lessons from 10 years of Ceph evolution. In Proceedings of the 27th ACM Symposium on Operating Systems Principles. 353–369.

Digital Library

[9]

Muhammad Yousuf Ahmad and Bettina Kemme. 2015. Compaction management in distributed key-value datastores. Proceedings of the VLDB Endowment 8, 8 (2015), 850–861.

Digital Library

[10]

Oana Balmau, Florin Dinu, Willy Zwaenepoel, Karan Gupta, Ravishankar Chandhiramoorthi, and Diego Didona. 2019. SILK: Preventing Latency Spikes in Log-Structured Merge Key-Value Stores. In 2019 USENIX Annual Technical Conference (USENIX ATC 19). 753–766.

[11]

Laurent Bindschaedler, Ashvin Goel, and Willy Zwaenepoel. 2020. Hailstorm: Disaggregated compute and storage for distributed lsm-based databases. In Proceedings of the Twenty-Fifth International Conference on Architectural Support for Programming Languages and Operating Systems. 301–316.

Digital Library

[12]

Brad Burres, Dan Daly, Mark Debbage, Eliel Louzoun, Christine Severns-Williams, Naru Sundar, Nadav Turbovich, Barry Wolford, and Yadong Li. 2021. Intel’s Hyperscale-ready infrastructure processing unit (IPU). In 2021 IEEE Hot Chips 33 Symposium (HCS). IEEE, 1–16.

[13]

Idan Burstein. 2021. Nvidia Data Center Processing Unit (DPU) Architecture. In 2021 IEEE Hot Chips 33 Symposium (HCS). IEEE, 1–20.

[14]

Hao Chen, Chaoyi Ruan, Cheng Li, Xiaosong Ma, and Yinlong Xu. 2021. SpanDB: A Fast,Cost-Effective LSM-tree Based KV Store on Hybrid Storage. In 19th USENIX Conference on File and Storage Technologies (FAST 21). 17–32.

[15]

Yifan Dai, Yien Xu, Aishwarya Ganesan, Ramnatthan Alagappan, Brian Kroth, Andrea C Arpaci-Dusseau, and Remzi H Arpaci-Dusseau. 2020. From wisckey to bourbon: A learned index for log-structured merge trees. In Proceedings of the 14th USENIX Conference on Operating Systems Design and Implementation. 155–171.

[16]

Peter Deutsch. 1996. Rfc1951: Deflate compressed data format specification version 1.3.

[17]

Chen Ding, Ting Yao, Hong Jiang, Qiu Cui, Liu Tang, Yiwen Zhang, Jiguang Wan, and Zhihu Tan. 2022. TriangleKV: Reducing Write Stalls and Write Amplification in LSM-Tree Based KV Stores With Triangle Container in NVM. IEEE Transactions on Parallel and Distributed Systems 33, 12 (2022), 4339–4352.

Digital Library

[18]

Siying Dong, Andrew Kryczka, Yanqin Jin, and Michael Stumm. 2021. Evolution of Development Priorities in Key-value Stores Serving Large-scale Applications: The RocksDB Experience. In 19th USENIX Conference on File and Storage Technologies (FAST 21). 33–49.

Digital Library

[19]

Daniel Firestone, Andrew Putnam, Sambhrama Mundkur, Derek Chiou, Alireza Dabagh, Mike Andrewartha, Hari Angepat, Vivek Bhanu, Adrian Caulfield, Eric Chung, 2018. Azure accelerated networking: Smartnics in the public cloud. In 15th { USENIX} Symposium on Networked Systems Design and Implementation ({ NSDI} 18). 51–66.

[20]

Peter-Jan Gootzen, Jonas Pfefferle, Radu Stoica, and Animesh Trivedi. 2023. DPFS: DPU-Powered File System Virtualization. In Proceedings of the 16th ACM International Conference on Systems and Storage (Haifa, Israel)(SYSTOR’23). Association for Computing Machinery, New York, NY, USA. https://doi. org/10.1145/3579370.3594769.

Digital Library

[21]

Xiaokang Hu, Fuzong Wang, Weigang Li, Jian Li, and Haibing Guan. 2019. QZFS: QAT Accelerated Compression in File System for Application Agnostic and Cost Efficient Data Storage. In 2019 USENIX Annual Technical Conference (USENIX ATC 19). 163–176.

[22]

Dongxu Huang, Qi Liu, Qiu Cui, Zhuhe Fang, Xiaoyu Ma, Fei Xu, Li Shen, Liu Tang, Yuxing Zhou, Menglong Huang, 2020. TiDB: a Raft-based HTAP database. Proceedings of the VLDB Endowment 13, 12 (2020), 3072–3084.

Digital Library

[23]

Arpan Jain, Nawras Alnaasan, Aamir Shafi, Hari Subramoni, and Dhabaleswar K Panda. 2021. Accelerating CPU-based distributed DNN training on modern HPC clusters using bluefield-2 DPUs. In 2021 IEEE Symposium on High-Performance Interconnects (HOTI). IEEE, 17–24.

[24]

Jongyul Kim, Insu Jang, Waleed Reda, Jaeseong Im, Marco Canini, Dejan Kostić, Youngjin Kwon, Simon Peter, and Emmett Witchel. 2021. LineFS: Efficient SmartNIC Offload of a Distributed File System with Pipeline Parallelism. In Proceedings of the ACM SIGOPS 28th Symposium on Operating Systems Principles. 756–771.

Digital Library

[25]

Baptiste Lepers, Oana Balmau, Karan Gupta, and Willy Zwaenepoel. 2019. Kvell: the design and implementation of a fast persistent key-value store. In Proceedings of the 27th ACM Symposium on Operating Systems Principles. 447–461.

Digital Library

[26]

Qiang Li, Lulu Chen, Xiaoliang Wang, Shuo Huang, Qiao Xiang, Yuanyuan Dong, Wenhui Yao, Minfei Huang, Puyuan Yang, Shanyang Liu, 2023. Fisc: a large-scale cloud-native-oriented file system. In 21st USENIX Conference on File and Storage Technologies (FAST 23). 231–246.

[27]

Jiaxin Lin, Tao Ji, Xiangpeng Hao, Hokeun Cha, Yanfang Le, Xiangyao Yu, and Aditya Akella. 2023. Towards Accelerating Data Intensive Application’s Shuffle Process Using SmartNICs. Proceedings of the ACM on Measurement and Analysis of Computing Systems 7, 2 (2023), 1–23.

Digital Library

[28]

Ming Liu, Tianyi Cui, Henry Schuh, Arvind Krishnamurthy, Simon Peter, and Karan Gupta. 2019. Offloading distributed applications onto smartnics using ipipe. In Proceedings of the ACM Special Interest Group on Data Communication. 318–333.

Digital Library

[29]

Yoshinori Matsunobu, Siying Dong, and Herman Lee. 2020. MyRocks: LSM-tree database storage engine serving Facebook’s Social Graph. Proceedings of the VLDB Endowment 13, 12 (2020), 3217–3230.

Digital Library

[30]

Patrick O’Neil, Edward Cheng, Dieter Gawlick, and Elizabeth O’Neil. 1996. The log-structured merge-tree (LSM-tree). Acta Informatica 33, 4 (1996), 351–385.

Digital Library

[31]

Hieu Pham. 2021. Remote compactions in RocksDB-cloud share. Retrieved January 21 (2021), 2021.

[32]

Henry N Schuh, Weihao Liang, Ming Liu, Jacob Nelson, and Arvind Krishnamurthy. 2021. Xenic: Smartnic-accelerated distributed transactions. In Proceedings of the ACM SIGOPS 28th Symposium on Operating Systems Principles. 740–755.

Digital Library

[33]

Spencer Shepler, Brent Callaghan, David Robinson, Robert Thurlow, Carl Beame, Mike Eisler, and David Noveck. 2003. Network file system (NFS) version 4 protocol. Technical Report.

[34]

Shangyi Sun, Rui Zhang, Ming Yan, and Jie Wu. 2022. SKV: A SmartNIC-Offloaded Distributed Key-Value Store. In 2022 IEEE International Conference on Cluster Computing (CLUSTER). IEEE, 1–11.

[35]

Xiaoliang Wang, Jianchuan Li, Peiquan Jin, Kuankuan Guo, Yuanjin Lin, and Ming Zhao. 2021. Supporting Elastic Compaction of LSM-tree with a FaaS Cluster. In 2021 IEEE International Conference on Cluster Computing (CLUSTER). IEEE, 819–820.

[36]

Ziye Yang, James R Harris, Benjamin Walker, Daniel Verkamp, Changpeng Liu, Cunyin Chang, Gang Cao, Jonathan Stern, Vishal Verma, and Luse E Paul. 2017. SPDK: A development kit to build high performance storage applications. In 2017 IEEE International Conference on Cloud Computing Technology and Science (CloudCom). IEEE, 154–161.

[37]

Teng Zhang, Jianying Wang, Xuntao Cheng, Hao Xu, Nanlong Yu, Gui Huang, Tieying Zhang, Dengcheng He, Feifei Li, Wei Cao, 2020. FPGA-Accelerated Compactions for LSM-based Key-Value Store. In 18th USENIX Conference on File and Storage Technologies (FAST 20). 225–237.

Cited By

Zhong KYu ZLi QLuo XLong LTan YRen ALiu D(2024)DPC: DPU-accelerated High-Performance File System ClientProceedings of the 53rd International Conference on Parallel Processing10.1145/3673038.3673123(63-72)Online publication date: 12-Aug-2024
https://dl.acm.org/doi/10.1145/3673038.3673123
Kim JYoo HLee SByun HPark S(2024)Coordinating Compaction Between LSM-Tree Based Key-Value Stores for Edge Federation2024 IEEE 17th International Conference on Cloud Computing (CLOUD)10.1109/CLOUD62652.2024.00054(419-429)Online publication date: 7-Jul-2024
https://doi.org/10.1109/CLOUD62652.2024.00054

Index Terms

DComp: Efficient Offload of LSM-tree Compaction with Data Processing Units
1. Information systems
  1. Data management systems
    1. Database management system engines
      1. DBMS engine architectures

Recommendations

D²Comp: Efficient Offload of LSM-tree Compaction with Data Processing Units on Disaggregated Storage
LSM-based key-value stores suffer from sub-optimal performance due to their slow and heavy background compactions. The compaction brings severe CPU and network overhead on high-speed disaggregated storage. This article further reveals that data-intensive ...
gLSM: Using GPGPU to Accelerate Compactions in LSM-tree-based Key-value Stores
Log-structured-merge tree or LSM-tree is a technological underpinning in key-value (KV) stores to support a wide range of performance-critical applications. By conducting data re-organization in the background by virtue of compaction operations, the KV ...
Collaborative Compaction Optimization System using Near-Data Processing for LSM-tree-based Key-Value Stores
Abstract
Log-structured merge tree (LSM-tree) based key–value stores are widely employed in large-scale storage systems. In compaction, high-level sorted string table files (i.e., SSTables) are merged with low-level overlapping key ranges and ...
Highlights
- We propose an NDP model-based key-value store framework named as Co-KV which can make full use of the parallelism between the host and the device and ...

Comments

Please enable JavaScript to view thecomments powered by Disqus.

Information & Contributors

Information

Published In

cover image ACM Other conferences

ICPP '23: Proceedings of the 52nd International Conference on Parallel Processing

August 2023

858 pages

ISBN:9798400708435

DOI:10.1145/3605573

Copyright © 2023 ACM.

Permission to make digital or hard copies of all or part of this work for personal or classroom use is granted without fee provided that copies are not made or distributed for profit or commercial advantage and that copies bear this notice and the full citation on the first page. Copyrights for components of this work owned by others than the author(s) must be honored. Abstracting with credit is permitted. To copy otherwise, or republish, to post on servers or to redistribute to lists, requires prior specific permission and/or a fee. Request permissions from [email protected].

Publisher

Association for Computing Machinery

New York, NY, United States

Publication History

Published: 13 September 2023

Permissions

Request permissions for this article.

Request Permissions

Check for updates

Author Tags

Qualifiers

Research-article
Research
Refereed limited

Conference

ICPP 2023

ICPP 2023: 52nd International Conference on Parallel Processing

August 7 - 10, 2023

UT, Salt Lake City, USA

Acceptance Rates

Overall Acceptance Rate 91 of 313 submissions, 29%

Contributors

Other Metrics

View Article Metrics

Bibliometrics & Citations

Bibliometrics

Article Metrics

2
Total Citations
View Citations
484
Total Downloads

Downloads (Last 12 months)381
Downloads (Last 6 weeks)36

Reflects downloads up to 12 Nov 2024

Other Metrics

View Author Metrics

Citations

Cited By

Zhong KYu ZLi QLuo XLong LTan YRen ALiu D(2024)DPC: DPU-accelerated High-Performance File System ClientProceedings of the 53rd International Conference on Parallel Processing10.1145/3673038.3673123(63-72)Online publication date: 12-Aug-2024
https://dl.acm.org/doi/10.1145/3673038.3673123
Kim JYoo HLee SByun HPark S(2024)Coordinating Compaction Between LSM-Tree Based Key-Value Stores for Edge Federation2024 IEEE 17th International Conference on Cloud Computing (CLOUD)10.1109/CLOUD62652.2024.00054(419-429)Online publication date: 7-Jul-2024
https://doi.org/10.1109/CLOUD62652.2024.00054

View Options

Get Access

Login options

Check if you have access through your login credentials or your institution to get full access on this article.

Full Access

Get this Publication

View options

PDF

View or Download as a PDF file.

eReader

View online with eReader.

HTML Format

View this article in HTML Format.

Media

Figures

Other

Tables

View Table of Contents